A field guide to assurance-managed AI development

A coding agent receives a task, edits a repository, runs tests, and reports that the work is complete. To decide whether that result can be used, I need answers to several older engineering questions:

What was the system supposed to do?
Which claim do the tests or reviews support?
Did the approved inputs and tool produce this exact output?
Which permissions applied to the action?
What happens when a check misses a defect?

Those questions do not form a new discipline called agent assurance. They belong to requirements engineering, assurance cases, formal methods, tool qualification, secure development, supply-chain security, runtime verification, and resilience engineering.

This guide is organized by the question each field helps answer. For every source, it also states what the source does not establish. That second part is important: provenance can prove where an image came from without proving the image is correct, and a passing test can verify a requirement without showing that the requirement was the right one.

The shortest useful route

For a first pass, read in this order:

SEBoK: Requirements Engineering, System Verification, and System Validation.
The SEI introductions to assurance cases and assurance-case resources.
Formal Arguments for Large-Scale Assurance for systems that keep changing.
RTCA DO-330 and ISO 26262-8 for the choice between trusting a tool and checking its output.
NIST SP 800-218 for secure development across the lifecycle.
in-toto and SLSA 1.2 for recording how an artifact was produced.
NIST SP 800-160 Volume 2 for recovery when prevention is incomplete.
Trusted AI-assisted Programming and Programming with Trust for current work that connects those methods to coding agents.

The sequence starts with intent and ends with agents. Starting with agent tools makes familiar engineering controls look newer and stranger than they are.

What should the system do?

Requirements engineering separates what stakeholders need from the properties the system must satisfy. It also distinguishes verification—did we build the specified thing?—from validation—does the integrated system serve its intended use in its real environment?

Source	Use it for	It does not decide
SEBoK: Requirements Engineering	Eliciting, analysing, defining, and maintaining requirements.	Which stakeholder objective should win.
SEBoK: System Requirements Definition	Writing necessary, feasible, traceable, and verifiable requirements.	Whether the selected outcome is socially or commercially right.
SEBoK: Verification and Validation	Separating conformance from fitness for intended use.	The other half of that distinction.
Trusted AI-assisted Programming	Research on turning natural-language intent into specifications, tests, and verification-aware interactions.	Whether the formalized intent is the intent people actually wanted.

A prompt is not a requirements process. A passing test suite is not, by itself, validation.

Why should a claim be believed?

An assurance case connects a specific claim to an argument and supporting evidence. It also exposes assumptions and context. This is a better model than attaching a generic confidence badge to a build.

Source	Use it for	It does not provide
SEI: Assurance Cases Overview	The basic claim–argument–evidence structure.	The engineering work that produces good evidence.
GSN Community Standard	A notation for goals, strategies, evidence, and assumptions.	Sound reasoning merely because the diagram is neat.
OMG SACM	A structured interchange model for claims, arguments, and artifacts.	A convincing argument merely because it is machine-readable.
SEI: Formal Arguments for Large-Scale Assurance	Maintaining assurance arguments as systems and evidence change.	Proof that reassessment is already cheap or automatic.

This is where the local phrase “derive status only from reproducible evidence” belongs. The evidence supports a named claim about an identified configuration; it is not a permanent property of the file that happened to pass.

How can the design or implementation be checked?

Different methods support different claims:

Source	Useful claim	Limit
TLA+ and Specifying Systems	A model of a concurrent or distributed design satisfies checked safety or liveness properties.	The implementation may differ, and the specification may be wrong.
Dafny	Code satisfies stated preconditions, postconditions, and invariants.	The stated properties may omit the real requirement.
AWS use of formal methods	Formal specification can find design defects in production-scale distributed systems.	Every system needs the same level of formal work.
Cedar	Authorization policy can be separated, analysed, and checked against a formal model.	Assurance of the whole application.
Runtime Verification	A running system can be monitored against a specification.	Claims about unobserved paths unless the monitor enforces them.

A small TLA+ model of a dangerous state transition and a verified program are both formal work. They answer different questions and cost different amounts.

Can the producer be trusted instead of every output?

Safety-critical engineering already has a direct answer. A development tool is qualified to a level appropriate to the errors it can introduce, or its output is verified.

Source	Use it for	Limit
RTCA DO-330	Tool qualification in airborne systems.	It does not remove output checking when the tool cannot be qualified.
ISO 26262-8	Confidence in software tools used for automotive development and verification.	Domain-specific judgment is still required.
CompCert	The limit case: a compiler backed by a formal correctness proof.	The source program and specification can still be wrong.

A general language model is non-deterministic, can inject many classes of error, and is opaque to the analysis these regimes expect. Under this framework, that selects output verification. It does not justify trusting generated code because the model is widely used or because the output looks conventional.

How does this fit ordinary development?

Source	Use it for	It does not provide
NIST Secure Software Development Framework	Secure practices integrated through an existing development lifecycle.	Local implementation or risk decisions.
SEBoK: Model-Based Systems Engineering	Maintaining relationships among requirements, architecture, analysis, verification, and validation.	Useful models without a maintenance discipline.
AI for the SDLC	Current guidance on autonomy levels, workflow controls, and human–AI teaming.	A universal architecture or proof that a chosen agent is safe.
GitHub Spec Kit	A concrete separation of specification, planning, tasks, and implementation.	Independent evidence that the implementation is correct.

The test is simple: do the lifecycle controls still work when an agent performs the implementation step? Renaming the lifecycle “agentic” does not strengthen it.

Who produced this result?

Source	Use it for	It does not establish
in-toto	Authorized supply-chain steps, actors, materials, products, and signed execution metadata.	That the requested software was semantically correct.
SLSA 1.2	Incremental source and build integrity guarantees and verifiable provenance.	That the source itself should be trusted.
Sigstore	Identity-bound signing and transparent verification of artifacts and attestations.	A policy deciding which identities and workflows are acceptable.

These methods make attempt records and artifact lineage stronger than ordinary logs. They still cannot prove that the result was wanted.

Who may act, and what happens after failure?

The reference monitor and security-kernel model says the trusted mechanism should mediate relevant access, resist modification, and be small enough to inspect. Cedar provides one practical example: principals, actions, resources, and context are evaluated by a policy engine rather than left to an agent to interpret from prose.

Prevention will remain incomplete. Recovery therefore belongs in the design:

Source	Use it for	It does not replace
NIST SP 800-160 Volume 2	Designing systems to anticipate, withstand, recover from, and adapt to adverse conditions.	Prevention or tested recovery procedures.
Google SRE books	Monitoring, gradual rollout, incident response, error budgets, and rollback.	Formal proof or domain-specific safety assurance.
Google SRE: Safe configuration change	Gradual deployment, stop conditions, hermetic change, and rollback.	A safe underlying state transition merely because rollout is gradual.

An agent should not be able to grant itself a permission, redefine a failed check, or hide the information needed to undo its action.

AI-specific work worth following

The established material above supplies the vocabulary and methods. Current AI work asks how to combine them without making every change prohibitively expensive.

Useful starting points include:

Trusted AI-assisted Programming and Programming with Trust, which connect agents with specification, testing, analysis, and verification;
AI for the SDLC workflow guidance, which works through task autonomy and oversight;
DARPA AI Cyber Challenge results, which demonstrate automated analysis and patching at substantial scale;
METR’s developer RCT, which found experienced developers slower with AI assistance while they believed they were faster;
DORA’s State of AI-assisted Software Development and AI Capabilities Model, which examine AI in delivery processes rather than as an isolated tool; and
the public Exploring Gen AI series, including work on human intervention, internal quality, spec-driven development, harness engineering, and context engineering.

This section is a dated watch list, not a curriculum. Research programmes, practitioner reports, government guidance, and vendor-sponsored surveys carry different kinds of evidence. None yet supplies a settled, proportionate method for composing all the controls above.

How this changes the local vocabulary

Local note	Established neighbourhood	Correction to keep
Derive status only from reproducible evidence	Assurance cases and configuration identification	Attach evidence to a claim and configuration, not a generic status.
The agent is not the application	Reference monitors and trusted computing bases	Keep the trusted mechanism smaller than the whole application.
The work between the ticket and the agent	Requirements traceability, provenance, and separation of duties	Treat a neutral attempt record as one implementation choice, not a new field.
The missing layer is binding, not intelligence	Systems integration, assurance cases, MBSE, and configuration management	Test the proposed bindings against established lifecycle methods.

Maintenance rule

Add a resource only when the entry can answer four questions:

Which engineering question does it help answer?
Which claim can its method support?
What does it leave unsolved?
What kind of source is it: standard, primary documentation, research result, industrial report, or interpretation?

Do not call a problem unsolved merely because the established method is expensive or poorly adopted. Do not call a structured agent workflow assurance merely because it produces more files than a chat session.

The purpose of this guide is modest: provide a reliable route from an agent workflow problem to the engineering fields that already know how to name and test it.