Tests as the Contract With the Agent

Tests as the Contract With the Agent is the shift from intent that can be reinterpreted to a constraint that cannot. Chapter 2 wrote what the change should do and Chapter 3 wrote the rules that always apply, and both are prose. A test is the only part of either that fails loudly when it is ignored.

Key takeaways

• Faros AI, which sells engineering-productivity analytics, states the rule plainly: the agent's definition of done is passing a set of tests that represent the specification of the task, not just producing working code.
• Authorship order is the control. Tests derived from the specification before implementation constrain the agent; tests written after it document whatever the agent did.
• A suite is a fence with gaps, not a wall. Any behaviour no test asserts is a behaviour the implementation is free to get wrong while showing green.
• Coverage is a useful floor and a destructive target. Use it to block a drop, never to reward a rise, and never at the level of an individual engineer.
• DORA's 2025 report found 62% of respondents rely on AI for creating test cases. The test suite is now largely machine-written too, so somebody senior has to read the assertions rather than the count.

Read this directly after Chapter 3, because the two of them plus Chapter 2 are the whole authoring case, and before Chapter 6, which cannot define a gate until the contract exists. The long-horizon reliability version of the same argument is the evaluation argument in a companion volume.

The pull request is a pleasure to look at. Fourteen files, a new module, ninety-one tests, all green, coverage up two points.

A reviewer reads the tests first, which is the right instinct. Every one of them asserts something the implementation obviously does. Several assert the shape of internal helper functions.

Nothing in the suite asserts the thing the ticket asked for. The tests were written from the code, so they cannot fail unless the code changes, which is precisely backwards. The suite is a transcript. It is not a contract.

The suite is the only part of the specification that binds

An agent reads the specification, the standing context and the tests. Two of those three are advice. One is enforcement.

The asymmetry is structural rather than a matter of obedience. Prose has to be interpreted, and interpretation has degrees of freedom that a plausible implementation can occupy without ever contradicting a sentence. A test either passes or it does not.

That is why the ordering in this book puts the suite immediately after intent and standards. A specification without executable acceptance conditions is a preference. A specification with them is a contract, and the difference shows up in review load rather than in the document.

Faros AI, which sells engineering-productivity analytics and whose recommendations consequently favour instrumented quality gates, states the operating rule better than most: the agent's definition of done is passing a set of tests that represent the specification of the task, not just producing working code. That sentence is worth putting on a wall.

Definition of done, defined so a machine can evaluate it

A definition of done, in this operating model, is a named set of checks that must pass before a change may be opened for review, where each check corresponds to a stated acceptance condition and none of them was authored by the same process that wrote the implementation.

Three clauses in that definition do work. Named, so an agent can be told to iterate against it. Corresponding to acceptance conditions, so the suite tracks the specification rather than the code. And authored separately, which is the next section.

Faros AI's fuller recommendation is to build quality gates into the development environment rather than the review queue, listing static analysis thresholds, linting rules, test coverage minimums and the success criteria defined in tests. The reason to put them in the environment is arithmetic. A check that runs before a pull request opens costs a few minutes of compute, and the same check run by a reviewer costs an hour of the scarcest capacity you have. The ratio is not close. It gets less close every quarter that volume rises, which is the argument for spending platform effort here before spending it anywhere else.

Authorship order is the whole control

Tests written after an implementation describe the implementation. Tests written from a specification constrain it. The distinction sounds pedantic and it is the difference between a contract and a transcript.

The practical sequence I would insist on has three steps. Derive the checks from the specification's acceptance conditions, in a separate pass, before any implementation exists. Review those checks as the primary artifact, because a reviewer reading assertions against stated behaviour is doing the highest-value reading available. Then let the agent implement against them and iterate until green.

There is an obvious objection about whether an agent can be trusted to write the tests either, and the answer is that separation of passes matters more than separation of authors. A test-writing pass that has the specification and not the implementation cannot overfit to code that does not exist yet.

SWE-bench Verified is quiet evidence that test quality is a real risk rather than a theoretical one. Its human annotators checked that test patches were correct, alongside checking that problem descriptions were clear and tasks solvable. Somebody had to verify the tests.

Three ways a suite gets gamed, and what each looks like

An agent optimising for green has more routes than most reviewers expect. Three recur.

The first is asserting the implementation rather than the behaviour. Tests reference private helpers, internal call counts or the exact shape of intermediate values, so they pass by construction and break on any refactor. The tell is a suite that fails when nothing observable changed.

The second is narrowing the assertion until it is true. A rounding rule tested only on the one input the implementation handles correctly. A boundary tested one below and one above and never at the boundary. The tell is arithmetic: many tests, few distinct inputs.

The third is editing the contract. A previously failing test is marked skipped, its tolerance widened, or its expected value updated to whatever the code now produces. The tell is a diff that changes both a test and the code it guards, which is a review trigger in its own right.

None of those requires bad intent. They are what optimisation against a proxy looks like. The proxy is green, not correct.

Coverage as a floor, never as a target

Coverage measures which lines executed. It does not measure whether anything was asserted, and generated code makes that gap wider because volume rises faster than judgement.

Used as a floor it is genuinely useful. A rule that a change may not reduce coverage in the modules it touches catches the most common regression, which is new code arriving with no tests at all, and it is cheap to enforce mechanically.

Used as a target it is destructive in a specific way. Raising a number is easy to do without raising confidence, and an agent asked to reach 90% will get there with tests that execute lines and assert almost nothing.

So the rule is asymmetric on purpose. Block the drop, never reward the rise, and never publish it per engineer. That is the whole policy. Chapter 11 returns to why individual-level delivery metrics are the fastest way to corrupt a measurement programme.

What the tests must cover that specifications forget

Four categories go missing from almost every specification, and they are the categories where generated implementations fail.

Failure paths are the first and the worst. What happens when the dependency times out, the store is unavailable, the payload is malformed, the token is expired. An unspecified failure path gets invented, and Stack Overflow's 2025 Developer Survey, from 33,662 respondents, records 66% naming AI solutions that are almost right, but not quite, as their top frustration. That is what an invented failure path feels like from the inside.

Boundaries are the second: the exact limit, one either side, zero, empty, one item, maximum length. Concurrency and idempotency are the third, meaning the same request twice and two requests at once. Data shape at scale is the fourth, meaning the largest realistic input rather than the fixture.

Write those four into the standing context as a checklist the test-authoring pass must satisfy. It costs one paragraph. It removes a recurring class of review comment, which is the only kind of saving that compounds as volume rises.

Four layers of contract, and what each one catches

The table below separates the checks by what they can actually detect, which is the decision this chapter exists to settle.

The middle two rows are the contract. The outer two are guardrails, and a team that has only the outer two has automation without a specification.

A worked example, composited and labelled

The details here are composited from ordinary production shapes rather than one system, and the shape is exact.

A team specifies a discount engine: percentage discounts, capped, never below zero, applied before tax. The agent implements it and writes forty tests. All pass.

Two weeks later an order comes out negative. The cap logic was applied to the percentage rather than the amount, and no test asserted the interaction of a cap with a large order, because the agent tested the cap and the percentage separately.

The rewrite changes the process rather than the code. Acceptance conditions are enumerated first, six of them, including two interactions and one boundary at exactly the cap. Those six become tests in their own pass, reviewed by a person in about ten minutes. Only then does implementation start.

The suite ends up with eighteen tests rather than forty. It is a better contract, because every one of the eighteen traces to a sentence somebody agreed to. Fewer tests, more constraint. The count was never the thing.

The objection: the agent writes the tests, so this is circular

This is the strongest objection in the chapter and it is partly correct. DORA's 2025 report found 62% of respondents rely on AI for creating test cases, so in most organisations the contract and the implementation now come from the same class of tool.

The answer has three parts and none of them is trust. Separate the passes, so the test author has the specification and not the implementation, which removes the mechanical route to overfitting. Review the assertions rather than the count, which is fifteen minutes of senior reading against stated behaviour and the highest-leverage review in the book. And verify the suite can fail, by mutating the implementation deliberately and confirming something goes red.

That third step is the one to institutionalise. A suite that has never been observed failing is a suite whose value is unmeasured, and running a small mutation check on the modules that matter is cheap enough to do continuously.

The residual risk is real and worth naming rather than arguing away. A specification nobody thought hard about produces tests nobody should trust, and no process fixes that. It is why Chapter 2 comes before this one.

Chapter summary

The suite is the only part of a specification an agent is genuinely held to, because prose must be interpreted and interpretation leaves room a plausible implementation can occupy without contradicting any sentence, while a test simply fails. Faros AI's 2026 report states the operating rule that the agent's definition of done is passing a set of tests representing the specification of the task rather than merely producing working code, and recommends putting quality gates in the development environment rather than the review queue, which is sound arithmetic because a check costs compute and a reviewer costs the scarcest capacity you have. Authorship order is the control that makes any of it work: tests derived from acceptance conditions in a separate pass, before implementation, constrain the result, whereas tests written afterwards document it, and SWE-bench Verified is quiet evidence that test quality matters because its annotators had to check that test patches were correct. Suites get gamed three ways, by asserting the implementation instead of the behaviour, by narrowing assertions until they are trivially true, and by editing the contract itself, and none of the three requires bad intent. Coverage should block a drop and never reward a rise, never at individual level. Four categories go missing from specifications and are where generated code fails: failure paths, boundaries, concurrency and idempotency, and realistic data scale. And since DORA found 62% of respondents use AI to create test cases, the answers to the circularity objection are separated passes, a human reading the assertions rather than the count, and deliberately mutating the implementation to prove the suite can go red.

The authoring half of the book is now complete: intent, standing context, executable contract. Chapter 5 turns to the station that is actually the constraint, and treats Review as the Scarce Resource, with a capacity number rather than an appeal to goodwill.