Scoring the Path, Not Just the Answer

Scoring the Path, Not Just the Answer is the practice that catches the failures an outcome metric cannot see. It is also the practice most likely to be abandoned, because the obvious implementation, comparing against a recorded sequence of steps, breaks on every legitimate improvement.

Key takeaways

• Assert invariants, not sequences. The question is whether required properties held, not whether the run took the route you had in mind.
• Four invariant shapes cover most cases: something must happen before the answer, something must never happen, something may happen at most once, and something must follow a failure.
• Score recovery explicitly. A run that hit a tool error and worked around it is better than one that never hit an error, and outcome metrics rate them identically.
• A right answer with a wrong path is a warning, not a pass. Record it as its own category, because it predicts the failure of the next similar input.
• Redundant work is the cheapest signal of a degrading run: the same query issued three times, or a loop with no new information, usually precedes a bad answer.

Read this beside Chapter 4, which placed trajectory in its own family, and Chapter 11, which supplies the traces this scoring reads. Chapter 12 uses the same records during an incident.

An agent is asked to update a customer's billing address. It calls the search tool four times with the same query, reads an unrelated record, then writes the correct address.

The outcome is perfect. The path says the run had no idea what it was doing until it did. The next address that is slightly harder to find will fail, and today's green result will have predicted nothing about it.

Why exact-sequence assertions fail

The natural first implementation is to record a known-good run and compare future runs against it, step by step.

It fails within a fortnight. A better tool description in the sense Anthropic's September 2025 guidance recommends changes which tool the model selects. A prompt improvement removes a redundant step. A model update reorders two independent calls. Every one of those is an improvement, and every one turns the suite red.

What follows is predictable. Someone re-records the expected sequences, which is tedious, so it happens late. The suite is red for a week, people learn to ignore it, and trajectory scoring is quietly disabled.

The deeper problem is conceptual. There is rarely one correct path. Reading the record first and then searching is as valid as searching first, and a suite that has an opinion about which is measuring style rather than correctness.

So assert what must be true of any acceptable path, and let the route vary.

The four invariant shapes

Almost every useful trajectory assertion is one of four shapes, and naming them makes them easy to write.

Precedence. Something must happen before something else. Retrieval before a grounded answer. A read of the current value before a write. A permission check before an export.

Prohibition. Something must never happen in this task family. No write to production during a read-only query. No call to the payments tool from the support flow. No retrieval from a corpus the requester cannot see.

Cardinality. Something may happen at most, or at least, some number of times. At most one destructive call. At least one verification step. No more than three retries of the same operation.

Recovery. Something must follow a condition. After a failed tool call, either a retry with changed arguments, an alternative tool, or an escalation, but never silent continuation.

Those four cover the majority of what matters, and they are all checkable from a structured trace without a model in the loop.

Score recovery as a first-class outcome

The most valuable trajectory property is the one that only appears when something goes wrong.

Inject failures deliberately in evaluation runs. Make a tool return an error, a timeout, an empty result or a malformed payload, and score what the run does next. That is a property no ordinary run will exercise, and it is exactly the property that decides how the system behaves on a bad day.

AgentLens is explicit that recovery belongs in the assessment, listing how an agent recovers from mistakes alongside instruction following, tool use and self-verification as dimensions of its trajectory review. The framing to borrow is that a trajectory review should produce a readable explanation rather than only a number.

Three recovery behaviours are worth separate scores. Detection, meaning the run noticed the failure at all rather than treating an empty result as an answer. Adaptation, meaning it changed something rather than repeating the same call. And termination, meaning it stopped or escalated rather than looping until a budget ran out.

Systems usually fail the first of those. An empty search result treated as evidence of absence is the single most common trajectory defect I have seen, and it is trivially detectable in a trace.

Right answer, wrong path is its own verdict

Once the two families are separated, a third category appears, and it should not be collapsed into either.

Record four verdicts per case rather than two. Correct answer with a sound path, which is the target. Correct answer with an unsound path, which is a warning. Wrong answer with a sound path, which usually means the evidence or the corpus is at fault rather than the reasoning. And wrong answer with an unsound path, which is the ordinary failure.

The second and third categories are where the diagnostic value sits. A rise in correct-but-unsound predicts a coming quality drop, because the system is being carried by luck that will not survive an input change. A rise in wrong-but-sound points at Chapter 3's corpus and Chapter 2's criteria rather than at the model.

Report the four counts, not a pass rate. The distribution across them says more about where to spend the next fortnight than any single number.

Reliability degrades along the path, not at the end

Trajectory measurement matters more as tasks lengthen, and there is measured support for treating length as its own variable.

Khanal, Tao and Zhou's March 2026 reliability framework proposes measures shaped for long horizons rather than single-attempt success, including a reliability decay curve and a meltdown onset point, and reports that decay varies sharply by domain, with software engineering falling from 0.90 to 0.44 across the horizon while document processing stayed close to flat.

The practical consequence for a suite is to record where in the run a failure first appears rather than only that the run failed. Step index, elapsed steps since the last verification, and whether the failure followed a recovery attempt.

That turns a set of failures into a shape. If failures cluster after step eight, the useful work is a checkpoint or a verification at step seven, and no amount of prompt improvement will substitute for it.

It also gives an honest way to describe your system's limits: this task family is reliable to about six steps, and beyond that it needs a human check. That sentence is worth more to a product owner than a percentage.

Keep the trace structured enough to assert on

None of this is possible if the trace is a log of strings.

The trace needs, per step: the tool or model called, the arguments in structured form, the result status, a hash or identifier of the result rather than the whole payload where it is large, the elapsed time, and any error. Chapter 11 covers the tracing conventions that make this portable.

Two properties matter for assertions specifically. Steps must be ordered unambiguously, including where parallel calls occurred, because a precedence assertion is meaningless otherwise. And arguments must be structured, since asserting that a write targeted the right record is a field comparison and not a substring search.

Retain traces long enough to compare across releases. Store enough to re-read the run by hand. When an assertion fails, somebody has to understand why, and a trace that supports assertions but not reading produces unexplainable red builds.

Where the invariants come from

Writing invariants from scratch produces a thin list. Three sources produce a good one.

Incidents. Every trajectory-related incident yields at least one invariant, phrased as the thing that should have been impossible. A run that deleted a record during a read-only request becomes a prohibition, permanently.

The tool contracts from your own API. If a write endpoint requires a fresh read for optimistic concurrency, that is a precedence invariant for free, and it was already documented.

The safety review. Anything the business would consider unacceptable regardless of outcome is a prohibition, and those are the invariants worth writing before launch rather than after. Payments, deletions, external communication and permission changes are the usual list.

Keep the set small and enforced rather than large and aspirational. Twelve invariants that always run beat sixty that somebody disabled during a release.

Review them when the product changes shape. An invariant written for a read-only assistant becomes wrong the day the assistant gains a write tool, and the failure mode is a suite that blocks legitimate work.

Use a judge for the parts assertions cannot reach

Some path properties resist mechanical assertion. Whether the run's plan was sensible. Whether it asked the user a reasonable clarifying question. Whether its explanation of what it did was accurate.

Those need a judge, with the cautions from Chapters 6 and 7. Give it the structured trace rather than a narrative, ask for a verdict per named property rather than an overall impression, and validate it against human-reviewed traces before trusting the numbers.

Zheng and colleagues' MT-Bench findings apply directly here, including verbosity bias: a judge shown a long trace with many steps may rate it as more thorough, when a shorter path was better. Counter it by giving the judge the step count and cost alongside the trace, and by asking explicitly whether any step was unnecessary.

Keep the judged share of trajectory scoring small. Mechanical invariants are cheap, stable and explainable; judged properties cost money on every run and drift when the judge model changes. A reasonable split is that every case gets the invariants and a sampled subset gets the judged review, which keeps the cost bounded while still surfacing the qualitative failures.

Chapter summary

Trajectory scoring catches what outcome metrics cannot, and it fails when implemented as exact-sequence comparison, because better tool descriptions, prompt improvements and model updates all change the route legitimately and turn the suite red until somebody disables it. Assert invariants instead, in four shapes: precedence, prohibition, cardinality and recovery, all checkable from a structured trace without a model. Score recovery as a first-class property by injecting tool failures deliberately and grading detection, adaptation and termination separately, since treating an empty result as evidence of absence is the most common defect and the easiest to detect. Record four verdicts rather than two, separating correct-with-sound-path from correct-with-unsound path and wrong-with-sound-path, because a rise in the second predicts a coming quality drop and a rise in the third points at the corpus rather than the model. Record where in the run failures first appear, since reliability decays along the path and the 2026 reliability work found decay varying sharply by domain, which turns a failure list into a shape and yields an honest statement of how many steps a task family supports. Keep traces structured with ordered steps, structured arguments, result status and timing, so assertions are field comparisons rather than substring searches and a red build can be read by a person. And use a judge only for the properties assertions cannot reach, giving it the trace with step count and cost to counter verbosity bias, asking for verdicts per named property, and keeping the judged share small.

The judge has now appeared in three chapters as something to be careful with. Chapter 6 is LLM Judges Need Their Own Test Sets, which treats the judge as a system under test with its own dataset, its own regressions and its own release process.