Outcome, Trajectory, and System Metrics

Outcome, Trajectory, and System Metrics are three answers to three different questions, and the habit of collapsing them into one number is how teams end up unable to explain their own results. The families interact, they trade against each other, and the trade is the information.

Key takeaways

• Outcome metrics say whether the result was right. They are necessary, insufficient, and the only family most teams measure.
• Trajectory metrics say whether the path was sound. A correct answer reached by a lucky route will not survive a slightly different input, and only the path shows that.
• System metrics say what it cost. Latency, tokens, tool calls and cost per successful outcome belong in the same report as quality, because a quality gain bought with triple the cost is a business decision rather than an engineering win.
• Do not average across families. A composite score hides which axis moved, and every real release decision is about the trade between them.
• Report cost per successful outcome rather than cost per call. A cheap call that fails and is retried twice is not cheap.

Read this beside Chapter 2, whose rubric supplies the outcome criteria, and Chapter 5, which develops trajectory scoring properly. Chapter 9 turns the three families into gate rules that behave differently.

A change to the prompt improves answer quality by four points on the suite. It also triples the number of tool calls per request, because the model now searches three times before answering.

Both facts were measurable. Only the first was measured, and the second arrives six weeks later as a bill and a latency complaint.

Outcome: was the result right

Outcome metrics evaluate the artifact the user receives, against the criteria written in Chapter 2.

The mechanical half is cheap and should be exhaustive. Required facts present. Required citation present and resolving to a real source. Prohibited content absent. Correct refusal when the case is out of corpus. Structural validity where the output is structured, meaning it parses and satisfies its schema.

The judged half covers what cannot be checked mechanically: faithfulness to the evidence, completeness, and whether the response answers the question that was asked. Chapter 6 is about making that half trustworthy, because an unvalidated judge produces outcome numbers with unknown error.

Two reporting rules keep outcome metrics honest. Report per criterion rather than as a single pass rate, because "83% pass" conceals whether the failures are missing citations or invented figures, and those are different emergencies. And report per stratum from Chapter 3, since a system that is excellent on common lookups and poor on refusals has a profile, not a score.

What outcome metrics cannot tell you is why. A drop from 88 to 81 says something changed and nothing about where, which is the gap the next two families fill.

Trajectory: was the path sound

For anything with steps, tools or retrieval, the path is measurable and it carries information the answer does not.

The reason is generalisation. A run that reached the right answer by searching for the wrong thing, getting nothing useful, and then guessing correctly from parametric knowledge has produced a correct outcome and a broken process. The next input in that family will fail, and the outcome metric will not have warned you.

AgentLens, published in July 2026, is the clearest published statement of this argument. It evaluates coding agents on the whole trajectory rather than as binary pass or fail, assessing how the agent follows instructions, uses its tools, verifies its own work, recovers from mistakes and communicates, and it combines formal verification with generated trajectory reviews so the scores come with readable explanations. The authors describe using it for nightly regression detection, which is the operational shape this book is arguing for.

Practical trajectory measures are simpler than the research versions. Did the run use the tools a competent engineer would have used. Did it retrieve before answering on questions that require evidence. Did it recover from a failed tool call rather than abandoning or looping. Did it avoid redundant work, meaning the same query issued three times with no new information. And did it stop when it had enough.

Chapter 5 covers how to assert these without over-constraining runs that legitimately take different routes.

System: what did it cost

System metrics are the family engineers instinctively have and evaluation reports routinely omit.

Latency, at the percentiles that matter rather than the mean, because the tail is what users complain about. Tokens in and out, separated, since input tokens scale with your context decisions and output tokens with verbosity. Tool calls per request. Model calls per request, which in an agentic system is often several times what anyone assumes. And cost per request, derived from the above.

Then the metric that reframes all of them: cost per successful outcome. A configuration that halves cost per call while failing twice as often has not saved anything, and only the joined metric shows it.

Two cautions. Measure system metrics on the same runs that produced the quality numbers, not separately, or the comparison is meaningless. And record the model version and configuration with every result, because provider-side changes move these numbers without any deploy on your side.

DORA's current metric set, updated in January 2026, remains the right instrument for delivery health, including change fail rate and failed deployment recovery time. Keep it separate from this family. Delivery metrics describe how well you ship; system metrics describe what a request costs; and neither detects a fluently wrong answer.

The three families trade against each other

The reason for keeping them apart is that improving one usually costs another, and the trade is the decision.

More retrieval improves outcome and raises token cost and latency. More retries improve outcome and raise cost. A tighter prompt lowers cost and can lower completeness. A smaller model lowers cost and latency and usually lowers outcome quality on the hard strata specifically, which a whole-suite average will understate.

A composite score erases all of that. Somebody produces a weighted number, the weights encode an opinion nobody re-examines, and a change that traded quality for cost appears as a small movement in a single digit.

So report three numbers, side by side, per change. If a stakeholder wants one number, give them the outcome number with the cost beside it, and say explicitly that no single figure exists.

Google's error budget framing from the SRE practice is the right way to make the trade explicit rather than implicit: state the quality you intend to provide, measure against it, and let the remaining budget decide how much cost or latency you are willing to spend on improvement.

What each family cannot tell you

Stating the limits is what stops each family being over-read.

Outcome metrics cannot tell you whether a passing run will keep passing, because a lucky path scores the same as a sound one. They also cannot tell you about cases you did not include, which is why the sampling design from Chapter 3 is part of the result.

Trajectory metrics cannot tell you whether the user was helped. A perfect path can produce an answer that is technically supported and useless, and a slightly odd path can produce exactly what the user needed.

System metrics cannot tell you anything about quality at all. A system that answers every question with "I do not know" has excellent latency, minimal cost and a perfect refusal rate on out-of-corpus cases.

That last case is worth keeping as a standing sanity check. Run it deliberately: what would the always-refuse system score on your suite. If the answer is respectable, your suite is not measuring usefulness and needs cases where a refusal is wrong.

Aggregate carefully, and show the distribution

Once the families are separate, there is still a question of how to aggregate within each.

Report per stratum first, then a total, and never only the total. The strata are why the sampling was designed, and the interesting movement is almost always in one of them.

Show the distribution rather than the mean for anything graded. A mean of 2.1 on a three-point scale can be everybody scoring two, or half scoring one and half scoring three, and those are different products.

Use counts for the hard requirements. Three cases with an invented figure is a sentence anybody understands, and 97% compliance on the no-invented-figures criterion is a sentence that hides three incidents.

And keep the run-level records, not just the summary. When a number moves, the first question is which cases changed verdict, and that question is answerable in seconds if you kept the per-case results and impossible if you kept an average.

One report, one page, every run

The families are only useful if somebody reads them, which makes the report format part of the design.

One page. Three blocks, one per family. Within each block, the per-stratum breakdown and the total, with the previous run's figures beside them and the delta marked. At the foot, the dataset version, the system version, the model version and the date.

Mark significance honestly. On fifty cases, a two-point movement is one case changing verdict, and presenting that as an improvement is how a team talks itself into a regression. Show the count of cases that changed rather than only the percentage.

Then name the cases. A short list of newly failing case identifiers is more actionable than any chart, and it turns the report into a work queue rather than a status update.

Watch for metric drift and gaming

Metrics change behaviour, including the behaviour of the team measuring them, and this family of systems makes gaming unusually easy.

An outcome metric that rewards citation counts produces answers with more citations, not better grounding. A judge that rewards thoroughness produces longer answers, and Zheng and colleagues documented verbosity bias directly in the MT-Bench work, along with position and self-enhancement biases. A completeness score with no cost counterweight produces a system that answers every question with everything it knows.

Two defences. Always pair a quality metric with its natural counterweight, so completeness sits beside conciseness and cost, and grounding sits beside answer rate. And re-read a sample of high-scoring outputs by hand each month, because gaming looks fine in aggregate and obvious in an individual answer.

The second defence is the one that gets skipped and the one that catches it. Twenty minutes of reading, monthly, is enough.

Chapter summary

Evaluation results belong to three families that answer different questions. Outcome metrics say whether the result was right, mechanically for required facts, citations, prohibitions, refusals and structural validity, and by judgement for faithfulness and completeness, reported per criterion and per stratum rather than as one pass rate. Trajectory metrics say whether the path was sound, which matters because a correct answer reached by a lucky route will not generalise, and AgentLens is the published argument for whole-trajectory assessment covering instruction following, tool use, self-verification, recovery and communication, used for nightly regression detection. System metrics say what a request cost in latency percentiles, input and output tokens, tool and model calls, resolving into cost per successful outcome rather than cost per call. The three trade against each other, since retrieval, retries and larger models buy quality with cost and latency, so a composite score hides the decision and the right report is three numbers side by side with an error-budget framing to make the trade explicit. Each family has a stated blind spot: outcome cannot distinguish luck from soundness, trajectory cannot tell you the user was helped, and system metrics are maximised by a system that refuses everything, which is worth running as a sanity check against your own suite. Aggregate per stratum before totalling, show distributions rather than means for graded scores, use counts for hard requirements, and keep per-case records so a movement can be traced to the cases that changed. And pair every quality metric with a counterweight while reading a sample of high scorers by hand each month, because gaming is invisible in aggregate and obvious in a single answer.

The trajectory family is the one with the least established practice and the most to give. Chapter 5 is Scoring the Path, Not Just the Answer, which is about writing assertions that hold across runs that legitimately differ, without pinning the system to one route through the problem.