Defining Done for an Open-Ended Output

Defining Done for an Open-Ended Output is the step that everything else in this book rests on. It is also the step teams skip, because it is a conversation about wording rather than an engineering task, and because it exposes that two people who both said the answer was good meant different things.

Key takeaways

• Write criteria, not expected answers. There is no single correct output for an open-ended request, and scoring against one reference punishes correct answers that were phrased differently.
• Split the rubric in two. Hard requirements are binary and mechanically checkable. Quality dimensions are graded and need a judge or a human.
• Test the rubric before you test the system. Two reviewers, twenty real outputs, scored independently.
• Disagreement is a defect in the rubric until proven otherwise. Rewrite the criterion, do not average the scores.
• Most graded scales should have three points, not five. Reviewers cannot reliably distinguish five levels of a fuzzy quality, and the extra resolution is noise.

Read this beside Chapter 1, which argued that nothing can be measured until done is defined, and Chapter 3, which uses this rubric to label the first dataset. Chapter 6 tests a judge against the labels this chapter makes possible.

Two engineers review the same twenty support answers. One rates sixteen acceptable, the other rates eleven.

They are not disagreeing about the system. They are disagreeing about whether an answer that is correct, complete and slightly brusque counts as acceptable, and nothing written down says.

There is no reference answer, so stop looking for one

The instinct from classical testing is to write the expected output and compare. It fails immediately here, for a reason worth stating precisely.

A good answer to a support question can be two sentences or six, can lead with the answer or with the caveat, can cite one source or three. All of those may be acceptable, and a comparison against one reference marks most of them wrong.

Worse, similarity scoring rewards the wrong property. An answer that closely matches the reference wording while omitting a required caveat scores well, and an answer that is correct and differently phrased scores badly. The measurement is now actively misleading.

So the unit of evaluation is a criterion, not a reference. "Names the fourteen-day window" is checkable against any phrasing. "Matches the reference answer" is not a statement about correctness at all.

Split the rubric into must and graded

Every workable rubric has two parts, and conflating them is the most common design error.

Hard requirements are binary, checkable, and usually mechanical. Does the answer cite at least one supplied source. Does it avoid stating a figure that is not in the evidence. Does it answer the question that was asked rather than an adjacent one. Does it decline when the evidence does not support an answer. Does it stay inside the audience's permissions.

Quality dimensions are graded and need judgement. Completeness. Clarity. Tone. Whether the structure suits the question.

The split matters for three reasons. Hard requirements can be automated cheaply, and they catch the failures that actually generate complaints. They are also the ones a release gate should block on, because a graded score moving by a tenth of a point is not a reason to stop a deploy. And separating them stops a beautifully written answer from compensating for a missing citation in an averaged total.

Write the hard requirements first, and expect five to eight of them. If you have twenty, most are quality dimensions in disguise.

Three points, not five

Graded scales invite five points and then produce noise.

Ask two reviewers to distinguish a four from a five on clarity and they will disagree about half the time, because the difference is not a property of the output. Ask them to sort answers into unacceptable, acceptable and good, and they will agree far more often, because those categories correspond to decisions.

Anchor each point with a description rather than a number. Unacceptable means a user would complain or be misled. Acceptable means a user gets what they needed. Good means a user gets what they needed and something they did not know to ask for.

Anchoring is what makes the scale portable. It survives new reviewers joining, and it gives a judge in Chapter 6 something concrete to apply rather than an adjective.

Test the rubric on twenty outputs

The rubric is a component and it needs its own test before anything downstream trusts it.

Take twenty real outputs, drawn from actual traffic rather than from the demo set. Have two people score them independently, without discussion, using the draft rubric. Then compare. Count the rows where they disagree, and read every disagreement out loud.

Each disagreement resolves into one of three causes. The criterion is ambiguous, which is a rubric defect and the most common outcome. The reviewers hold different background assumptions about the product, which is a specification gap worth surfacing. Or the output genuinely sits on a boundary, which is the only acceptable residue.

Rewrite the ambiguous criteria and repeat. Two rounds usually gets a rubric from chaotic to usable, and the second round takes an hour.

Two reviewers is a minimum rather than a target. Three is better where the stakes justify it, because it lets you see whether one reviewer is systematically stricter, which is a fact about the person rather than about the outputs.

Agreement is a number, so record it

"They mostly agreed" is not a measurement, and the number matters later.

Record raw agreement first, meaning the share of items where both reviewers gave the same verdict. Then adjust for chance, because on a binary criterion two reviewers guessing randomly agree half the time. Chance-corrected agreement, in the form of Cohen's kappa or a similar statistic, is the honest figure and Chapter 7 develops it properly for judges.

That figure sets a ceiling you cannot exceed. If two humans applying your rubric agree on 80% of items, no judge can be usefully measured above that, because the labels themselves carry that much disagreement. Zheng and colleagues framed exactly this comparison when reporting that strong judges reached over 80% agreement with humans, which was around the level humans reached with each other.

So publish the human agreement number alongside every eval score your team reports. It is the error bar on everything downstream, and teams that omit it end up arguing about a two-point difference that sits well inside the noise.

Different tasks need different definitions of done

One rubric across a whole product is a false economy, because tasks fail differently.

A factual lookup is dominated by hard requirements: correct figure, correct scope, cited source. Graded quality barely matters.

A summarisation task inverts that. Faithfulness is the hard requirement, and after that almost everything is graded.

An agentic task adds a whole dimension, because the answer can be right while the path was wrong. AgentLens, published in July 2026, makes that argument for coding agents, assessing instruction following, tool use, self-verification and recovery rather than reducing a run to pass or fail. Chapter 5 covers the path in detail; the point here is that its criteria belong in the rubric from the start.

And a refusal task, where the correct behaviour is to decline, needs its own criteria entirely. The hard requirement is that the system declined, and the graded dimension is whether the decline was useful, meaning it explained the limit and offered the next step.

Write one rubric per task family, and expect three or four families in a typical product.

What done means when the corpus cannot answer

The single most valuable criterion in most rubrics is the one teams add last: whether the system correctly said it did not know.

Every corpus has boundaries, so every product receives questions past them. The default behaviour of a language system is to produce a fluent answer regardless, and the failure looks exactly like a success on any metric that does not test for it.

Write it as a hard requirement with two halves. When the evidence does not support an answer, the system says so. And when it says so, it says what it would need, or where the user should go instead.

Then populate the dataset with cases that trigger it, deliberately, in Chapter 3's stratified sample. A suite with no unanswerable questions is a suite that will never catch the most damaging behaviour in the system.

Who writes it, and who signs it off

A rubric written by engineers alone encodes what is easy to check. A rubric written by the business alone encodes what is easy to want. Both fail in production.

Draft it with three people in the room. Someone who understands the model's behaviour, someone who owns the customer relationship, and someone who will handle the complaint when it goes wrong. The third person contributes the criteria nobody else thinks of, because they have read what users write when they are angry.

Then have it signed off by whoever carries the consequences of a bad answer. That signature is what gives the gate in Chapter 9 its authority: a red run is not an engineer's opinion, it is a violation of criteria the business agreed.

There is a sequencing trap worth avoiding. Do not draft the rubric from the system's current behaviour, because a criterion written to match what the system already does guarantees a passing score and measures nothing. Write it from what a good answer requires, then find out how far away you are.

Expect the first honest run against a real rubric to score badly. That is the measurement working.

The rubric is a document with a version

Treat it as a versioned artifact, in the repository, alongside the evaluation code.

Version it, because a score is meaningless without knowing which rubric produced it, and a rubric that changes silently makes historical comparison invalid.

Date every change and record why. Six months later the question of when a criterion was added, and after which incident, is genuinely useful.

Keep the examples with it. Each criterion should carry one passing and one failing real example, which is worth more to a new reviewer than any amount of definition, and which Chapter 6 reuses directly in the judge prompt.

And review it quarterly against fresh complaints. The rubric is a model of what users consider good, and that model drifts.

Chapter summary

Open-ended output has no single correct form, so done is written as criteria rather than as an expected answer, since similarity to a reference rewards phrasing and punishes correct answers that were worded differently. Split the rubric into hard requirements that are binary and mechanically checkable, of which five to eight is typical, and graded quality dimensions that need judgement, because only the first group should ever block a release and mixing them lets good prose hide a missing citation. Use three anchored points rather than five, describing each in terms of what a user experiences, since reviewers cannot reliably separate five levels of a fuzzy quality. Test the rubric before testing the system, with two reviewers scoring twenty real outputs independently, and treat every disagreement as a rubric defect until it is shown to be a genuine boundary case. Record agreement as a number, corrected for chance, and publish it alongside every eval score, because human agreement is the ceiling on what any judge can be measured against, which is the comparison the MT-Bench work drew when reporting judges at over 80% agreement. Write one rubric per task family, since lookups, summaries, agentic runs and refusals fail differently and the path-level criteria that AgentLens argues for belong in the rubric from the start. Include the criterion teams add last and need most, which is whether the system correctly declined when the corpus could not answer, and populate the dataset with cases that trigger it. And keep the rubric versioned in the repository with dated changes, worked examples per criterion, and a quarterly review against fresh complaints.

With done defined, the next question is what to apply it to. Chapter 3 is Building the First Dataset From Real Traffic, which covers sampling from what production already sends, stratifying so the tails are represented, labelling at a cost you can sustain, and keeping a held-out split that stays uncontaminated.