LLM Judges Need Their Own Test Sets

LLM Judges Need Their Own Test Sets because a judge is not an oracle, it is a component with a measurable error rate that nobody has measured. Every downstream number inherits that error, and a suite built on an untested judge produces confident figures with unknown meaning.

Key takeaways

• The judge is a system under test. It needs a reference set, a measured agreement figure, a version, and a regression check before it is changed.
• Build the reference set from human annotations on real outputs, including the hard cases. A hundred or so annotated items is enough to measure a judge usefully.
• Measure agreement per criterion, not overall. A judge can be excellent at detecting missing citations and poor at judging completeness, and one number hides that.
• Strong judges reach around 80% agreement with humans, which is roughly what humans reach with each other. Treat that as the ceiling rather than as a disappointment.
• Read the disagreements. Some of them will be the human being wrong, and those cases improve the rubric rather than the judge.

Read this beside Chapter 2, whose rubric the judge applies, and Chapter 7, which covers calibration, bias probes and the error bar in detail. Chapter 9 decides how much authority a judged score may have over a release.

A team reports that faithfulness improved from 71% to 78% after a prompt change. The number came from a judge nobody has ever evaluated.

Ask what the judge's agreement with a human is and the room goes quiet. The seven-point improvement might be real, might be noise, and might be the judge preferring the new answer style. Nothing in the process can distinguish those.

Treat the judge as a component, not as an oracle

The mental shift is small and it changes everything. A judge is a piece of the measurement apparatus, in the same category as a test harness, and apparatus gets calibrated.

Concretely that means four things. It has a version, covering the model, the prompt and the rubric it applies. It has a dataset, being the human-annotated reference set. It has a measurement, being its agreement with those annotations. And it has a change process, so no judge change ships without re-running that measurement.

Without those, a change to the judge is indistinguishable from a change to the system under test. A model upgrade on the judge side can move every score in your suite, and nobody will be able to say whether the product got better.

Anthropic-style guidance and academic practice converge here. AgentLens combines formal verification with generated trajectory reviews precisely so that judged assessments come with explanations rather than bare numbers, which is the same instinct: a judged score should be auditable.

Build the reference set from human annotation

The reference set is a small dataset of outputs with human verdicts, and it is different from the evaluation dataset in Chapter 3.

Chapter 3's dataset holds inputs and required outcomes. This one holds inputs, the system's actual outputs, and a human's verdict on each against the rubric. It is what the judge is tested against.

Build it from the labelling work you already did. When two reviewers scored twenty outputs in Chapter 2 to test the rubric, those became annotated items for free. Continue that habit: every time a human reviews an output for any reason, capture the verdict in the reference set.

Aim for around a hundred annotated items to start, weighted toward the boundary cases where judgement is hard. Easy cases tell you little, because a judge that agrees with a human on obvious passes has demonstrated nothing.

Where two humans disagreed, keep both verdicts and mark the item as contested. Those items are the honest ceiling: a judge that disagrees with one of two disagreeing humans is not necessarily wrong.

Measure per criterion, and report the ceiling

Overall agreement is the number teams quote and the least useful one available.

Measure separately for each criterion the judge applies. In practice, judges are strong on mechanical-adjacent criteria such as whether a citation is present or whether a figure appears, and weaker on holistic ones such as completeness or whether the response suits the audience.

That distribution is directly actionable. Move the strong criteria into mechanical checks where possible, keep the judge for the middle, and treat its verdict on the weakest criteria as advisory rather than as a gate.

Report the human ceiling alongside. Zheng and colleagues found strong judges matching controlled and crowdsourced human preferences at over 80% agreement, and noted that this was about the level of agreement observed between humans themselves. A judge at 78% where your own annotators agree 80% of the time is performing at the limit of the measurement, not failing.

Chapter 7 covers chance-corrected agreement, which is the version of this number that survives scrutiny.

Read the disagreements individually

The aggregate says whether to trust the judge. The disagreements say what to fix.

Pull every case where judge and human differ and sort them into four groups. The judge was wrong on a criterion it should handle, which is a prompt problem. The judge was wrong because the criterion is ambiguous, which is a rubric problem and belongs back in Chapter 2. The human was wrong, which happens more often than people expect, particularly on long outputs. And the case is genuinely contested, which is the irreducible residue.

The second and third groups are where the value is. A judge that keeps failing one criterion usually reveals that the criterion means two different things, and splitting it improves both human and judge agreement at once.

Do this reading with two people. One person's classification of a disagreement is itself a judgement, and the exercise is cheap enough to double up on. Keep the results in the reference set as annotations, so the next person to look at the judge inherits the reasoning rather than repeating it.

Write the judge prompt like a rubric, not like a request

Judge prompts drift toward asking for an overall impression, which is the least reliable thing a judge can produce.

Four properties make a judge prompt behave. It asks for a verdict per criterion, in a structured format, with the criterion named. It provides the anchored definitions from Chapter 2 rather than adjectives. It supplies the evidence the answer was supposed to use, so faithfulness can be checked rather than guessed. And it asks for a short reason per verdict, which improves consistency and gives a human something to read when the verdict is wrong.

Two things to keep out. Do not give the judge the system's own reasoning or chain of thought, because a persuasive rationale changes the verdict without changing the output. And do not tell the judge which version produced the answer, since it invites exactly the preference you are trying to measure.

Keep the prompt in version control with the code, and hash it into the judge version. A judge prompt edited in a console is an unversioned change to your measurement apparatus.

Judges regress, so re-run the measurement

The judge's agreement is not a one-off certification. It is a number that moves.

Re-measure on four triggers. When the judge model version changes, including provider-side updates you did not initiate. When the judge prompt changes. When the rubric changes. And on a schedule, quarterly at least, because provider behaviour drifts without announcements.

Keep a small canary set inside the reference set: fifteen or twenty items whose verdicts are unambiguous and stable. Run those on every evaluation run, not just on re-measurements. If the judge starts disagreeing with a canary, something changed underneath you, and knowing that on the day is worth the handful of extra calls.

The reliability framing from Khanal, Tao and Zhou in March 2026 is a reminder of why this matters for judged trajectory work in particular: they report that model rankings shift at longer horizons, so a judge validated on short outputs may not be valid on long ones. Validate on the length distribution you actually judge.

Cheaper judging without losing the measurement

Judged scoring costs a model call per criterion per case, which becomes real money at CI frequency. Four reductions keep it affordable without abandoning it.

Judge a sample rather than the whole suite on every run. Mechanical checks run on everything; the judged criteria run on a fixed sampled subset, with the full set run nightly. The sample is chosen once and held stable, so day-to-day comparisons are against the same cases.

Batch criteria into one call per case rather than one call per criterion, provided the output stays structured per criterion. That is usually a three-to-one saving with no measurable loss, and it is worth verifying against the reference set rather than assuming.

Use a smaller model for the criteria where the reference set shows it agrees as well as the large one. That is a per-criterion decision with an experiment behind it, not a blanket downgrade.

And cache by content hash. Re-running the suite after a change that affected six cases should not re-judge the other ninety-four, and a hash of input, output and judge version makes that safe.

Know when not to use a judge at all

A judge is the most expensive and least stable scoring mechanism available, so the question of whether it is needed comes before the question of how to test it.

Do not use a judge where a mechanical check works. Presence of a figure, a citation, a required field, a refusal, schema validity, a forbidden phrase. Those are string and structure operations, they are free, they never drift, and they never need a reference set.

Do not use a judge as the sole authority on a gate that blocks releases. Chapter 9 treats mechanical checks and judged scores differently for exactly this reason.

And do not use a judge to compare two of your own systems where a deterministic metric exists. LiveBench's design makes the same choice at benchmark scale, scoring against objective ground truth rather than an LLM judge specifically to avoid judging bias.

Where a judge is the only option, which is most holistic quality assessment, use it with the apparatus in this chapter and publish its error bar. A measured judge with a known error is a legitimate instrument. An unmeasured one is a number generator.

Chapter summary

A judge is a component of the measurement apparatus rather than an oracle, so it needs a version covering model, prompt and rubric, a human-annotated reference set, a measured agreement figure, and a change process that re-measures before shipping. Build the reference set from annotation work already happening, aiming for around a hundred items weighted toward boundary cases, keeping both verdicts where two humans disagreed and marking those items contested. Measure agreement per criterion rather than overall, because judges are strong on mechanical-adjacent criteria and weak on holistic ones, and that distribution tells you which criteria to automate, which to judge and which to treat as advisory. Report the human ceiling beside the judge's number, since strong judges reach around 80% agreement with humans and humans reach about the same with each other, so a judge near that level is at the limit of the measurement. Read every disagreement and sort it into judge error, rubric ambiguity, human error or genuine contest, because the middle two improve the rubric rather than the judge. Write the judge prompt as a structured per-criterion rubric with anchored definitions, the supporting evidence and a short reason per verdict, while withholding the system's own reasoning and the identity of the version being judged. Re-measure on model change, prompt change, rubric change and quarterly, with a stable canary subset run on every evaluation. And prefer no judge at all where a mechanical check suffices, since LiveBench deliberately scores against objective ground truth to avoid judging bias.

Agreement is only half of the judge's profile. Chapter 7 is Calibration, Bias, and Disagreement With Humans, which covers chance-corrected agreement, the specific biases that have been measured in judges, and how to publish an error bar that stops a two-point difference being reported as an improvement.