Calibration, Bias, and Disagreement With Humans

Calibration, Bias, and Disagreement With Humans is the chapter that turns a judge from a plausible tool into a measuring instrument. It is also the chapter that will tell you some of your reported improvements were never real.

Key takeaways

• Raw agreement overstates a judge. On a binary criterion, two parties agreeing at random hit 50%, so chance-corrected agreement is the honest figure.
• Probe for the three biases that have been measured in judges: position, verbosity and self-enhancement. Each probe is a controlled pair of inputs and takes an afternoon to build.
• Publish an error bar with every score. On fifty cases, a two-point movement is one case, and one case is not a result.
• Calibrate the threshold, not just the judge. A judge that is systematically strict is usable once you know by how much.
• Disagreement with humans is data about the rubric as often as it is data about the judge, which is why the two are developed together.

Read this beside Chapter 6, which built the reference set this chapter measures against, and Chapter 9, which decides what a score with an error bar is allowed to block. Chapter 4's separation of families still applies: these numbers are outcome measurements, not delivery metrics.

A quarterly review presents faithfulness rising from 74% to 78%. The suite has fifty cases. The difference is two cases.

Nobody in the room can say whether two cases is more than the noise, because no error bar was ever computed. The improvement is reported, the roadmap is set by it, and the whole thing may be a coin landing the same way twice.

Raw agreement flatters everyone

Start with the arithmetic that makes raw agreement misleading.

On a binary criterion where both parties say pass most of the time, agreement is high before anybody has done anything. If a judge and a human both pass 90% of cases independently, they agree on roughly 82% of items by chance alone. A reported 85% agreement is then barely better than guessing with the same prior.

Chance-corrected agreement fixes that by subtracting the agreement expected at random and rescaling. Cohen's kappa is the standard form for two raters on a categorical verdict, and it is the number to report.

Read it conservatively. Values near zero mean nothing useful is happening. The judge is reproducing the base rate. Middling values mean it is useful and imperfect. High values mean it is close to the human ceiling, which Chapter 6 established is itself well below one.

Two practical notes. Compute kappa per criterion, since the base rates differ wildly between criteria. And report the raw agreement alongside it, because kappa is unfamiliar to most stakeholders and the pair together is more honest than either alone.

Probe for position bias

The first measured bias is order dependence, and it is trivially testable.

Zheng and colleagues documented position bias in their MT-Bench work, where a judge's decision is influenced by the order in which candidates are presented. If your judge ever compares two answers, this applies directly.

The probe is a controlled pair. Take fifty comparisons, run each twice with the two answers swapped, and count how often the verdict changes. A judge with no position bias gives the same winner both times, except on genuine ties.

Report the flip rate. It is a single number and it is immediately actionable: if the rate is material, either average over both orders in production scoring, which doubles the cost, or stop using pairwise comparison and score each answer independently against the rubric.

Independent scoring is usually the better answer for a release suite. It is cheaper, it is stable when one side changes, and it produces per-criterion verdicts rather than a preference.

Probe for verbosity bias

The second measured bias is a preference for longer answers, and it distorts exactly the metrics teams care about.

The probe: take fifty answers the judge rated acceptable, produce a padded version of each that adds words without adding information, and re-score. Any systematic rise in the score is verbosity bias, measured on your own judge and your own rubric.

The consequence is worse than a wrong number. A judge with verbosity bias creates an optimisation pressure: teams tune prompts until the score rises, and the way the score rises is longer answers, which users dislike and which cost more to produce.

Two counters. Include conciseness as an explicit criterion so length is scored rather than rewarded implicitly. And give the judge the token count, since a stated length lets it apply the criterion rather than sensing thoroughness.

Then verify the counter worked by re-running the probe. A mitigation that was never measured is a hope.

Probe for self-enhancement bias

The third measured bias is a preference for output from the judge's own model family, which Zheng and colleagues also documented.

It matters because the cheapest architecture is to use the same provider for generation and judging, and that is precisely the arrangement in which this bias operates.

The probe needs answers from at least two model families, scored by your judge, with human verdicts on the same items. Compare the judge's error rate by generator. A judge that is systematically more generous to answers from its own family will show it as a gap between those error rates.

The mitigation is a different judge model from the generator, where the budget allows it. Where it does not, the fallback is to keep the judged criteria narrow and mechanical-adjacent, and to lean on the deterministic checks from Chapter 4 for anything that decides a release.

This is one place where the public benchmark world has already voted with its feet. LiveBench scores against objective ground-truth values rather than an LLM judge, specifically to avoid judging bias, and that choice is available to you wherever your criteria can be made objective.

Human annotation drifts too, and needs its own controls

The reference set is the standard everything is measured against, which makes the humans producing it part of the apparatus.

Three effects show up on any annotation effort that runs longer than a fortnight. Annotators get stricter as they see more output, because their sense of the range shifts. Different annotators hold different thresholds, so a set labelled mostly by one person carries that person's calibration. And fatigue on long outputs produces systematically more lenient verdicts late in a session.

Three controls handle most of it. Re-label a fixed subset every month with the same people and compare, which measures drift directly. Have each annotator label a shared overlap set so per-person offsets are visible. And cap session length, because a two-hour labelling session is measurably worse in its second hour.

None of this is exotic and all of it is skipped. The point is not perfection in the labels. It is knowing the size of their error, since that error is the floor under every judge measurement in Chapter 6.

Publish an error bar, and refuse to read inside it

Two sources of uncertainty sit under every judged score, and both are computable.

Sampling uncertainty comes from having fifty or a hundred cases rather than all possible ones. On fifty cases at around 75%, the interval is wide, and any movement of a few points sits inside it.

Judge uncertainty comes from the disagreement measured in Chapter 6. If the judge agrees with humans on four cases in five, its reported score carries that error regardless of sample size.

Combine them into one published interval and put it on every chart. Then adopt the rule that matters: a movement inside the interval is not reported as an improvement or a regression. It is reported as no measurable change.

That rule is unpopular for a fortnight and then becomes the most valuable convention the team has, because it stops the roadmap being steered by noise, and it makes the real improvements legible when they happen.

Google's error budget framing pairs well with it. State the quality level you intend to hold, measure against it with the interval visible, and act on the gap rather than on the movement.

Two more biases worth probing on your own data

Three biases are documented in the literature. Two more show up repeatedly in practice and are worth a probe of your own, because they are properties of your rubric as much as of the judge.

Formatting bias. Take the same content and present it once as prose and once as a structured list, then re-score. Judges frequently prefer the structured version, which matters because your product may have a house style that the judge is quietly penalising.

Confidence bias. Take an answer with a hedge, such as a stated uncertainty or a caveat about the source, and compare it with the same answer stated flatly. A judge that rewards the flat version is training your system out of the honesty you asked for in Chapter 2, which is the opposite of what a grounded product needs.

Both probes are cheap: fifty pairs, one re-score, one difference. Run them once when the judge is introduced and again whenever the judge model changes.

Where a bias is found and cannot be removed, name it in the report. A score with a known lean is usable. A score with an unknown lean is the thing this chapter exists to prevent.

Calibrate the threshold rather than chasing agreement

A judge does not have to agree with humans to be useful. It has to be predictable.

If a judge is systematically stricter than your annotators by a known margin, that is a calibration offset, and the score remains a valid comparison across runs. What is fatal is inconsistency: a judge whose strictness varies by topic, length or day.

So measure stability as well as agreement. Run the same items through the judge several times and count verdict changes, since non-zero temperature and provider variability both produce them. Then run the same items a month later and compare, which is how provider drift is caught.

Where a judge is stable and offset, set thresholds against the judge's own scale rather than against an absolute idea of quality. The gate in Chapter 9 cares about movement against a baseline, and a consistently strict judge serves that perfectly well.

AgentLens' insistence that trajectory reviews produce readable explanations is the complementary discipline. A judge that explains itself lets a human check the offset rather than guessing at it.

Chapter summary

Raw agreement flatters a judge, because two parties with the same skewed base rate agree most of the time by chance, so report chance-corrected agreement per criterion, with the raw figure beside it for legibility. Probe for the three biases documented in the MT-Bench work. Position bias is measured by swapping candidate order and counting verdict flips, and the durable fix is independent per-criterion scoring rather than pairwise comparison. Verbosity bias is measured by padding answers without adding information and re-scoring, and it is dangerous because it creates an optimisation pressure toward longer answers that users dislike, countered by scoring conciseness explicitly and supplying the token count. Self-enhancement bias is measured by comparing the judge's error rate across generator families, and mitigated by using a different judge model or by narrowing judged criteria, which is the choice LiveBench makes at benchmark scale by scoring against objective ground truth. Publish an error bar combining sampling uncertainty and measured judge disagreement, then refuse to report any movement inside it as an improvement, which is unpopular briefly and then becomes the convention that stops noise steering the roadmap. And calibrate rather than chase agreement, since a judge that is stable and systematically strict is a usable instrument, while one whose strictness varies by topic or over time is not, which makes stability testing across repeats and across months as important as the agreement figure itself.

The judge is now an instrument with a known error. The next question is what to measure with it, given that the public numbers everyone quotes have stopped discriminating. Chapter 8 is Benchmarks Saturate; Your Suite Should Not, which is about why external scores converge, what contamination does to them, and how to grow a suite that stays informative as your product improves.