Benchmarks Saturate; Your Suite Should Not

Benchmarks Saturate; Your Suite Should Not is about the difference between measuring the field and measuring your product. Public scores answer a question you were not asking, and they answer it less well each year.

Key takeaways

• Saturation is a discrimination failure, not a difficulty failure. When every serious model clusters near the top, the benchmark can no longer rank them, whatever the absolute number says.
• Contamination is the second failure. Al-Lawati and colleagues argue that benchmark datasets are compromised by appearing in pretraining corpora, so a score partly measures memorisation.
• Contamination-limited designs exist and are instructive. LiveBench refreshes questions monthly from recent sources and scores against objective ground truth, and reports top models below 70%.
• Your suite is contamination-resistant by construction, because your production traffic is not in anyone's training data, and it stays discriminative if you feed it failures.
• The health check for a suite is separation, not score. If a change you know to be bad does not move it, the suite has saturated too.

Read this beside Chapter 3, which built the dataset, and Chapter 11, which measures context stages rather than answers. Chapter 9 turns a discriminative suite into a gate.

A vendor sends a comparison table. Their model scores two points higher than yours on four public benchmarks, and the table is presented as a reason to switch.

Nothing in that table tells you whether their model answers your customers' questions better, and nothing in it can, because your customers' questions are not in it.

Saturation is about separation, not difficulty

The word saturation gets used loosely. Precisely, a benchmark is saturated when the differences between the systems you care about are smaller than the benchmark's own noise.

That can happen while the absolute scores still look impressive and while the benchmark is still hard for weaker models. What has been lost is resolution at the top, which is exactly where a purchasing or upgrade decision sits.

Two consequences follow for anyone reading a leaderboard. Small differences between frontier systems are usually not meaningful, and a table of near-identical numbers should be read as "these are indistinguishable on this measure" rather than as a ranking. And improvements reported against a saturated benchmark say little about behaviour on the hard tail, which Chapter 1 argued is where products fail.

The field's own response is instructive rather than merely academic. LiveBench, revised in April 2025 and published as an ICLR Spotlight, was designed to stay discriminative by refreshing its questions monthly from recent sources, and it reported top models scoring below 70%. Difficulty was restored deliberately.

Contamination is the second failure, and it is quieter

Saturation is visible. Contamination is not.

Al-Lawati and colleagues, writing in May 2026, argue that benchmark datasets are contaminated by appearing in pretraining corpora and propose contamination-resistant datasets, ones that are unlearnable yet still support inference, as the response. The problem they are addressing is that a model may have seen the questions.

The effect on a score is not a uniform inflation, which would at least be correctable. It is uneven: heavily circulated items are more likely to be memorised, so the benchmark's easier and more famous questions inflate more than its obscure ones, which distorts the shape of the result as well as its level.

For a practitioner the implication is simple and slightly bleak. A high score on a well-known public benchmark is weak evidence, and it becomes weaker the longer the benchmark has existed. Treat any benchmark older than a couple of years as partly a memorisation test.

The designs that resist this are worth knowing because they show the mechanism: frequent refresh from recent material, objective ground-truth scoring, and questions generated after the models' training cutoffs.

What public benchmarks are still good for

None of this makes them worthless. It makes them the wrong instrument for the release decision, which is a narrower claim.

They are good for tracking the field. Whether long-horizon capability is improving, whether a new model family is competitive at all, whether a class of task has become tractable. That is genuinely useful for planning.

They are good for screening. A model that performs badly on a relevant public benchmark is unlikely to perform well on your suite, so benchmarks are a cheap filter before an expensive evaluation.

They are good as shared vocabulary. When a vendor and a buyer discuss capability, a common reference point is better than two sets of private numbers, provided both sides agree on which version was run and when.

What they cannot do is tell you whether to ship your change, because they do not contain your corpus, your prompts, your tools, your users' phrasing or your definition of done.

Your suite is contamination-resistant by construction

The good news is structural. Production traffic from your own product is not in anyone else's training data, your rubric encodes your product's requirements, and your required-outcome labels reflect your corpus.

That gives your suite three properties public benchmarks are fighting to recover. Freshness, because you add cases continuously from real failures. Specificity, because the questions are the ones your users actually ask. And privacy, because the set is not published, so no model is trained on it.

Two obligations come with those properties. Keep the held-out split sealed, as Chapter 3 insists, because the contamination risk that remains is you fitting to your own set. And be careful about what you send where: a suite pasted into a third-party tool, or used as few-shot examples in a prompt, has partially published itself.

There is one more discipline worth adopting from LiveBench's design. Refresh deliberately, retiring cases whose documents no longer exist and adding recent ones, so the set tracks the product rather than freezing at the moment it was built.

Keep your own suite discriminative on purpose

A private suite saturates too, and for a good reason: you fix the failures it finds.

After two quarters of honest work, most cases pass, the score sits high, and every change looks the same. At that point the suite has stopped doing its job, and the response is not to celebrate.

Three habits maintain difficulty. Add every production failure as a case, permanently, which is the main mechanism and costs nothing beyond the labelling. Retire nothing that still fails, since a case that keeps failing is your most valuable case. And periodically add a batch of harder cases deliberately: multi-hop questions, cases requiring a refusal, cases where two sources conflict.

Track the pass rate per stratum rather than overall, since saturation usually arrives one stratum at a time and the common cases saturate first. Then run the separation check. Take a change you are confident is bad, such as removing retrieval or halving the context budget, and confirm the suite catches it. A suite that does not flinch at deliberate damage will not catch accidental damage either.

AgentLens is a useful reference here, since its authors describe using trajectory reviews for nightly regression detection rather than for a one-off ranking. A suite is a monitor, and monitors need to be able to detect things.

Borrowing a public benchmark properly

There is a middle path between ignoring public benchmarks and deferring to them, and it is worth knowing because it is cheap.

Borrow the format rather than the score. If a public benchmark tests multi-hop reasoning over documents and your product does that, take its question shapes and build equivalent cases from your own corpus. You get a well-designed task definition without inheriting either the contamination or the irrelevance.

Borrow the scoring method too, where it is objective. LiveBench's decision to score against ground-truth values rather than an LLM judge is a design choice you can copy per criterion, and every criterion you move from judged to objective removes an error term from Chapter 7.

Run the public benchmark itself only when you are choosing a model, and read it as a screen rather than as a decision. A model that fails a relevant public test will not surprise you on your own suite. A model that passes might.

And record which public benchmarks you consulted, with the date and version. Six months later, when a vendor cites a different number, the disagreement is usually about which version was run.

Score bands, not points

One habit makes both public and private numbers less misleading.

Report bands rather than exact figures where the error bar from Chapter 7 is wider than the difference. "Between 72 and 80" is honest and unexciting, and "76%" invites comparison at a resolution the measurement does not support.

For vendor comparisons, insist on the same discipline. Ask which benchmark version was used, when it was refreshed, what the sample size was, and whether the difference exceeds the interval. Most comparison tables fail the fourth question and are quietly withdrawn from the conversation.

For internal reporting, band the suite too. Chapter 7's rule applies: a movement inside the interval is no measurable change, and reporting it as progress is how a roadmap gets steered by noise.

Bands also make the honest good news visible. When a change moves the score outside the band, everybody knows it is real, which is worth more than a series of unfalsifiable small wins.

The suite as an asset with a stated shelf life

One last framing helps when arguing for the investment. A suite is an asset, and like any asset it has maintenance and a useful life.

Its value rises with age, because it accumulates cases nobody would have thought to write, drawn from failures that actually happened. That is the opposite of most test suites, which decay.

Its maintenance is small and non-optional: labelling new cases, retiring dead ones, re-measuring the judge, and refreshing the harder strata. Half a day a month for a product-sized suite.

And its shelf life depends entirely on that maintenance. A suite left alone for two quarters is measuring a product that no longer exists, and its green result is worse than no result because it is trusted.

Say all of that when the suite is proposed, so the maintenance is funded with the build rather than discovered afterwards.

Chapter summary

A benchmark saturates when the differences between the systems you care about fall inside its own noise, which is a loss of resolution at the top rather than a lack of difficulty, so near-identical leaderboard rows should be read as indistinguishable rather than ranked. Contamination is the quieter failure, with benchmark questions appearing in pretraining corpora, and it inflates unevenly because famous items are memorised more than obscure ones, which distorts the shape of a result as well as its level. The designs that resist both, exemplified by LiveBench's monthly refresh from recent sources and objective ground-truth scoring with top models reported below 70%, show the mechanism worth borrowing. Public benchmarks remain useful for tracking the field, for cheap screening before expensive evaluation, and as shared vocabulary with vendors, and they remain unable to tell you whether to ship, since they contain none of your corpus, prompts, tools or users. Your own suite is contamination-resistant by construction, being fresh, specific and unpublished, with the residual risks being your own fitting to it and any leakage through third-party tools or few-shot prompts. Private suites saturate too as you fix what they find, so maintain difficulty by adding every production failure permanently, retiring nothing that still fails, and periodically adding harder cases, then verify with a separation check that deliberately damages the system and confirms the suite notices. And report bands rather than points wherever the interval exceeds the difference, applying the same discipline to vendor comparisons, which makes the real improvements legible when they arrive.

A suite that separates good changes from bad ones is worth wiring into the path where that distinction matters. Chapter 9 is Wiring Evals Into CI as a Release Gate, which covers thresholds that do not produce daily false alarms, regression rules that survive flakiness, and the question every evaluation programme eventually turns on, which is who may override a red run.