Why Demos Pass and Products Fail

Why Demos Pass and Products Fail is a question about sampling rather than about quality. The demo worked because a person selected its inputs, and production is the same system meeting inputs selected by everybody else, including people actively trying to break it.

Key takeaways

• A demo samples a distribution chosen by someone who wanted it to work. Production samples the real one, whose tails contain the malformed, the ambiguous, the adversarial and the unanswerable.
• The failure is quiet. A probabilistic system does not error on the hard cases, it produces something fluent and wrong, which no exception handler catches and no uptime graph shows.
• Public benchmark scores do not close the gap. They measure a model on somebody else's distribution, and the top scores on contamination-limited benchmarks are still well short of solved.
• Evaluation is the apparatus for sampling the production distribution before your users do, and it is a build-system problem rather than a research problem.
• Nothing works until done is defined. An open-ended output with no written criteria cannot be scored by a person, a script or a judge.

Read this beside Chapter 2, which defines done, and Chapter 3, which builds the first dataset from real traffic. Chapter 9 is where the resulting suite acquires the authority to stop a release.

The demo is on a Thursday. Eleven questions, all of them asked before, and the assistant answers all eleven well. Somebody asks a twelfth from the room and it goes fine too. The room agrees it is ready.

Six weeks later a customer receives a confident answer about a policy that was withdrawn last year. Nothing errored, nothing paged, and the first anyone hears of it is a complaint routed through support.

The demo is a biased sample and everyone knows it

Nobody chooses demo inputs adversarially. They choose inputs that show the system doing what it is meant to do, which is the correct thing to do in a demo and a catastrophic basis for a release decision.

Three biases operate at once. The questions are the ones the builder already knows work, because the ones that failed were fixed or quietly dropped during development. The phrasing is clean, because the builder writes like someone who understands the system. And the context is favourable, since demos run on curated accounts with tidy data rather than on the account with eleven years of history and three merged organisations.

That produces a distribution that is narrow, central and unrepresentative. The system is being measured on the middle of the range while it will be paid on the whole range.

The remedy is not a better demo. It is a sample of the real distribution, which Chapter 3 builds from traffic that already exists.

What lives in the tails

The tails are where the money and the embarrassment live, and they are populated by five recognisable groups.

Malformed and empty input. A blank submission, a pasted spreadsheet, a message that is only an emoji, a file that failed to upload.

Compound requests. Three questions in one message, with an implicit dependency between them and no punctuation to separate them.

Out-of-corpus questions. Perfectly reasonable requests whose answer is not in your knowledge base at all, where the correct behaviour is to say so and where the default behaviour is to answer anyway.

Adversarial input. People testing the boundaries out of curiosity, and people testing them for gain. Both belong in the suite, because a system that behaves well only for cooperative users is not ready for a public surface.

Rare but critical cases. The regulated question, the account with an unusual configuration, the request that triggers a legal obligation. Individually rare, and collectively the reason a suite needs stratified sampling rather than random sampling.

Fluent wrongness is the failure mode that matters

Classical software fails loudly. It throws, it times out, it returns a status code, and the monitoring you already have notices.

A language system fails fluently. It produces well-formed, confident, plausible output that is wrong, and every layer of conventional monitoring reports success. The request returned 200. Latency was normal. The token count was unremarkable.

That is why evaluation is not an optional maturity step for this class of system. It is the only detector available for the primary failure mode, and its absence means that quality is measured by complaint volume with a lag of weeks.

DORA's current metrics, updated in January 2026, are still the right frame for delivery health, including change fail rate and failed deployment recovery time. They do not detect fluent wrongness, because from their point of view nothing failed. Your evaluation suite is the instrument that does, and Chapter 4 keeps the two families of measurement separate.

Public benchmarks do not answer your question

The reflex when asked for evidence is to reach for a benchmark score. It is the wrong instrument twice over.

A benchmark measures a model on a distribution somebody else chose, which is the exact problem the demo had, at a larger scale. Your system is not a model: it is a model plus a corpus, prompts, tools, retrieval and a product surface, and the benchmark measures none of those.

Contamination compounds it. Al-Lawati and colleagues argued in May 2026 that benchmark datasets are contaminated by appearing in pretraining corpora, and proposed contamination-resistant datasets as the response. A score on a contaminated benchmark is a measurement of memorisation as much as capability.

The useful anchor is what a contamination-limited benchmark shows. LiveBench, published in June 2024 and revised in April 2025, refreshes questions monthly from recent sources and scores against objective ground truth rather than a judge, and it reports top models scoring below 70%. That is the honest state of the art on hard, fresh questions, and it is a reminder that headline saturation numbers describe older, easier sets.

Chapter 8 develops this. The short version: benchmarks tell you about the field, and your suite tells you about your product.

Evaluation is a build system, not a research project

The framing that gets an evaluation programme funded is not scientific. It is infrastructural.

A build system has versioned inputs, deterministic execution, reproducible outputs and the authority to fail. An evaluation suite needs the same four properties: a versioned dataset, a defined scoring procedure, results you can compare across runs, and a threshold that blocks a deploy.

That framing also sets the standard for the judge. If a judge is part of the scoring procedure, it is part of the build system, and it needs its own tests. Zheng and colleagues, in the MT-Bench work published in June 2023, found strong judges matching human preferences at over 80% agreement, which is roughly the agreement humans reach with each other, while also documenting position bias, verbosity bias and self-enhancement bias. A component with that profile is usable and cannot be trusted untested, which is Chapters 6 and 7.

And the suite has to cover the path, not only the answer. AgentLens, published in July 2026, makes the argument for trajectory-level assessment of coding agents, evaluating instruction following, tool use, self-verification and recovery rather than reducing a run to pass or fail, and reports being used for nightly regression detection. Chapter 5 takes that up.

The objection, and why it does not hold

The argument against all of this is familiar and worth answering directly. Evaluation is expensive, the field moves quickly, and a suite written now will be obsolete by the time it is finished.

Two parts of that are true. Labelling costs real time, and any suite tied to a particular model's quirks does age.

The conclusion does not follow. What ages is the scoring of specific behaviours. What does not age is the dataset: a stratified sample of your own traffic with agreed correct outcomes is as valid against next year's model as against this one, and it gets more valuable as it grows because it accumulates the failures you have already seen.

There is a cost argument too, and it is stronger than the quality argument in most rooms. Without a suite, every release is decided by an argument between people with different intuitions, and every regression is found by a customer. The suite converts both into a number produced before the deploy, which is cheaper than either.

Start small enough that the objection cannot land. Fifty labelled cases and one gate is a week of work, and it is enough to catch the class of regression that currently reaches users.

Start with done, or measure nothing

Every evaluation programme that stalls, stalls in the same place. Somebody asks what score would be good enough, and the room discovers it has never agreed what a correct answer is.

Do that first, before writing a single test. Chapter 2 is the procedure: take twenty real outputs, have two people rate them independently against draft criteria, measure where they disagree, and rewrite the criteria until the disagreement is about the output rather than about the rubric.

It is unglamorous and it is the gate on everything else. A dataset labelled against a vague rubric is noise. A judge calibrated against noisy labels is noise with a confidence score. A CI gate wired to a noisy judge blocks releases at random and gets switched off within a month, which is the most common way an evaluation programme dies.

What this book builds, in order

The order is deliberate, and each step depends on the one before.

Define done, in criteria two reviewers agree on. Build a first dataset from real traffic, stratified across the tails, with a held-out split you do not contaminate. Separate metrics into outcome, trajectory and system, because they answer different questions. Score the path as well as the answer. Test the judge against human annotation, and publish its error bar. Grow your own suite as public benchmarks saturate. Wire the result into CI with thresholds, regression rules and a named override. Extend it online with shadow traffic and canaries. Trace it with the OpenTelemetry GenAI conventions and know where they stop. And run incident response for the class of error that reads as confident.

None of it requires research. All of it requires the discipline to measure the distribution you will actually be judged on.

Chapter summary

A demo samples a distribution chosen by someone who wanted the system to work, with questions already known to succeed, clean phrasing and curated context, so it measures the middle of a range while the product is paid on the whole range. The tails hold malformed input, compound requests, out-of-corpus questions, adversarial probing and rare critical cases, and they are where both the money and the embarrassment sit. The failure mode is fluent wrongness rather than an exception, so conventional monitoring reports success throughout and DORA's delivery metrics, correct as they are, cannot see it. Public benchmarks do not close the gap, because they measure a model on someone else's distribution while your system is a model plus corpus, prompts, tools and retrieval, and contamination makes many scores partly a measurement of memorisation, with LiveBench's contamination-limited design reporting top models below 70% as the honest anchor. Treat evaluation as a build system with versioned inputs, defined scoring, comparable results and the authority to fail, which also means the judge is a component under test, given that strong judges reach over 80% agreement with humans while showing position, verbosity and self-enhancement biases. Cover the path as well as the answer, following the trajectory-level argument AgentLens makes for coding agents. And define done before writing a single test, because a vague rubric produces noisy labels, a noisy judge and a gate that gets switched off within a month.

Everything therefore begins with a definition. Chapter 2 is Defining Done for an Open-Ended Output, which is the procedure for turning "a good answer" into criteria two reviewers apply independently and agree on, and for finding out how much of your current disagreement is about the output rather than about the words in the rubric.