Building the First Dataset From Real Traffic

Building the First Dataset From Real Traffic is where evaluation stops being a plan. The dataset is the asset: scores are perishable, tooling is replaceable, and a labelled set of real cases keeps its value across models, frameworks and rewrites.

Key takeaways

• Sample from production, not from imagination. Synthetic questions written from your own corpus are answerable by construction, which is the same selection bias the demo had.
• Stratify by risk rather than by frequency. Rare-but-critical cases and out-of-corpus questions need over-sampling, because a proportional sample will contain almost none of them.
• A hundred labelled cases is a working suite. Fifty is enough to start and enough to catch the regressions that currently reach users.
• Split working and held-out from the beginning, and treat the held-out set as sealed. Everything you iterate against gets fitted to, including by you.
• Redact before labelling. The dataset outlives the conversation it came from, and it will be copied into notebooks, tickets and prompts.

Read this beside Chapter 2, whose rubric supplies the labels, and Chapter 8, which is about keeping a suite discriminative as public benchmarks stop being. Chapter 9 wires the result into CI.

The team decides to build an eval set and immediately writes a hundred questions from scratch. They are reasonable questions. They read like the documentation, because they were written by people who had just read the documentation.

The suite passes at 94% on day one, which everyone finds encouraging. It should be alarming, because a suite that passes before any work has been done is measuring something other than the product.

Take the questions from traffic that already exists

Whatever your system does today, it is receiving inputs. Those inputs are the sample you need, and three sources cover most of it.

Production logs, which give real phrasing including the ungrammatical, the abbreviated and the multi-part. Support tickets and complaints, which give the failures users cared enough to report. And the queries that produced no answer, an escalation or an abandoned session, which are the highest-value cases in the system and are usually already logged as an operational metric rather than as an evaluation asset.

Synthetic questions have one legitimate use. When a category exists in principle and almost never appears in traffic, such as a regulated request or a rare account configuration, write those deliberately and label them as synthetic so their share is visible.

Everything else should be real. Al-Lawati and colleagues argued in May 2026 that benchmark datasets are compromised by appearing in pretraining corpora, which is the public version of the same problem: a question the system has effectively already seen does not measure capability.

Stratify by risk, not by frequency

A random sample of traffic is dominated by the easy majority, and the easy majority is not where the product fails.

Define strata before sampling. Common lookups. Compound requests. Ambiguous phrasing. Out-of-corpus questions where the correct answer is a refusal. Adversarial or boundary-testing input. Rare-but-critical cases, meaning regulated, high-value or legally consequential requests.

Then sample to fill each stratum rather than in proportion. If out-of-corpus questions are two per cent of traffic and half of your embarrassing failures, they should be far more than two per cent of the suite.

Record the sampling design alongside the dataset, because a score is only interpretable against it. A suite that is 40% hard cases will show a lower absolute number than a proportional one, and that is fine as long as nobody compares the two.

One consequence worth stating. Suite scores are not estimates of production quality. They are a measurement against a deliberately hard distribution, and treating them as a prediction of user experience will mislead in both directions.

A hundred cases is a real suite

The instinct is to postpone until the set is large. The set never gets large that way.

Fifty labelled cases catch the regressions that currently reach users, because those regressions are rarely subtle. A hundred gives enough per stratum to see where a change helped and where it hurt. A thousand is a research asset and requires a labelling budget most teams cannot sustain in the first quarter.

Start at fifty, ship the gate, then grow the set from failures. Every production complaint becomes a case, which means the suite improves exactly where the system is weakest and grows at a rate the team can absorb.

Labelling cost is the real constraint, so reduce it deliberately. Hard requirements from Chapter 2 are often mechanically checkable and need no human. Graded dimensions need one, but only on the subset where they matter. And a case can be labelled with partial information: knowing that an answer must cite a particular document is enough to be useful, without a full reference answer.

Split working and held-out on day one

Every set you look at gets fitted to. Not through dishonesty, through iteration.

Split at creation. Roughly seventy per cent working, thirty per cent held out, split within each stratum so both halves have the same shape. The working set is for daily iteration, prompt changes, retrieval tuning and the CI gate. The held-out set is opened rarely, on a schedule, to answer whether the improvements generalised.

Rules that keep the seal meaningful: the held-out set is never used to debug an individual failure, never used to tune a threshold, and never shown in a dashboard that engineers watch daily. When it is run, the result is recorded with the date and the system version.

If the working set improves and the held-out set does not, you have been fitting to the working set. That comparison is the only reliable detector of it, and it is worth the discipline of keeping thirty per cent of your labelling effort in reserve.

LiveBench's design makes the same argument at the level of a public benchmark, using frequently refreshed questions from recent sources precisely because a static set stops measuring what it was built to measure.

Redact before it becomes a fixture

A dataset built from real traffic contains whatever your users wrote, which on most products includes personal data.

The lifecycle matters. Evaluation data is copied into notebooks, pasted into tickets, sent to model providers as prompts, and committed to repositories. It escapes the retention policy of the system it came from, and it does so quietly. OWASP's 2025 entry on sensitive information disclosure is the general warning; this is one of the more direct routes.

Three steps handle most of it. Redact identifiers at extraction time rather than at use time, replacing them with stable placeholders so the case still makes sense. Drop cases whose content cannot be redacted without destroying the meaning, and note the category so the gap is visible. And record the provenance of every case, meaning where it came from and when, so a deletion request can be honoured against the dataset as well as against the source system.

Stable placeholders matter more than they look. Replacing every name with the same token destroys cases where two people are distinguished, so use consistent per-case placeholders instead.

Label the outcome, not the wording

For each case, record what a correct response requires rather than what it should say.

The required facts, meaning the figures, dates and conditions that must appear. The required sources, meaning which documents a grounded answer would cite. The required behaviour, which for out-of-corpus cases is a refusal with a reason. And any prohibited content, such as a figure that is plausible and wrong, or advice the product must not give.

That shape survives model changes, which a reference answer does not. It also makes the mechanical half of scoring straightforward, since checking for a figure or a citation is a string operation rather than a judgement.

Zheng and colleagues' MT-Bench work is the reference point for the judged half, having found strong judges agreeing with humans at over 80% while carrying position, verbosity and self-enhancement biases. Chapter 6 tests the judge against exactly these labels, so their quality bounds everything downstream.

Where the first fifty cases come from, concretely

The abstract version of this chapter is easy to agree with and hard to start. Here is the concrete version, which takes about a day.

Pull last month's sessions. Filter to those that ended in an escalation, a thumbs-down, an abandonment or a support ticket, and read fifty of them. Those give you fifteen to twenty cases immediately, and they are the cases most likely to recur.

Then pull a random sample of successful sessions and read thirty. Roughly a third usually contain something a careful reviewer would not have shipped: a missing caveat, an unstated assumption, a citation to the wrong version. Those give another ten.

Add your known hard categories deliberately. Five out-of-corpus questions where the correct answer is a refusal. Five compound requests. Five with awkward phrasing, taken verbatim from real users rather than cleaned up. Five rare-but-critical cases, written with whoever owns that risk.

That is fifty, sampled by risk, mostly real, with provenance. It fits in a spreadsheet and it is more useful on Monday than a thousand-case plan is next quarter.

Keep it versioned and describable

The dataset is code, and it needs the same handling.

Version it in the repository, with a changelog. Record for each release what was added and why, since a score that improved because easy cases were added is a score that has told you nothing.

Write a one-page description: the strata, the sampling method, the labelling procedure, the agreement measured between labellers, and the known gaps. That page is what lets somebody six months from now interpret a number, and it takes an hour.

Then set a refresh cadence. Retire cases whose underlying documents no longer exist, add cases from recent failures, and keep a stable core so comparisons across quarters remain possible. A suite that is entirely refreshed is a new suite, and its scores are not comparable to last quarter's.

One more habit pays for itself. Give every case an identifier and use it everywhere: in the dataset, in the CI output, in the ticket when a failure is investigated, and in the postmortem when one reaches a customer. Cases then accumulate history, and a case that has failed three times in a year is telling you something about the product that no aggregate score will.

Chapter summary

The first dataset comes from traffic you already have, drawn from production logs, support complaints and the sessions that escalated or were abandoned, because synthetic questions written from your own corpus are answerable by construction and repeat the demo's selection bias. Stratify by risk rather than frequency, filling categories for compound requests, ambiguity, out-of-corpus refusals, adversarial input and rare-but-critical cases, and record the sampling design because a suite score is a measurement against a deliberately hard distribution rather than a prediction of user experience. Fifty labelled cases is enough to start and a hundred is a real suite, grown afterwards from production failures so it strengthens where the system is weakest, with labelling cost controlled by automating the hard requirements and labelling only the graded dimensions that matter. Split working and held-out at creation, roughly seventy to thirty within each stratum, keep the held-out set sealed against debugging and threshold tuning, and read a divergence between the two as evidence of fitting, which is the argument LiveBench makes publicly with monthly refreshed questions. Redact at extraction with stable per-case placeholders, drop what cannot be redacted and record the gap, and keep provenance so deletion requests can be honoured. Label what a correct response requires, being facts, sources, behaviour and prohibitions, rather than a reference wording, since that shape survives model changes and makes half the scoring mechanical. And version the dataset with a changelog, a one-page description including measured labeller agreement, and a refresh cadence that keeps a stable core.

The dataset exists and the rubric can be applied to it. The next question is what to count. Chapter 4 is Outcome, Trajectory, and System Metrics, which separates three families of measurement that get averaged together, and states what each one cannot tell you.