Measuring Context Quality Independently of the Model

Measuring Context Quality Independently of the Model is what makes every other chapter actionable. Without it, a pipeline has one number, the answer is either good or bad, and the only lever anyone can pull is the model.

Key takeaways

• Three questions, measured separately: was the evidence available, was it selected, was it used. Each has a different fix, and a single answer score cannot distinguish them.
• Recall at a generous depth is the first and most important measurement, because a failure there cannot be repaired by anything downstream.
• Score position, not just presence. Evidence selected into the middle of a long window is measurably less likely to be used, so record where it landed.
• Attribution closes the loop: check whether each claim in the answer traces to supplied evidence, which separates a retrieval failure from a generation failure.
• Build the labelled set from production traffic and keep it versioned. Fifty real questions beat a thousand synthetic ones, and the set is the asset rather than any single score.

Read this beside Chapter 3, whose stage logs supply the data, and Chapter 12, which is the migration this measurement makes safe. The practice generalises in an evaluation harness that guides releases.

A team reports that answer quality is 72%. Nobody asks what the number is measuring. Someone asks what would raise it. The room offers a better model, a bigger window, a different embedding, more chunks, a reranker.

Every one of those is a guess, and they cost between an afternoon and a quarter. The measurement that would sort them takes a day to build.

Three questions, not one

Split the pipeline's outcome into three independent questions and the guessing stops.

Was the evidence available. For a labelled question with a known-correct source, did that source appear in the candidate set at a generous depth, say the top 200. This is recall, and it is the only failure that later stages cannot fix.

Was it selected. Did the correct evidence survive fusion, reranking and the budget, and where in the window did it land. This is precision and position together, because both decide whether the evidence had a chance.

Was it used. Does the answer's claim trace to the evidence that was supplied, or did the model answer from parameters, from conversation, or from something else in the window.

Each question has a distinct fix. Availability points at indexing, chunking and query understanding. Selection points at ranking, fusion and budget. Use points at ordering, prompt and distractor count. A single answer score points at nothing.

Recall at depth is the first number

Measure recall before anything else, at a depth well beyond what you actually send.

The procedure is small. For each labelled question, run candidate generation and fusion, then record whether the known-correct chunk appears in the top 200, the top 50 and the top 20. Report all three. The gap between 200 and 20 is the amount of quality your ranking is leaving on the table, and the failure rate at 200 is the ceiling nothing downstream can beat.

Anthropic's contextual retrieval work reports precisely this style of metric, tracking top-20 chunk retrieval failure rate and moving it from 5.7% to 2.9% with contextual embeddings plus contextual BM25, and to 1.9% with reranking. Adopting the same shape of measurement makes your numbers comparable to published ones, which is useful when deciding whether a technique is worth adopting.

One caution about labelling. A question often has several acceptable sources, so scoring against exactly one chunk understates recall. Label the set of chunks that would each be sufficient, and count a hit if any appears.

Score position, not just presence

Presence in the window is necessary and not sufficient, which is why this measurement exists as its own step.

Record, per question, the rank of the correct evidence in the final window and its token offset from the start. Then compare use rates across those positions on your own traffic. Liu and colleagues found accuracy highest at the beginning and end of the input and lowest in the middle, and the useful thing is not to re-derive their result but to find where the effect bites in your assembly.

Two derived metrics are worth keeping. The share of correct evidence landing in the middle third of the window, which should be low and which Chapter 7's ordering rule directly controls. And the distractor count, meaning how many selected chunks were not useful, since Chroma measured a single distractor reducing performance and the effect growing with length.

Those two numbers turn assembly from a matter of preference into something with a score attached, which is the point of the chapter.

Attribution separates retrieval failure from generation failure

The third question is the one that requires care, because it is about the answer and still not about answer quality.

For each claim in the answer, ask whether it is supported by the supplied evidence. Three outcomes: supported, contradicted, or unsupported but not contradicted, which is the interesting middle case where the model produced something plausible from elsewhere.

Running this by hand is slow, so most teams use a model as a checker, and the rules from evaluation practice apply. Give the checker the claim and the evidence, not the reasoning that produced the claim. Ask for a specific verdict per claim rather than a score. And validate the checker against a few hundred human judgements before trusting it, because an unvalidated automatic judge is a number nobody should act on.

The payoff is a clean split. High availability, high selection and low attribution means the evidence arrived and was ignored, which is an assembly problem. Low availability with good attribution means retrieval is the constraint and the generation side is fine. Those two situations look identical in an answer score and need opposite work.

Measuring without labels, when you have none yet

Labelling takes a morning and teams still postpone it, so there is a weaker measurement worth running on day one from production data alone.

Self-consistency of retrieval. Issue each production query twice with small paraphrases and compare the chosen sets. Wide disagreement means ranking is unstable, which is a finding without any labels at all.

Citation coverage. What share of answers cite at least one supplied source, and what share of sentences in an answer are traceable to evidence. A falling coverage rate is a strong early signal that the pipeline has stopped supplying usable material.

Evidence-free answers. Count the requests where the model answered with an empty or near-empty evidence set. Those are recall failures the system already knows about and nobody counted.

Escalation and complaint rate by query type, joined to the retrieved identifiers from Chapter 9. Complaints cluster, and the cluster names the corpus area to label first.

None of these replace the labelled set. All of them are available before it exists, and each one produces the questions you should be labelling.

Build the labelled set from real traffic

The set is the durable asset. Individual scores are perishable and the questions are not.

Start with fifty real questions, drawn from production logs, support tickets and the queries that already produced complaints. Include the awkward ones deliberately: ambiguous phrasing, questions whose answer changed recently, questions that span two documents, questions with no good answer at all.

Label each with the sufficient evidence and the expected answer's key facts, rather than an expected wording. Wording comparisons produce noise; fact checks produce signal.

Grow the set from failures. Every production complaint becomes a labelled case, which means the set gets better precisely where the system is worst. Version it, and record which version produced which measurement, because a curve that improved when the set changed has told you nothing.

Keep a small held-out portion. It is easy to tune a pipeline into the set you look at every day, and the held-out part is what tells you whether the improvement generalised.

The metrics that mislead

Three numbers are reported constantly and answer questions nobody asked.

Average similarity score. It measures how confident the retriever is, not whether it was right, and it moves when the embedding model changes for reasons unrelated to quality. Two pipelines with identical recall can have entirely different score distributions.

Answer similarity to a reference answer. It rewards phrasing and punishes correct answers that were worded differently, which is why the labelled set holds key facts rather than expected text.

Aggregate quality on a synthetic set. Questions generated from the corpus tend to be answerable by construction, because they were written from a chunk that therefore exists and ranks. They measure the pipeline against itself.

A fourth is subtler and worth naming: overall accuracy without a breakdown by question type. Systems usually fail unevenly, being fine on lookup and poor on comparison or recency, and a single average hides exactly the segment that generates complaints. Report by type, and the roadmap writes itself.

Run it as a gate, not as a report

A measurement that runs when someone remembers is a measurement that stops running.

Wire the labelled set into the deployment path. Any change to chunking, embeddings, retrieval parameters, the assembler or the prompt runs it, and a regression beyond a stated threshold blocks the change. Report recall, selection, position and attribution separately, because a change that trades one for another should be visible rather than netted out.

Add cost and latency to the same report. A configuration that improves attribution by a point while doubling cost per query is a trade someone should make deliberately, and RULER's finding that effective context falls short of advertised context is a standing reminder that more tokens is not automatically more quality.

Keep the run cheap enough that nobody argues about it. Fifty questions, three configurations and a checker is minutes of compute, and a gate that takes an hour will be skipped under deadline pressure by the person who most needs it.

Then re-run the whole set on a schedule even when nothing changed, because the corpus changes underneath you. Documents are added, permissions move, the ingestion job fails quietly. A stable pipeline over a drifting corpus is not a stable system.

Chapter summary

Measure the window before the answer, splitting the outcome into three independent questions: was the evidence available, was it selected, and was it used. Recall at a generous depth comes first because nothing downstream can repair its absence, and reporting it at several depths shows how much your ranking is leaving on the table while matching the shape of published numbers such as Anthropic's move from 5.7% to 2.9% to 1.9% top-20 failure. Score position as well as presence, recording where the correct evidence landed and how many distractors accompanied it, since the position effect and the distractor cost are both measured and both controlled by the assembler rather than by the model. Attribute each claim in the answer to supplied evidence, using a validated checker that sees the claim and the evidence but not the reasoning, because high availability with low attribution is an assembly problem and low availability with good attribution is a retrieval problem, and the two look identical in an answer score. Build the labelled set from real traffic, fifty questions to start, labelled with sufficient evidence and key facts rather than expected wording, grown from production failures, versioned, with a held-out portion to catch tuning to the set. And run it as a deployment gate with cost and latency reported beside quality, re-running on a schedule even when nothing changed, because the corpus drifts even when the code does not.

You can now tell which stage is failing and prove that a change helped. Chapter 12 is Migrating a RAG System Into a Context System, which puts the whole book in order: what to change first in a plateaued pipeline, how to run the old and new paths together, and how to avoid the rebuild that produces a different set of unmeasured problems.