Retrieval as an Evidence Pipeline

Retrieval as an Evidence Pipeline is the reframing that makes the rest of this book operable. Most teams treat retrieval as a single call that either works or does not. It is six stages, each with its own failure mode, and the failures are indistinguishable downstream unless you make them distinguishable on purpose.

Key takeaways

• Six stages, each with a contract: query understanding, candidate generation, fusion, reranking, selection, assembly. A document lost at any one of them produces the same symptom, which is an answer with no support.
• The first question in any retrieval investigation is whether the correct document was ever a candidate. Recall failures and ranking failures need opposite fixes, and teams routinely apply the ranking fix to a recall problem.
• Log the count entering and leaving each stage, plus the rank of the known-correct document where you have one. That single log turns a week of speculation into a ten-minute query.
• Stage contracts should be explicit about what is carried, not just what is returned. Provenance, permissions and timestamps have to survive every boundary or they are unavailable when the window is assembled.
• The pipeline exists to produce supported claims rather than relevant text. Relevance is the proxy; support is the product.

Read this beside Chapter 1, which named these stages once, and Chapter 11, which turns the per-stage logs into attribution. Chapters 4 to 7 take the individual stages in order. The applied version of the fusion stage lives in hybrid search and rank fusion.

An answer comes back wrong, citing nothing useful. Someone checks the index and the correct document is there. Someone else re-runs the query in a notebook and the document comes back second.

Both observations are true. Neither explains anything, because nobody knows what the production pipeline did between those two points. The investigation stalls in the gap between "it is in the index" and "it was not in the window".

The six stages, and what each one can lose

Naming the stages is not bureaucracy. Each has a different failure and a different fix. Conflating them is why retrieval work so often produces no measurable improvement, and why two engineers can disagree for a week about a system they are both describing correctly.

Query understanding turns a user's message into retrieval queries. It loses documents by searching for the wrong thing. Four causes cover most of it: a pronoun that needed resolving from earlier conversation, a product name the user abbreviated, a question that needed splitting into two.

Candidate generation runs those queries against indexes. It loses documents by recall failure. That is the only failure in the pipeline no later stage can repair.

Fusion merges ranked lists from multiple retrievers. It loses documents by weighting, which is why Chapter 4 argues for rank-based fusion over score-based.

Reranking reorders a shortlist with a more expensive model. It loses documents by scoring them below a cutoff, and it can only rerank what fusion handed it.

Selection decides how many survive under a budget. It loses documents to a number that was chosen once and never revisited.

Assembly places the survivors into the window. It loses documents by position, since material in the middle of a long input is used less, as Liu and colleagues measured in 2023.

Recall first, always

The single most useful diagnostic habit in retrieval work is to establish, before anything else, whether the correct document was ever a candidate.

The test is cheap. Take a failing query. Retrieve a large candidate set, say the top 200 rather than the top 20, and check whether the correct document appears anywhere in it. If it does, you have a ranking problem and every downstream tool applies. If it does not, you have a recall problem, and reranking, fusion tuning and prompt changes are all irrelevant.

Recall problems have their own short list of causes. The chunk does not contain the words or the meaning the query expressed, which is Chapter 6's subject. The query was transformed badly upstream. The document is filtered out by a permission or metadata condition, which is correct behaviour that looks like a bug. Or the index is stale and the document was never embedded.

Teams skip this test because it feels remedial, and then spend weeks improving a reranker that never saw the document. In my experience it is the highest-yield twenty minutes available in a retrieval investigation.

Contracts between the stages

A contract states what each stage receives, what it returns, and what it must carry through untouched. The third part gets dropped.

Query understanding receives the conversation and returns one or more query strings with a stated intent and any filters implied by the request. Candidate generation receives those and returns ranked lists per retriever, each item carrying a document identifier, a chunk identifier, a score and the metadata needed later. Fusion receives the lists and returns one ordered list with the contributing ranks preserved. Reranking receives a shortlist and returns it reordered with scores. Selection receives the reordered list and a budget and returns the chosen set. Assembly receives the chosen set and returns the window.

What must survive every boundary is the evidence's identity and its attributes: source, chunk identity, timestamp, permission tags, and the retriever that found it. Losing those at any stage means the window cannot carry provenance, which Chapter 9 requires and OWASP's 2025 guidance on vector and embedding weaknesses treats as a control, recommending classification and tagging of knowledge base content and detailed retrieval logging.

Instrument every boundary

The logging that makes this pipeline debuggable is small and specific.

Per request, per stage, record the count in and the count out. Record the elapsed time per stage. Record the identifiers of the top few items leaving each stage. And where you have a labelled evaluation query with a known-correct document, record that document's rank at each stage, or its absence.

That last field is the one that pays. A table of known-correct rank by stage tells you immediately where the pipeline loses the truth: present at rank 12 after candidate generation, present at rank 15 after fusion, absent after selection because the cutoff was 10.

The cost is a few tens of bytes per request and one dashboard. The alternative is the investigation that opened this chapter. Repeated monthly.

The stages have different rates of change

Not every stage deserves equal attention, and the difference is structural rather than aesthetic.

Candidate generation and chunking change slowly and expensively. Re-embedding a corpus is a batch job with a bill, so those decisions want to be made carefully and revisited rarely.

Fusion, reranking and selection are cheap to change and easy to evaluate offline against a labelled set, which makes them the right place to iterate weekly.

Assembly changes with every product decision, since new features add sections to the window, and it is the stage most likely to regress silently. That is why Chapter 1 argued for the assembler being ordinary versioned code with golden-file tests.

A useful consequence: when a quality change is needed urgently, look at the cheap stages first, not because they are more likely to be the cause, but because the experiment costs an afternoon rather than a week.

One query is rarely the right number

A stage that gets skipped entirely deserves naming here, because it is cheap and it raises recall more than most reranking work does.

The user's message is one phrasing of an information need. The corpus was written in other phrasings. Issuing two or three query variants, then fusing their results, recovers documents that any single phrasing misses: the abbreviation expanded, the synonym the documentation actually uses, the question restated as the statement a document would contain.

Keep it bounded. Variants cost a retrieval call each and a little latency, and beyond about three the returns fall off while the distractor count rises. Generate them deterministically where you can, from a synonym map or from the conversation, rather than with a model call on every request.

There is a failure mode to watch. A rewritten query that drops a constraint, such as a date range or a product version, retrieves confidently wrong material. Carry filters separately from the query text, so a rewrite cannot silently widen the search.

Relevance is the proxy, support is the product

It is worth being precise about what the pipeline is for, because the metrics people adopt encode the answer.

Relevance metrics ask whether retrieved chunks are about the query. Support asks whether the assembled evidence is sufficient for the model to produce a correct, grounded answer. Those diverge in two directions that both matter.

A chunk can be relevant and unsupportive: it discusses the topic without containing the specific figure, date or condition the question needs. And a set of chunks can be individually relevant while jointly missing the qualifier that changes the answer, which is the failure Chapter 6 attributes to chunk boundaries.

Anthropic's contextual retrieval evaluation is instructive on the mechanics here: they measured retrieval failure rate for the top 20 chunks and reduced it from 5.7% to 2.9% by combining contextual embeddings with contextual BM25, and to 1.9% with reranking added. Those are retrieval-side numbers, and they matter because the answer cannot be correct if the evidence never arrived. Support is the next question, and Chapter 11 measures it separately.

What a mature pipeline looks like

Four properties distinguish a pipeline that can be improved from one that can only be replaced.

Every stage is separately runnable, so an engineer can execute candidate generation alone against a labelled set without standing up the whole system. Every stage is separately measurable, with recall at the top of the pipeline and precision toward the bottom. Evidence carries its attributes end to end. And there is a labelled query set, however small, that any change is evaluated against before it ships.

Fifty labelled queries is enough to start. Far better than none, and small enough to assemble in a morning from questions the support inbox already contains. The set will be biased toward the questions you already know about, which is a limitation to state rather than a reason to skip it, and Chapter 11 covers growing it from production traffic.

Chapter summary

Retrieval is six stages rather than one call: query understanding, candidate generation, fusion, reranking, selection and assembly, each with a distinct failure that produces an identical symptom downstream. The first diagnostic is always recall, established by retrieving a much larger candidate set and asking whether the correct document is present at all, because a recall failure cannot be repaired by any later stage and teams routinely spend weeks tuning rerankers that never saw the document. Each boundary needs a contract that states what is returned and what must be carried through untouched, with the evidence's source, chunk identity, timestamp, permission tags and originating retriever surviving to assembly, since provenance that dies in the middle of the pipeline cannot be attached to the window later. Instrument every boundary with counts in and out, elapsed time, the top identifiers, and the rank of the known-correct document where a labelled query exists, which is the field that turns a week of speculation into a query. Treat the stages according to their cost of change: chunking and embedding slowly and carefully, fusion and reranking and selection weekly against an offline set, and assembly with golden-file tests because it regresses silently. Keep the goal precise, which is supported claims rather than relevant text, since a chunk can be on-topic without containing the figure the question needs. And judge maturity by whether each stage can be run and measured alone, whether attributes survive end to end, and whether a labelled query set exists at all.

The first stage where most pipelines leave value on the table is candidate generation, because they use one retriever where two disagree usefully. Chapter 4 is Lexical and Vector Retrieval, Together, which covers why exact identifiers still need lexical matching, how reciprocal rank fusion combines two ranked lists without tuning, and what the rank constant actually does.