Rerankers and the Cost of Being Sure

Rerankers and the Cost of Being Sure is a chapter about proportion. A reranker is the most accurate scoring you can put in a retrieval pipeline and the easiest place to spend money without changing an answer.

Key takeaways

• A cross-encoder reads the query and the candidate together, which is why it beats embedding similarity and why it costs one model call per document scored.
• The only reranking that matters is reranking that changes the chosen set. Measure the rate at which it does, per query type, before deciding the shortlist depth.
• Shortlist depth is the real parameter. Too shallow and the reranker cannot rescue anything fusion mis-ordered; too deep and you pay for scoring documents that were never going to be chosen.
• Reranking is not only a precision tool. Its second use is deciding what to drop, and with a large window the better question is often ordering rather than exclusion.
• Anthropic measured reranking moving top-20 retrieval failure rate from 2.9% to 1.9%. Useful, bounded, and smaller than the gain from adding lexical retrieval in the first place.

Read this beside Chapter 4, whose fused list is the input here, and Chapter 7, which places what survives. Chapter 11 measures whether any of it improved the answer.

A team adds a reranker and latency rises by 400 milliseconds per query. Quality on their evaluation set improves by a point and a half.

Nobody asks the follow-up question, which is how often the reranker changed the documents that reached the model at all. When they finally measure it, the answer is about one query in six. The other five paid the latency for a reordering that produced the same window.

What a reranker actually does differently

Retrieval scores a query and a document separately. The document was embedded ahead of time, the query is embedded now, and the comparison is a distance between two vectors that never met.

A cross-encoder reranker puts the query and the candidate into one model call and scores them jointly. That lets it use interactions between the two: the specific number in the question against the specific number in the passage, the negation, the qualifier, the date range. It is strictly more informed than a vector comparison.

The cost follows from the same property. Because scoring is joint, nothing can be precomputed, so the work is linear in the number of candidates and it happens inside the request. Fifty candidates is fifty scoring passes.

That is the whole trade. More accurate because it reads both together, more expensive for exactly the same reason.

The only question that matters: does the chosen set change

A reranker that reorders the shortlist without changing which documents reach the window has done nothing, unless order itself matters, and Chapter 7 shows that it sometimes does.

So instrument the change rate. For each query, record the set of chunk identifiers selected before reranking and after. Then report two numbers: how often the sets differ at all, and how often they differ in the top few positions where the position effect is strongest.

Expect the answer to vary sharply by query type. Ambiguous conceptual questions, where fusion produces a jumble of plausible candidates, are where reranking earns its cost. Exact-identifier lookups, where lexical retrieval already put the right document first, are where it changes nothing.

That variation is an argument for conditional reranking. Skip it when the top fused result is well separated from the rest, and run it when the top scores are close. That one rule typically removes a large share of reranking calls with no measurable quality cost, and it is cheaper to implement than any model change.

Shortlist depth is the parameter to think about

Depth decides both the ceiling and the bill, and it is chosen carelessly more often than any other number in a pipeline.

Too shallow and the reranker is decorative. If fusion put the right document at rank 40 and you rerank the top 20, no amount of cross-encoder accuracy helps. Too deep and you are paying to score documents that would never have been selected, which is pure latency.

The way to pick it is empirical and takes an afternoon. Take a labelled set, run retrieval and fusion, and record the rank of the correct document in the fused list. Plot the distribution. The depth that captures the great majority of correct documents is your shortlist size, and anything beyond it buys the tail at linear cost.

Two adjustments follow. Cap the depth by latency budget rather than by ambition, since a reranker that pushes a request past a user's patience has cost more than it returned. And revisit the number when fusion changes, because a better fused ordering means a shallower shortlist suffices.

Reranking also decides what to drop

The precision framing hides a second job that is more useful once windows are large.

With a small window, selection is a hard cutoff and the reranker's score decides who is in. With a large window, you can afford to include more, and the interesting question becomes ordering rather than exclusion, because position and distractor count both affect the outcome.

Chroma's Context Rot report of 14 July 2025 is the relevant evidence: across 18 models, a single distractor lowered performance relative to baseline, with the effect growing as inputs lengthened. So including a marginal document is not free even when the tokens are.

That gives a useful way to read the score distribution. Where the reranker's scores fall off sharply after a handful of documents, cut there and keep the window tight. Where they decline gently, the evidence is spread across many chunks, which is a signal about the corpus and the question rather than about the reranker.

Anthropic's measurement is the counterweight to over-trimming: in their evaluation the top 20 chunks outperformed the top 5 and top 10. Cut on the score shape, not on a habit.

Choosing one, and the properties that decide it

Reranker choice is usually made on a leaderboard. Four operational properties matter more.

Maximum sequence length. A cross-encoder truncates, and a reranker whose limit is shorter than your chunks is scoring the first part of each candidate while ignoring the rest. Check the limit against your actual chunk size distribution, not against the average.

Latency at your batch size. Scoring 50 candidates is a different operation from scoring 5, and providers differ more in how they behave under batch than in their headline accuracy. Measure at the depth you will actually use.

Language and domain coverage. A reranker trained on general web text can be worse than your embedding model on legal, clinical or code corpora, which is the one case where adding the stage lowers quality.

Deployment shape. A hosted reranker adds an external dependency in the request path with its own rate limits and version changes. A self-hosted one adds GPU capacity you have to run. Neither is wrong. The choice belongs to whoever carries the pager, and it should be made before the model comparison rather than after.

Scores are for ordering, not for thresholds

A reranker returns a number, and the number invites a threshold. Be careful with it.

Cross-encoder scores are comparable within one query's candidate set, which is what ordering needs. They are not reliably comparable across queries, because the scale shifts with the query and the corpus. A fixed cutoff of 0.7 will therefore drop everything on hard queries and nothing on easy ones, which is the opposite of the behaviour you want.

Use relative rules instead. Keep candidates within some fraction of the top score. Cut where the largest gap in the sorted scores appears. Or keep a fixed count and let the ordering do the work, which is the simplest option and often the right one.

If you do need an absolute threshold, for example to decide that no document is good enough and the honest answer is "I do not know", calibrate it per query type against a labelled set and re-check it when the reranker version changes. That answer is worth having, and it costs more care than a constant in a config file.

What reranking cannot fix

Three failures survive a reranker, and spending more on it makes none of them better.

Recall failures, which Chapter 3 insists you test for first. A reranker only sees what fusion handed it, so a document that was never a candidate cannot be rescued at any depth.

Chunking failures, where the correct chunk contains a claim but not the qualifier that bounds it. The reranker will score the chunk as relevant, correctly, and the answer will still be wrong. That is Chapter 6.

Assembly failures, where the right documents are chosen and then placed in the middle of a long prompt, which Liu and colleagues showed in 2023 is where information is used least. The evidence arrived and was ignored, and no retrieval-side improvement touches it.

Naming these matters because reranking is where teams reach when quality plateaus, and in my experience the plateau is more often chunking or assembly than ranking. The diagnostic order from Chapter 3 applies unchanged: establish recall, then ranking, then the window. A reranker only touches the middle term.

Cost, stated honestly

The reranker's bill has three components and teams usually count one.

The direct cost is a model call per candidate per query, which is the visible part. The latency cost is added to every request that runs it, including the majority where it changes nothing, which is the part that shows up in user behaviour rather than in the invoice. And the operational cost is another model dependency in the request path, with its own availability, rate limits and version changes.

The measured benefit, from Anthropic's evaluation, is the move from 2.9% to 1.9% top-20 retrieval failure, against 5.7% for the baseline pipeline. In other words: adding lexical retrieval to contextual embeddings removed more failures than reranking did, and cost less to run.

That ordering is the practical guidance in this chapter. Fix recall, add the second retriever, fuse by rank, then rerank conditionally where the fused scores are close. Doing it in the opposite order is how a pipeline ends up expensive and unchanged.

Chapter summary

A cross-encoder reranker scores the query and each candidate jointly, which makes it more accurate than a vector comparison and linear in cost inside the request, so the same property explains both its value and its bill. The only reranking worth paying for is reranking that changes the chosen set, so instrument the change rate per query type and expect ambiguous conceptual queries to benefit while exact-identifier lookups do not, which is the basis for running it conditionally when the top fused scores are close. Choose shortlist depth empirically from the distribution of the correct document's rank in the fused list, cap it by latency budget rather than ambition, and revisit it whenever fusion improves. Remember the second job: with a large window the question shifts from exclusion to ordering, and since Chroma measured a single distractor degrading performance with the effect growing at length, a marginal document is not free even when its tokens are, so cut where the score distribution falls off sharply and keep more where it declines gently. Accept what reranking cannot fix, being recall failures, chunking failures where the qualifier is missing, and assembly failures where good evidence lands in the middle of a long prompt. And count the cost honestly across the model calls, the latency paid on every request including the many that change nothing, and the added dependency, against a measured benefit of 2.9% to 1.9% top-20 failure, which is smaller than the gain from adding lexical retrieval in the first place.

Everything so far assumes the chunk is the right unit. Chapter 6 is Chunking Is a Document-Modelling Problem, which is where the assumption breaks: a claim separated from the condition that bounds it retrieves perfectly and answers wrongly, and no ranking improvement can detect that.