Lexical and Vector Retrieval, Together

Lexical and Vector Retrieval, Together is the cheapest large improvement available in most pipelines. The two retrievers fail in opposite directions, which is exactly the property that makes combining them worth more than improving either.

Key takeaways

• Dense retrieval fails on exact strings: error codes, SKUs, version numbers, surnames, function names. Lexical retrieval fails on paraphrase. Neither weakness is fixable by a better model of the same kind.
• Fuse by rank, not by score. Scores from different retrievers are not comparable, and any weighting between them is a tuning parameter that goes stale.
• Reciprocal rank fusion sums one over the rank constant plus the rank, with a default constant of 60 in Elastic's implementation, and requires no tuning at all.
• The rank constant controls how much advantage a top-ranked document keeps. Larger flattens the contribution, smaller rewards being first.
• The gain is measured, not theoretical: Anthropic reports contextual embeddings plus contextual BM25 cutting top-20 retrieval failure rate by 49%, and by 67% with reranking added.

Read this beside Chapter 3, whose fusion stage this is, and Chapter 5, which spends compute on the shortlist that fusion produces. The applied version, with code, is hybrid search and rank fusion.

A support engineer searches for ERR-4471 and gets nothing useful. The runbook for that exact error exists, it is indexed, and the dense retriever ranks it fourteenth, because the string ERR-4471 carries almost no semantic weight and the surrounding prose looks like every other runbook.

A keyword search finds it instantly. That is the whole argument for hybrid retrieval, and everything else in this chapter is mechanics.

The two failure modes are opposites

Dense retrieval embeds text into vectors and compares them by similarity. It is good at paraphrase, at synonyms, at questions phrased differently from the document. It is weak wherever meaning is carried by an exact token rather than by sense.

Identifiers are the clearest case. Error codes, order numbers, SKUs, semantic versions, table names, function names, surnames. To an embedding model these are low-information strings, and near-identical strings sit near each other in vector space, so ERR-4471 and ERR-4471B are neighbours when they should be distinct.

Lexical retrieval, meaning BM25 or a similar term-weighted scheme, has the mirror profile. It matches tokens, so identifiers, rare words and quoted phrases work perfectly. It fails when the user's words and the document's words differ: the user asks about "charging twice" and the document says "duplicate transaction capture".

Neither weakness improves with a better model of the same type. They are properties of the matching mechanism, which is why running both is not a hedge. It is the correct design.

What BM25 is doing, in one paragraph

It is worth knowing the mechanism, because it explains both the strengths and the sharp edges.

BM25 scores a document by how often the query terms appear in it, discounted by how common each term is across the corpus and by how long the document is. Rare terms count for more. Repeated terms saturate rather than accumulating without limit. Long documents are penalised so that length alone does not win.

Three consequences follow directly. A rare identifier is close to decisive, which is the behaviour that rescues the error-code query. A term that appears in every document contributes almost nothing, so boilerplate is ignored for free. And chunk length affects scores, which means the chunking decisions in Chapter 6 change lexical ranking as well as dense ranking.

The sharp edge is tokenisation. Whether ERR-4471 is one token or two, whether case is folded, whether stemming turns "charging" into "charg", all decide what matches. Those settings are usually left at a default nobody reviewed, and on a corpus full of identifiers the default is often wrong.

Fuse by rank, because scores are not comparable

The instinct is to normalise scores and take a weighted sum. Resist it.

A BM25 score is unbounded and corpus-dependent. A cosine similarity is bounded and model-dependent. Their distributions differ per query, so any normalisation is an approximation, and the weight you choose is fitted to whatever queries you happened to test with. It will drift when the corpus grows or the embedding model changes, and nothing will alert you.

Rank is the common currency. Both retrievers produce an ordering, and orderings can be combined without knowing anything about how each was produced.

Elastic states the property directly for reciprocal rank fusion: it requires no tuning, and the different relevance indicators do not have to be related to each other to achieve high-quality results. That is the sentence to quote when someone proposes a weighted score blend, because the alternative is a parameter somebody has to own.

How reciprocal rank fusion works

The formula is small enough to hold in your head, and Elastic's documentation gives it plainly: for each document, sum one divided by the rank constant plus that document's rank in each result set it appears in. The default rank constant is 60. Ranks start at one.

Work through the mechanics. A document at rank 1 in one list contributes 1/61. At rank 2 it contributes 1/62. The difference between first and second place is small, and the difference between rank 1 and rank 20 is larger but still bounded. A document that appears in both lists at middling rank can therefore beat a document that is first in one list and absent from the other, which is the behaviour you want: agreement between independent retrievers is evidence.

The rank constant is the one knob, and it does one thing. A larger constant flattens the contribution of position, so appearing in many lists matters more than appearing first in one. A smaller constant sharpens it, rewarding top placement. The default of 60 comes from the original method, published by Cormack, Clarke and Buettcher at SIGIR in 2009, and it is a reasonable starting point precisely because the method is not sensitive to it.

Two implementation notes matter more than the constant. Take enough candidates from each retriever, because fusion cannot rescue a document neither list contains. And keep the contributing ranks on each fused item, so a later investigation can see which retriever found what.

Contextual chunks change both retrievers

There is a preparation step that improves lexical and dense retrieval at once, and it is the highest-value change in this chapter after fusion itself.

Anthropic's contextual retrieval, published 19 September 2024, prepends a short generated description to each chunk before it is indexed, situating the chunk within its document. The measured effect on their evaluation is specific. Contextual embeddings alone reduced the top-20 retrieval failure rate by 35%, from 5.7% to 3.7%. Contextual embeddings combined with contextual BM25 reduced it by 49%, to 2.9%. Adding a reranker took it to 67%, or 1.9%.

Two things are worth noting about those numbers. They are vendor-published, on their own evaluation set, and should be described that way in any deck. And the second row is the one that carries this chapter: adding lexical retrieval to a contextual dense pipeline moved failures from 3.7% to 2.9%, which is a fifth of the remaining errors removed by running a second retriever you already know how to build.

They also report that retrieving the top 20 chunks outperformed the top 5 and top 10, which matters for Chapter 7's budget arithmetic more than it does here.

What to index, and how many times

Hybrid retrieval means two indexes over the same corpus, and keeping them consistent is an operational problem rather than a modelling one.

Index the same chunk identities in both, so a fused list refers to one set of things. Divergent chunking between the lexical and dense indexes produces results that cannot be fused meaningfully, and it happens easily when the two were built by different people at different times.

Re-index both when chunking changes. A partial re-embed with a stale lexical index is a source of failures that look like ranking problems and are not.

Filters belong in both, applied identically. If the dense retriever filters by tenant and the lexical one does not, the fusion stage has just become a data-leak path. OWASP's 2025 guidance on vector and embedding weaknesses names cross-context leakage in shared stores among its primary risks, and this is one of the plainer ways to create it.

Operating two indexes without drift

The failure that eats hybrid pipelines is not ranking quality. It is the two indexes falling out of step.

Write to both from one path. If a document update triggers a dense re-embed through one job and a lexical index update through another, the two will diverge whenever one job fails, and the symptom is a document that ranks well in one list and is absent from the other for reasons no one can reproduce.

Track a per-document version in both indexes and reconcile on a schedule. The reconciliation query is simple: which document versions differ between the two stores. Run it daily and alert on a non-zero count, because a silent divergence looks exactly like a model regression when it finally reaches a user.

Watch cost asymmetry too. Lexical indexing is cheap and dense indexing is not, so a tempting shortcut is to update the lexical index immediately and batch the embeddings overnight. That is a defensible design and it needs to be stated, because during the gap the two retrievers genuinely disagree about what the corpus contains.

When one retriever is enough

Hybrid is the default recommendation and it is not free, so the exceptions are worth stating.

A corpus with no identifiers, no codes and no proper nouns, where every query is conceptual, gets little from lexical retrieval. Narrative documentation sometimes looks like this. Measure before removing anything.

A corpus that is almost entirely identifiers, such as a log store or a parts catalogue, may not need dense retrieval at all. Users search for exact things and get them.

And a small corpus, under the threshold in Chapter 2, may not need retrieval in either form.

The way to find out is a labelled query set and one experiment, not an argument. Run lexical alone, dense alone and fused, and compare recall at 20. If fusion does not beat both, you have learned something specific about your corpus and saved an index.

Chapter summary

Dense and lexical retrieval fail in opposite directions, with embeddings weak on exact identifiers and lexical matching weak on paraphrase, and neither weakness is fixable by a better model of the same kind. Combine them by rank rather than by score, because BM25 scores and cosine similarities are not comparable and any weighting between them is a parameter that will drift as the corpus and the embedding model change. Reciprocal rank fusion sums one over the rank constant plus the rank across every list a document appears in, with a default constant of 60 in Elastic's implementation, requiring no tuning and no relationship between the relevance indicators, and originating with Cormack, Clarke and Buettcher at SIGIR in 2009. The constant flattens or sharpens the advantage of top placement, and it matters less than taking enough candidates from each retriever and preserving the contributing ranks for later investigation. Prepend generated context to chunks before indexing: Anthropic measured contextual embeddings cutting top-20 retrieval failure by 35%, contextual embeddings with contextual BM25 by 49%, and both with reranking by 67%, all on their own evaluation set and all worth attributing as vendor-published. Keep the two indexes on identical chunk identities, re-index them together, and apply filters identically in both, since an unfiltered lexical index beside a filtered dense one is a leak. And test the exceptions rather than assuming them, because a corpus of pure identifiers or pure narrative may need only one retriever.

Fusion produces an ordered shortlist that is better than either input and still imperfect. Chapter 5 is Rerankers and the Cost of Being Sure, which is about spending cross-encoder compute only where it changes the evidence set, how to decide the shortlist depth, and why a reranker is the easiest place in a pipeline to spend money without improving an answer.