Migrating a RAG System Into a Context System

Migrating a RAG System Into a Context System is the last chapter because it is the one that puts the others in order. A plateaued pipeline does not need replacing. It needs five changes, in a sequence where each one makes the next cheaper to evaluate.

Key takeaways

• Do not rebuild. A rewrite exchanges problems you have measured for problems you have not, and the old pipeline's quirks are usually load-bearing for someone.
• Measurement comes first, always. Without the labelled set from Chapter 11, every subsequent change is an opinion and no regression is detectable.
• Then assembly, because it is the cheapest change with the largest effect: ordering, budgets and structural separation cost days and require no re-indexing.
• Then the cheap retrieval changes, being a second retriever and rank fusion, before the expensive ones, being re-chunking and re-embedding.
• Governance last but not optional: provenance, permissions, freshness and the memory write policy, each of which is a data-modelling change with a security consequence.

Read this beside Chapter 11, which supplies the gate every step passes through, and Chapter 1, which named the stages this migration reorganises. Everything between them is the detail of each step.

The pipeline was built eighteen months ago. It works. It has not improved in six months, and nobody can say precisely why. Someone proposes a rebuild on a new framework, with a three-month estimate that everyone privately doubles.

The rebuild will fix the parts nobody can name. It will lose the parts nobody documented. There is a cheaper path, and it starts by finding out what is actually wrong.

Step one: measure, and do nothing else for a week

Change nothing yet. Instrument first, and let the numbers pick the work.

The week produces four things. Fifty labelled questions from real traffic. Per-stage logging of counts in and out, as Chapter 3 describes. Recall at 200, 50 and 20 for the current pipeline. And the position of correct evidence in the assembled window.

Those numbers usually reorder the roadmap on their own. In most plateaued pipelines I have looked at, at least one of three things is true and unsuspected: recall at 200 is already poor, so ranking work is wasted; or recall is fine and the evidence is landing in the middle of the window; or the evidence is present, well placed, and the answer still ignores it, which points at instructions rather than retrieval.

Each of those implies a different first change. Skipping the week means picking one at random.

One more artifact belongs in week one: a written description of the current pipeline, stage by stage, with the parameters each stage uses. It takes an afternoon and it is usually the first time anyone has seen the whole thing on one page. Expect two surprises in it. Something is running that nobody remembers adding, and a parameter somewhere is set to a value that was a temporary experiment.

Step two: fix the assembly

Assembly is first among the fixes because it costs days, needs no re-indexing, and resolves the most common finding from step one.

Give every section of the window a token budget that sums to a working total, and enforce it per section rather than by truncation. Reorder so the decisive evidence sits at the start or immediately before the question, since Liu and colleagues measured the middle as the weakest position. Separate evidence from instructions structurally, with retrieved content clearly delimited as data. Cap conversation history and instructions, the two sections that grow without limit.

Then re-run the labelled set. This step alone frequently moves attribution by several points, and it is reversible in an afternoon if it does not.

One caution. Assembly changes interact with prompt caching, so watch cost as well as quality, and keep the stable prefix stable.

Step three: add the second retriever and fuse by rank

The next cheapest change is running lexical retrieval alongside dense retrieval and fusing the two ranked lists.

It requires an index rather than a re-embedding, which is a smaller operation, and reciprocal rank fusion needs no tuning: Elastic's implementation sums one over the rank constant plus the rank with a default constant of 60, and the relevance indicators do not have to be related to one another.

Measure recall at 20 before and after. Anthropic's evaluation moved top-20 failure from 3.7% with contextual embeddings alone to 2.9% with contextual BM25 added, and the shape of that gain is what you are looking for on your own set.

Add conditional reranking after fusion, not before it, and only where the top fused scores are close. Chapter 5's warning applies: a reranker that runs on every query and changes the chosen set in one query out of six has bought latency rather than quality.

Step four: re-chunk and re-embed, once, deliberately

This is the expensive step and it belongs after the cheap ones, because the cheap ones change what the ideal chunking looks like.

Treat it as a migration rather than a config change. Build the new index alongside the old one. Re-chunk on document structure rather than on a character count, keep rules with their exceptions and tables with their headers, and attach the breadcrumb of source, section, effective date and version to every chunk.

Consider generated situating context at the same time, since it is a change to the same pipeline. Anthropic measured it cutting top-20 retrieval failure by 35% alone, and by 49% with contextual BM25, on their evaluation set. Whether it repays its indexing cost on your corpus is exactly the sort of question the labelled set answers.

Cut over only when the new index beats the old one on your own questions, and keep the old index until it has been beaten for a week in production rather than for an hour in an evaluation.

Step five: govern the context

The last step is the one that gets deferred until an incident forces it, which is the most expensive possible timing.

Attach permissions to every chunk from its source document and filter inside the retrieval query rather than after it, because OWASP's 2025 guidance on vector and embedding weaknesses names cross-context leakage in shared stores as a primary risk. Render dates into the window so recency can be reasoned about. Assign trust levels and enforce the rule that no privileged action follows from retrieved text alone. And write the memory policy down: what may be written, with what provenance, and with what expiry.

None of these improve a benchmark score, which is why they are last in the sequence and non-negotiable in the plan. They are the difference between a system that works and a system that can be operated.

Running two paths without paying twice

The safe way through the middle steps is to run the new path beside the old one, and the cost of doing it badly is a doubled bill for the duration of the migration.

Shadow the new path on a sample rather than on everything. Ten per cent of traffic is usually enough to compare distributions, and the sample can be weighted toward the query types that fail today.

Compare on retrieval, not on answers, wherever possible. Running both retrieval paths and diffing the chosen sets costs no generation tokens at all, and it catches the majority of regressions before an answer is ever produced.

Give the comparison a fixed end date. A shadow deployment with no cutover criterion becomes permanent infrastructure, and two pipelines running forever is a worse outcome than either one alone.

Then cut over per query type rather than globally. The types where the new path clearly wins move first, the ambiguous ones stay on the old path, and the migration becomes a series of small reversible decisions instead of one large irreversible one.

Who does this, and how long it takes

The sequence assumes an owner, which is where Chapter 1 started, and it is worth being concrete about the shape of the work.

One engineer can do steps one to three. Measurement is a day or two of instrumentation plus a morning of labelling. Assembly is a few days. The second retriever and fusion is about a week including the index. That is a month of one person's time, and it resolves most plateaus.

Step four needs more hands, because re-chunking touches ingestion, storage and cost. Budget a few weeks and a bill, and do not start it until steps one to three have been measured, since they change what the right chunking looks like.

Step five is not an engineering project so much as a set of decisions, and it needs somebody with authority over data policy rather than over code. Permissions, retention and trust levels are choices the business makes and engineering implements.

Two organisational notes. Give the labelled set an owner, because a shared evaluation asset with no owner rots within a quarter. And publish the numbers weekly in a place non-engineers read, since a migration that cannot show progress gets cancelled around week five, which is exactly when step four is asking for money.

What not to do, in any order

Four moves look like progress and are not.

Do not switch embedding models as a first response. It is a re-index with a plausible story attached, and step one usually shows the constraint is elsewhere.

Do not raise the retrieved chunk count to compensate for poor ranking. Chroma measured distractors lowering performance, with the effect growing at length, so more chunks buys recall at the cost of precision and the trade is often negative.

Do not add a second framework alongside the first. Two pipelines with unclear ownership is the state you were trying to escape, and it is how a migration becomes permanent.

Do not skip the old path's quirks. Somebody depends on that odd filter, and finding out who during the cutover is worse than reading it now. Write each quirk down as you find it, with a guess at why it exists, and confirm the guess with whoever added it while they are still reachable.

Chapter summary

A plateaued pipeline needs a migration in place rather than a rebuild, because a rewrite trades measured problems for unmeasured ones and discards undocumented behaviour that somebody depends on. Spend the first week measuring only, producing fifty labelled questions from real traffic, per-stage logs, recall at several depths and the position of correct evidence in the window, since those numbers usually reorder the roadmap by revealing which of three unsuspected conditions holds. Fix assembly second, because budgets, ordering, structural separation and caps on the two unbounded sections cost days, need no re-indexing and address the most common finding. Add the second retriever and fuse by rank third, which needs an index rather than an embedding pass and requires no tuning, with conditional reranking after it rather than on every query. Re-chunk and re-embed fourth, as a real migration with a parallel index, structure-aware boundaries, breadcrumbs on every chunk and a decision about generated context measured rather than assumed, cutting over only after the new index has beaten the old one on your own questions in production. Govern last and without exception: permissions filtered inside the query, dates rendered into the window, trust levels bound to behaviour, and a written memory policy. And avoid the four moves that look like progress, being a reflexive embedding swap, more chunks to paper over ranking, a second framework beside the first, and a cutover that ignores the old system's quirks.

One last note on expectations. A migration run this way rarely produces a single dramatic jump. It produces four or five modest gains that compound, and a system whose failures now have names. The second outcome is the one that keeps paying, because the next plateau arrives with a diagnosis attached rather than with another rebuild proposal.

That is the book. Its argument in one line: the context window is the surface you control, so treat it as an engineered artifact with stages, contracts, budgets, provenance and measurement, and most of what gets blamed on the model turns out to have a name, a stage and a fix.