Context Is the Only Thing You Control

Context Is the Only Thing You Control is the premise the rest of this book depends on. The window is the single surface where your engineering decisions reach the model, and in most systems nobody owns it end to end.

Key takeaways

• The controllable surface is small and specific: what is retrieved, what survives ranking, how it is chunked, in what order it sits, what budget each section gets, and what is remembered between runs.
• Context assembly is usually built by several people at different times, so no single person can say why a given token is in the window. That absence of ownership is the root cause behind most quality plateaus.
• The window is not a bucket. Position, length and the presence of similar-looking distractors all change what the model does with the same evidence.
• Long context did not remove the problem, it moved it. Under about 200,000 tokens the argument for retrieval weakens sharply, and above it the assembly decisions get harder rather than easier.
• Judge a context system by whether a bad answer can be attributed to a stage. If it cannot, every improvement after this point is a guess.

Read this beside Chapter 2, which puts numbers on what long windows fixed, and Chapter 11, which is the attribution machinery this chapter argues for. The retrieval mechanics start in Chapter 3.

A team spends six weeks upgrading its embedding model. Retrieval quality improves on their offline set and the product does not change.

The reason turns up later, and it is mundane. The retrieved chunks were being placed in the middle of a long prompt, after a large block of boilerplate instructions, and the model was using them less than anyone assumed. Nothing was wrong with retrieval. The window was assembled badly, and no one was responsible for the window.

What you actually control

Start with the list, because it is shorter than people expect and it is the whole scope of the discipline.

You control which corpus is searched and how it is indexed. You control the query that is issued, and whether it is the user's words or a rewritten form. You control which candidates survive ranking, and how many. You control chunk boundaries. You control the order of what appears in the window and the token budget each section receives. You control what is written to memory, what is retrieved from it, and when it expires. And you control what is attached to each piece of evidence, meaning its source, its age and the permissions that govern it.

What you do not control is everything else: the weights, the tokeniser, the attention implementation, and the model the provider ships next quarter. Teams spend a surprising share of their effort on the second list, mostly by switching models and hoping.

The reason this matters is not philosophical. Effort spent on the controllable list compounds across model changes, and effort spent on model selection has to be redone every time the field moves.

The window is not a bucket

The common mental model is that the window is storage: put the relevant material in, and the model will use it. Three measured results say otherwise, and every chapter after this one is downstream of them.

Position matters. Liu and colleagues, in work published in July 2023 and revised that November, found that performance is highest when relevant information sits at the beginning or end of the input and degrades significantly when the model must use information in the middle.

Length matters independently of content. Chroma's Context Rot report of 14 July 2025, covering 18 models, found that performance varies significantly as input length grows even on simple tasks. That is a claim about the length itself rather than about difficulty.

Similarity matters. The same report found that lower semantic similarity between the question and the answer produces a steeper decline as inputs lengthen, and that a single distractor reduces performance, with the effect growing at longer lengths.

Read together, those three make the window an ordered, budgeted artifact rather than a container. A chunk placed at position forty in a long prompt is not the same evidence as the identical chunk at position two.

Nobody owns the assembly

Look at how the code arrives, in most teams, and the plateau explains itself.

The retrieval layer is built first, by whoever set up the vector store. The prompt template is written by whoever built the first feature, and it accumulates instructions from every incident since. Memory is added later, by someone who wanted personalisation. Compaction appears when a long conversation first overflows, written under time pressure. Each part is reasonable. Nobody has seen the assembled result.

Two symptoms identify this pattern. First, no one can answer, for a given production response, why each block of text was in the window. Second, the prompt template contains instructions that nobody can trace to a decision, and everyone is afraid to remove them.

The remedy is an owner and an artifact. One person accountable for what the model sees, and a single function that assembles the window, so that every block of text enters through one place and can be logged there. That function is where Chapters 7 and 9 attach.

Long context moved the problem, it did not remove it

The strongest argument against this book is that windows are now large enough to make assembly unnecessary. Take it seriously, because it is partly true.

Anthropic's contextual retrieval work of 19 September 2024 states the practical threshold plainly: knowledge bases under roughly 200,000 tokens, about 500 pages, can be put in the prompt without retrieval at all. Below that line, a retrieval pipeline may be pure cost, and teams that built one because it was expected should measure before defending it.

Above the line, two things get harder rather than easier. Effective context is smaller than advertised context, and RULER, published in April 2024, found that of models claiming 32K or more, only half maintained satisfactory performance at 32K. And cost per query rises with everything you include, so a window filled indiscriminately is a bill as well as a distraction.

So the decision is no longer retrieval versus long context as an identity. It is a calculation over corpus size, freshness, access control and cost per query, which is Chapter 2's subject and one of this book's stated outcomes.

Access control lives here too, whether or not you designed it

One consequence of context assembly is easy to miss and expensive to discover: the window is a place where authorisation is enforced or lost.

If a retrieval layer can return any document in the index, then the permission model of your product is whatever the retriever happens to do. In a multi-tenant system that is a data-leak path with no logging on it. OWASP's 2025 entry on vector and embedding weaknesses names exactly this, listing unauthorised access and cross-context leaks in shared vector databases among its primary risks, and recommending permission-aware vector stores and immutable retrieval logs.

Freshness behaves the same way. A retrieved chunk carries no timestamp unless you attach one, so a model asked about current pricing will answer confidently from a document written two years ago, and nothing in the window told it otherwise.

Both of these are properties of the assembly rather than of the model, which puts them inside the controllable list. Chapter 9 makes them explicit fields on every piece of evidence.

There is a third property of the same kind, and it is the one that turns a quality problem into a security problem. Retrieved text is input. It arrives from a store that users, customers or partners can often write to, and it lands in the window beside your instructions with no marker separating the two. OWASP's 2025 prompt-injection entry names this route directly: modifying documents in a repository used by a retrieval system changes the model's output when those documents are retrieved, and the injected content does not have to be visible to a human reading the page.

That is why this book treats the window as an artifact with fields rather than as a string. A string cannot record where a passage came from, whether the requester was entitled to see it, how old it is, or whether it should be trusted as instruction or merely as evidence. Those four attributes decide most of what goes wrong later, and none of them survive concatenation.

Assembly is code, and it deserves the same treatment

Prompt templates tend to live outside the practices the rest of the codebase enjoys. They are edited in a hurry, rarely reviewed line by line, and almost never tested in isolation, which is strange for the component that determines what the model sees.

Treat the assembler as ordinary code. One function, versioned with the application, with the template composed from named sections rather than concatenated inline. Unit tests that assert structural properties: that the budget holds, that evidence carries its provenance fields, that no section is silently empty, that the ordering rule was applied. And a golden-file test on the assembled window for a handful of representative requests, so that a change to one section is visible as a diff rather than as a mystery next week.

The payoff shows up during incidents. When the assembled window is reconstructable from the request and the code version, an odd answer becomes something you can read, rather than something you have to reproduce.

The stages, named once

The rest of the book follows the path a piece of evidence takes, and naming the stages here gives the vocabulary for attribution later.

Query understanding turns a user's message into one or more retrieval queries. Candidate generation runs those against indexes, usually more than one. Fusion merges ranked lists into a single ordering. Reranking spends more compute on a shortlist to improve precision. Selection decides how many survive, under a budget. Assembly places the survivors into the window in a chosen order with the instructions and memory. Memory read and write happen either side of the run.

Each stage has a contract, and each is a place a bad answer can originate. That is the point of naming them. A team that can say "recall was fine, fusion dropped it, the reranker never saw it" is doing engineering, and a team that says "the model got it wrong" is not.

What good looks like

A context system is working when four things are true, and none of them is about answer quality directly.

Every block of text in the window can be traced to a decision, with a source and a reason. The window has a budget, stated in tokens per section, and the budget is enforced rather than aspirational. Evidence carries provenance, age and permissions from the store into the prompt. And a bad answer can be attributed to a stage within minutes, using logs that already exist.

Answer quality follows from those, and it does not follow from any of them individually. That is why this book is ordered the way it is: the measurement chapter comes late, but the decision to make the system measurable has to be taken early, because retrofitting attribution to an assembly nobody owns is most of the work in Chapter 12.

Chapter summary

The controllable surface in an LLM system is small and specific: the corpus and its index, the query, the candidates, the chunk boundaries, the order and budget of the window, what memory holds, and the provenance attached to every piece of evidence. Everything else, including the weights and next quarter's release, is outside your control, and effort spent there has to be repeated each time the field moves. The window is not a bucket, because position, length and distractor similarity all change the outcome for identical evidence, which is what Lost in the Middle and Chroma's Context Rot report between them establish. The usual reason a system plateaus is that assembly has no owner: retrieval, prompt template, memory and compaction were built by different people at different times, so nobody can say why a given block of text is in the window. Long context genuinely changed the calculation, with a practical threshold around 200,000 tokens below which retrieval may be unnecessary, and above it effective context still falls short of advertised context while cost rises with everything included. Access control and freshness are assembly properties too, and OWASP's 2025 guidance names cross-context leakage in shared vector stores as a primary risk. Name the stages once, from query understanding through to memory write, because attribution is the difference between engineering and guessing. And judge the system by whether every block is traceable, the budget is enforced, provenance survives into the prompt, and a bad answer can be attributed to a stage.

The obvious next question is how much of this long context has already solved. Chapter 2 is What Long Context Fixed, and What It Did Not, which puts the measured numbers against the claim, gives the decision rule for retrieval versus a large window, and shows why effective context and advertised context are different quantities.