Assembling the Window: Order, Budget, Compaction

Assembling the Window is where retrieval either pays off or is quietly discarded. The same evidence, in a different position and under a different budget, produces a different answer, and nothing upstream can compensate for an assembly that puts the decisive material where it will be ignored.

Key takeaways

• Give every section an explicit token budget that sums to a stated total. A window with no budget is one where the least important section grows until it crowds out evidence.
• Order for position sensitivity. Accuracy is highest at the start and end of the input, so decisive evidence belongs at one of those ends and never in the middle of a long window.
• Filling the window is not free. Length degrades performance on its own, and a single distractor measurably lowers it, so include what supports the answer rather than everything that retrieved.
• Compaction must be reversible. Keep the identifier of whatever was summarised so the full text can be recovered, and never compact evidence into prose that cannot be traced back to its source.
• Assemble through one function, log what it produced, and test it with golden files. The window is the artifact everything else is judged by.

Read this beside Chapter 2, whose measurements justify the ordering rule, and Chapter 6, whose chunk sizes decide how many pieces fit. Chapter 9 adds the fields that travel with each piece of evidence.

A pipeline retrieves the right twelve chunks. The assembler places them after 3,000 tokens of accumulated instructions, before a long conversation history, in the middle of a 60,000-token window.

The model answers from the conversation, because that is at the end. Every part of the retrieval system worked. The evidence was in the room and out of position.

Give every section a number

Start with the inventory. A production window usually contains: system instructions, tool or schema definitions, stable reference material, retrieved evidence, memory, conversation history, and the current question.

Assign each a token allowance, and make the allowances sum to a target that is below the model's advertised limit. RULER's April 2024 finding is the reason for the margin: of models claiming 32K or more, only half maintained satisfactory performance at that length, so the working limit is not the datasheet limit.

Then enforce the allowances in code. Enforcement means each section is measured before assembly and reduced by its own rule if it is over, rather than the whole prompt being truncated at the end, which is the default behaviour and the worst one, because truncation removes whatever happens to be last.

Two allocations deserve stating explicitly. Instructions should have a cap, because they grow after every incident and nobody removes them. Conversation history should have a cap, because it grows without limit by construction.

Order for position, not for convenience

Liu and colleagues established the position effect in July 2023: performance is highest when relevant information sits at the beginning or the end of the input and degrades significantly when the model must use information in the middle.

The ordering rule that follows is simple. Put the most decisive evidence in a prime position, meaning immediately after the instructions or immediately before the question. Put the least decisive material in the middle. Put the question last.

There is a tension with prompt caching, which rewards a stable prefix, and the resolution is usually clean: instructions and stable reference material at the top, where they cache well, then the primary evidence, then the variable material, then the question. Where a system needs both a large stable prefix and prime-position evidence, splitting evidence across the two prime positions is a legitimate answer, with the strongest one or two chunks placed just before the question.

Two smaller ordering decisions matter more than they look. Order evidence chunks by rank descending, so if the model reads unevenly it reads the best material first. And keep chunks from the same document adjacent, because scattering them makes the model reconcile the same source repeatedly.

Include what supports the answer, not everything that retrieved

The temptation with a large window is to include the top 50 chunks and let the model sort it out. The measurements say that costs something.

Chroma's Context Rot report of 14 July 2025, across 18 models, found that performance varies significantly as input length grows even on simple tasks, that a single distractor lowers performance relative to baseline, and that the effect of distractors grows at longer inputs.

Against that, Anthropic's contextual retrieval evaluation found the top 20 chunks outperforming the top 5 and top 10. So the answer is not minimalism either. It is that the right count is an empirical question with a real cost on both sides, and it should be measured on your corpus rather than argued.

A useful default: select by score shape rather than by fixed count, as Chapter 5 describes, then cap by budget. Where the reranker's scores fall off a cliff after six chunks, send six. Where they decline gently across thirty, send closer to twenty and accept that the question is diffuse.

Compaction has to be reversible

Long conversations and long evidence sets both eventually exceed the budget, and the usual response is to summarise. Summarisation is lossy, and the loss is invisible unless you design for it.

Three rules keep compaction safe. Compress the least decisive material first, which is usually old conversation turns rather than evidence. Keep the identifier of whatever was compressed, so the full text can be re-fetched if the model or a later stage needs it. And never compact across a provenance boundary: summarising six documents into one paragraph destroys the mapping from claim to source, which Chapter 9 needs and which citations depend on.

Prefer structured compaction to prose. A compacted conversation is better represented as a short list of durable facts with their turn identifiers than as a paragraph of narrative, because the facts survive further compaction and the narrative degrades each time it is rewritten.

Watch for the compounding case. A conversation summarised repeatedly is a summary of a summary, and errors introduced early become authoritative later. Re-derive from the original turns rather than re-summarising the summary whenever the original is still available.

Formatting is part of the assembly

How evidence is rendered changes how reliably it is used, and it costs nothing to get right.

Give every chunk a visible header carrying its breadcrumb from Chapter 6: source, section, date, version. The model needs those words to answer questions about recency and scope, and a citation cannot be produced from metadata the model never saw.

Number the chunks. A numbered list of evidence lets the model refer to items by index, which makes citations cheap to parse and mismatches easy to detect.

Keep the rendering stable and machine-generated. Hand-written string concatenation drifts, and the day a newline goes missing between two chunks is the day two documents silently merge into one apparent source.

Prefer plain delimiters over elaborate markup. Long runs of decorative separators cost tokens on every request and add nothing, while a simple consistent header per chunk costs a handful of tokens and does the whole job.

Never concatenate untrusted text into instructions

Assembly is the point where the security posture of the whole system is decided, because it is where retrieved text meets your instructions in one string.

Keep them separated by structure rather than by hope. Evidence goes in a clearly delimited section, labelled as data, with its source attached. Instructions state plainly that content inside the evidence section is information rather than instruction. The question is separate again.

That does not make injection impossible, and Chapter 10 covers the rest of the defence. It does remove the easiest version of the attack, where a retrieved document that says "ignore previous instructions" is placed in exactly the position where the model expects instructions.

The same structural discipline pays off in debugging. When the window is assembled from named sections, a log of what went into each section is readable by a human, and a wrong answer can be traced to a section rather than to a wall of text.

Decide what to drop before you are forced to

Every window eventually overflows, usually during the request that matters most. The order in which material is sacrificed should be a decision rather than an accident of which section the truncation reached first.

Write the drop order down, in the assembler, as an explicit list. A workable default runs: oldest conversation turns first, then low-ranked evidence, then memory entries below a confidence or recency threshold, then optional reference material. What is never dropped is the question, the instructions that carry safety constraints, and the top-ranked evidence.

Make the drop visible. When something is dropped, note it in the window itself, in one line, saying that earlier turns were summarised or that additional sources were omitted. A model that knows its view is partial behaves better than one that believes it has everything, and a user who sees the note has a reason to ask a narrower question.

Then alert on it. Frequent drops mean the budget is wrong or the retrieval count is too generous, and both are fixable. A system that has been silently dropping evidence for six weeks looks exactly like a system whose model got worse.

One assembler, logged and tested

The assembler should be a single function with a single output, and everything that reaches the model should pass through it.

Log the assembled window's shape per request: tokens per section, chunk identifiers included with their ranks, what was compacted, and the total. That log is what makes Chapter 11's attribution possible, and it is small.

Test it with golden files. A handful of representative requests, an assembled window committed as a fixture, and a test that fails when the shape changes unexpectedly. Prompt regressions are otherwise invisible: nothing errors, the tests pass, and quality moves for reasons nobody can locate three weeks later.

And version it. When quality changes on a Tuesday, the first question is what changed, and an assembler version in the request log answers it in seconds.

One measurement belongs beside those logs. Track the share of the window spent on evidence against the share spent on instructions, memory and history, as a ratio per request type. In most systems that have run for a year, evidence is the minority tenant in its own window, and nobody noticed because each addition was individually reasonable.

Chapter summary

Assembly spends or wastes everything the pipeline produced, so give each section of the window an explicit token allowance that sums to a working total below the advertised limit, since RULER found only half of models claiming 32K or more performing satisfactorily at that length. Enforce allowances per section rather than truncating the whole prompt at the end, and cap the two sections that grow without limit, being instructions and conversation history. Order for the position effect that Liu and colleagues measured, with decisive evidence at the start or immediately before the question and the weakest material in the middle, reconciling with prompt caching by keeping stable material at the top and splitting evidence across both prime positions when necessary. Include what supports the answer rather than everything retrieved, because length and distractors both degrade performance measurably while Anthropic found the top 20 chunks beating the top 5 and 10, which makes the count an empirical question resolved by score shape and budget. Make compaction reversible by compressing the least decisive material first, keeping identifiers so the original can be recovered, refusing to compact across provenance boundaries, preferring structured facts to narrative, and re-deriving from originals rather than summarising summaries. Separate evidence from instructions structurally, labelling retrieved content as data, which removes the easiest form of injection even though Chapter 10 is needed for the rest. And run one assembler, logging tokens per section and chunk identifiers per request, tested with golden files and versioned, because a prompt regression is otherwise invisible.

The window now holds the right evidence in the right order. What it also holds, in most systems, is memory that nobody has a policy for. Chapter 8 is Memory: What to Write, What to Forget, which is about the write policy, the provenance every remembered fact needs, and why an expiry date is not optional.