Chunking Is a Document-Modelling Problem

Chunking Is a Document-Modelling Problem because a chunk boundary decides what a sentence means. Split a rule from its exception and both halves retrieve well, score well, and produce a confidently wrong answer that no ranking improvement can catch.

Key takeaways

• A chunk is the unit of meaning your system can retrieve. A fixed character count chooses that unit by accident, and the accident lands hardest on rules with exceptions, tables with headers, and figures with units.
• Segment on the document's own structure first: headings, list items, table rows with their header, clauses. Fall back to size limits only inside a structural unit that is genuinely too large.
• Attach the breadcrumb. Document title, section path, effective date and product version belong on every chunk, because the retrieved fragment arrives without the page around it.
• Generated context helps measurably. Anthropic reports prepending chunk-situating context cutting top-20 retrieval failure by 35%, and by 49% when combined with lexical retrieval.
• Chunk size is a distribution, not a setting. Report the spread, and treat the long tail as a modelling problem rather than trimming it.

Read this beside Chapter 4, where chunk length changes lexical scoring, and Chapter 7, where chunk size decides how many pieces fit the budget. Chapter 11 is where a chunking change is proven rather than assumed.

The support assistant tells a customer their bespoke order is refundable within fourteen days. The policy document says refunds are issued within fourteen days. The next sentence says this does not apply to bespoke orders.

That sentence was in a different chunk. Retrieval worked, ranking worked, the model read exactly what it was given, and the answer was wrong.

The boundary is the model

Every chunking scheme is a claim about what constitutes a self-contained unit of meaning in your corpus. Most teams never make that claim explicitly, so the claim is whatever the default in the library said, usually a token count with an overlap.

Three document patterns punish that default reliably.

Rules with exceptions, where the qualifier follows the rule and a split between them inverts the answer. Policy, legal, medical and pricing documents are full of these.

Tables, where the header row carries the meaning of every cell beneath it. A chunk containing rows 40 to 60 of a table is a grid of numbers with no columns.

Figures with units and conditions, where "220" means nothing without "milligrams" and "for adults over 40" three lines up.

The pattern across all three is the same: meaning is carried by proximity in the original document, and chunking destroys proximity unless it is told not to.

Segment on structure, then constrain by size

The order matters. Structure first. Size is a constraint applied inside it.

Most corpora arrive with structure already present: headings and subheadings in Markdown or HTML, numbered clauses in contracts, question and answer pairs in support content, function and class boundaries in code. Those boundaries were chosen by an author to delimit a complete thought, which is exactly what you are looking for.

Apply size limits within a structural unit rather than across it. If a section fits inside your target size, it is one chunk regardless of how short it is. If it exceeds the limit, split it at the next structural level down, at paragraph or clause boundaries, and repeat the breadcrumb on every piece.

Two extra rules prevent the common damage. Never split a table from its header: repeat the header on every chunk of the table, or keep small tables whole. And never split a list item from its introducing sentence, because "you must do all of the following" carries the meaning of the items beneath it.

Overlap is a blunt substitute for this. A sliding window with 20% overlap sometimes keeps a rule with its exception by luck, and it inflates the index while doing so. Structure-aware chunking does the same job on purpose, and overlap is then a small safety margin rather than the mechanism.

Attach the breadcrumb

A retrieved chunk arrives with no page around it. Whatever it does not carry, the model cannot know.

Put four things on every chunk: the document title, the section path, the effective date or last-modified date, and any version or scope identifier such as a product edition, region or plan tier. Store them as metadata and render them into the chunk text at assembly, because filters need the fields and the model needs the words.

The scope identifier is the one that prevents a specific and embarrassing failure. Documentation for version 3 and version 4 of the same product is textually similar, so a dense retriever rates both highly, and without a version on the chunk the model answers from whichever ranked first. That is a chunking and metadata defect that presents as a hallucination.

Dates matter for the same reason and are covered properly in Chapter 9. Note here that the date belongs on the chunk at index time, not inferred at query time from a file system timestamp that changed when someone re-uploaded the corpus.

Generated context, and what it measured

Anthropic's contextual retrieval, published 19 September 2024, addresses the orphaned-fragment problem directly: before indexing, prepend a short generated description that situates the chunk within its document.

The numbers on their evaluation are specific. Contextual embeddings alone reduced the top-20 retrieval failure rate by 35%, from 5.7% to 3.7%. With contextual BM25 as well, by 49%. With reranking on top, by 67%. They also note that chunks are usually no more than a few hundred tokens, and that retrieving the top 20 outperformed the top 5 and top 10.

Two practical notes before adopting it. The generation is a one-off cost per chunk at index time, paid again whenever you re-chunk, which is a real bill on a large corpus and a trivial one on a small corpus. And the generated line should be factual situating, of the form "from the refunds section of the 2026 UK terms, following the general refund rule", rather than a summary that introduces claims the chunk does not contain.

Where a corpus already has strong structure, much of the same benefit comes free from the breadcrumb, which costs nothing to generate. Measure both before paying for the model pass.

Different corpora want different units

There is no single right chunk. There is a right unit per document type, and most corpora contain several types.

Reference documentation splits cleanly at headings, and a section is usually the correct unit because it was written to be read alone.

Contracts and policies split at numbered clauses, with the parent clause repeated as context, because a sub-clause without its parent is the orphaned-exception failure in its purest form.

Support tickets and conversations split by exchange rather than by size. The customer's description and the resolution belong together, and the intervening back and forth usually does not.

Code splits at function or class boundaries, with the file path and the imports kept as breadcrumb, because a function body without its imports is ambiguous about which library it calls.

Tabular data is the case that resists chunking altogether. Often the right answer is not to chunk it but to query it: keep the table in a store that can be filtered, and retrieve rows on demand rather than embedding a grid of numbers as prose.

Mixing all of these under one splitter is the usual state of a corpus that grew, and it is why a per-type chunking policy beats a global setting.

Size is a distribution, and the tail is information

Teams report chunk size as a single number, which hides the part worth knowing.

Plot the distribution instead. A corpus segmented on structure produces a spread: many short chunks from list items and headings, a mass in the middle, and a tail of long structural units that resisted splitting. The tail is the interesting part. It is usually large tables, long procedures, or documents with no internal headings at all, and each of those is a modelling decision waiting to be made.

Very short chunks deserve attention too. A chunk of one line has almost no retrievable signal on its own, and it is the case the breadcrumb rescues. Merge trivially short structural units with their neighbours, rather than indexing a heading as a chunk.

The size choice interacts with everything downstream. Longer chunks mean fewer, richer units and more tokens per selected item, which fills the budget in Chapter 7 faster. Shorter chunks mean more precise retrieval and more distractors, and Chroma's Context Rot report of 14 July 2025 measured a single distractor lowering performance, with the effect growing at longer inputs.

Test chunking against questions, not against intuition

Chunking looks like a matter of taste until it is measured, at which point it becomes an ordinary engineering choice.

The test is the labelled query set from Chapter 3. Re-chunk into a second index, run the same fifty questions, and compare recall at 20. Do not compare average chunk quality. Compare whether the questions get answerable evidence.

Add one check that catches the failure this chapter opened with. For each question whose correct answer depends on a qualifier, assert that the qualifier is present in the retrieved set, not merely that the right document was found. That is a different assertion and it is the one that fails when a rule is split from its exception.

Watch the cost side too. A chunking change alters the number of chunks, the index size, the embedding bill and the tokens per selected item. A scheme that improves recall by two points while doubling the corpus is a trade to make deliberately.

Re-chunking is a migration, not a config change

Changing chunk boundaries invalidates every embedding, every stored identifier, and every evaluation number you have.

Plan it as a migration. Build the new index alongside the old one, keep chunk identities stable enough that a labelled query set can be re-scored, and compare recall at 20 on the same questions before switching. A chunking change that improves your offline metric by a point while breaking the mapping to your labelled set has not been measured at all.

Keep the raw documents and re-derive chunks from them, rather than editing chunks in place. Chunks are a projection of the source, and treating them as the source is how a corpus becomes impossible to re-model later.

One security note, because chunking touches it. Permissions belong to the source document and must be copied onto every chunk derived from it. OWASP's 2025 guidance on vector and embedding weaknesses names unauthorised access and cross-context leakage as primary risks, and a re-chunk that drops permission tags creates both silently.

Chapter summary

A chunk boundary decides what a claim means, so a fixed character count is a document-modelling decision made by accident, and it punishes rules with exceptions, tables split from their headers, and figures separated from their units and conditions. Segment on the document's own structure first, using headings, clauses, list items and code boundaries, and apply size limits only within a structural unit, never across one, with tables keeping their header on every piece and list items keeping their introducing sentence. Attach a breadcrumb of document title, section path, effective date and scope identifier to every chunk, stored as metadata for filtering and rendered into the text for the model, because the version identifier in particular prevents an answer drawn confidently from the wrong edition of the same document. Prepending generated situating context measured well in Anthropic's evaluation, cutting top-20 retrieval failure by 35% alone and 49% with lexical retrieval, at a one-off indexing cost that is repaid or not depending on corpus size, and it should situate rather than summarise. Report chunk size as a distribution and read the tail as a list of unresolved modelling problems, merging trivially short units and remembering that shorter chunks buy precision at the price of more distractors, which Chroma measured as a real cost. And treat re-chunking as a migration with a parallel index, a re-scored labelled set and permission tags copied from the source document, since a re-chunk that drops them creates a leak that nothing downstream will notice.

The evidence is now well formed and correctly bounded. What remains is where to put it. Chapter 7 is Assembling the Window: Order, Budget, Compaction, which turns the position effect into an ordering rule, gives each section of the prompt an explicit token allowance, and treats compaction as something that must be reversible.