Caching Without Serving the Wrong Answer Faster

Caching Without Serving the Wrong Answer Faster is the discipline of making sure a cache hit means what it claims. A cache asserts that two requests are equivalent. Every field the answer depends on and the key omits is a way to hand somebody a confidently wrong response in two milliseconds instead of two seconds.

Key takeaways

• A cache hit asserts that two requests are the same request. For an AI feature the answer usually depends on the asker, their tenant, their permissions and the corpus, none of which appear in a naive key.
• RFC 9111 defines the cache key as request method plus target URI, extended by the header fields named in Vary, and forbids a shared cache reusing a response to an authenticated request unless a directive explicitly allows it.
• Put the principal, the tenant, the policy revision and the corpus revision in the key. If any of those changes the correct answer, it belongs in the identity of the cached item.
• Semantic caching trades correctness for hit rate by design, because near-match is not match. Zhang and colleagues report in a January 2026 arXiv paper that the locality which raises hit rate is fundamentally in tension with collision resistance.
• A cache with no invalidation story is a stale-answer generator with good latency numbers. Decide what expires an entry before you decide what populates it.

Read this after Chapter 8, which set the latency budget a cache is bought to defend, and before Chapter 10, which is the other place a system quietly recycles its own output. What goes into the prompt, and therefore into the key, is the subject of the context-assembly argument.

The cache goes in on a Thursday and the latency graph looks wonderful. Hit rate settles at thirty-one per cent, cost drops by roughly the same, and somebody puts it in the sprint review.

Nine days later a customer sees an answer that mentions a supplier they have never worked with.

The key was a hash of the normalised question. Two customers had asked the same question about their own data, and the second one got the first one's answer, faster than either of them could have got their own.

A cache is a claim that two requests are the same request

The entire discipline follows from taking that sentence literally.

Caching is not primarily a performance technique. It is an equivalence assertion, and the performance is what you get for being right about it. When the assertion is wrong, the cache does not degrade gracefully. It returns a confident answer to a question it was not asked.

For deterministic services this is usually safe, because the equivalence is easy to state. The same URL with the same parameters produces the same document. The key is obvious and the failure mode is staleness rather than substitution.

For an AI feature the equivalence is much harder, because the answer depends on things absent from the request. Who is asking. What they are allowed to see. Which tenant's documents were retrieved. Which policy version was in force. Which revision of the corpus retrieval ran against. All of that shapes the answer. None of it is in the question text.

That is the whole problem. The key is shorter than the dependency.

The cache key is the design object, and there is a standard for it

Web infrastructure solved a version of this decades ago, and the specification is worth reading rather than reinventing.

RFC 9111, the IETF's HTTP caching specification published in June 2022 and edited by Fielding, Nottingham and Reschke, defines the cache key as the information a cache uses to choose a response. At minimum that is the request method and the target URI. Where a response varies by negotiation, caches incorporate some of the original request's header fields into the key as well, using the Vary response header field.

The mechanism is the one to steal. The origin declares which parts of the request the response depended on, and the cache must include them in the identity of the stored item. Identity is asserted by the producer, not guessed by the consumer.

Applied to an AI feature, the code producing a cacheable answer should also produce the list of things the answer depended on. The cache layer should refuse to store anything undeclared. A cache API that takes a key and a value and asks nothing else is an invitation to the failure above.

Three kinds of cache, and what each one can get wrong

The three approaches differ in how loose the equivalence claim is, and therefore in how wrong they can be.

The fourth row is the recommendation buried in the table. Caching retrieval results, the parsed document, the extracted structure or the embedding is usually a larger win than caching the final answer. Those stages are deterministic given their inputs, their equivalence is easy to state, and a stale one is visible rather than silent.

Normalisation deserves one warning. Harmless-looking rules erase meaning: stripping punctuation can merge a question with its negation, lowercasing can merge an identifier with a word. Every normalisation rule is a claim that a distinction does not matter. Write it down as one.

The principal belongs in the key

The most damaging cache bug in a multi-tenant AI feature is cross-tenant substitution, and RFC 9111 names the general form of it.

The specification is explicit. A shared cache must not use a cached response to a request carrying an Authorization header field to satisfy any subsequent request, unless the response contains a directive allowing shared storage, such as must-revalidate, public or s-maxage. The default is that authenticated responses are not shared. You opt in deliberately.

An AI answer grounded in retrieved documents is authenticated content by that standard, even when the HTTP layer around it is not. Retrieval ran with somebody's permissions, so the answer encodes what that person was allowed to see. Reusing it for a different person is an authorisation decision made by a cache.

So the key needs the tenant identifier, and usually the principal too. Per-tenant scoping stops the worst class of leak. Per-user scoping is required wherever permissions vary within a tenant, which is most enterprise products.

There is a cost to admitting this. Scoping to the principal collapses your hit rate, because two users asking the same question now get two entries. That is not a flaw. That is the honest hit rate, and the earlier number was measuring a bug.

Policy and corpus revision belong in the key too

Two more fields are routinely missing, and both produce one symptom: an answer that was correct when it was computed and wrong when it is served.

The corpus revision is the first. If a grounding document has since been updated, superseded or deleted, the cached answer cites something no longer true. So the key needs a version identifier for the corpus, or for the partition retrieved from. The cheapest workable form is a monotonic revision counter that increments on any write to the index.

The policy revision is the second, and it is forgotten more often. The system prompt, the tool definitions, the guardrail configuration, the model version and the temperature all shape the output. An answer generated under last month's policy is not the answer the current system would give. Serving it makes the policy change look like it never took effect.

Model version is the sharpest example. A cache that survives a model upgrade serves the old model's answers under the new model's name, until the entries expire. That makes the upgrade impossible to evaluate.

The rule is one line. If changing it would change the correct answer, it belongs in the key.

Semantic matching buys hit rate with correctness

Semantic caching is the technique people reach for when the exact and normalised hit rates disappoint, and it deserves precision about what it is doing.

The technique is established. Fu Bang's GPTCache paper at the NLP-OSS workshop in 2023 describes an open-source semantic cache that stores model responses and serves them for semantically similar queries, reporting substantially faster responses on hits. The mechanism is an embedding of the query used as the key, with a similarity threshold deciding what counts as a hit.

The trade is structural rather than incidental. Zhang and colleagues, in an arXiv paper first posted on 30 January 2026, conceptualise semantic cache keys as fuzzy hashes. They argue that the locality needed to maximise hit rate fundamentally conflicts with the collision resistance you would want from a key. They report an automated black-box collision attack achieving an 86% hit rate in hijacking model responses in their evaluation, presented as the first systematic study of integrity risk from cache collisions rather than privacy risk.

Read that as an argument about the mechanism rather than only about attackers. If an adversary can find a colliding query on purpose, an ordinary user will find one by accident. The accidental version arrives with no alert attached.

That gives a usable rule. Semantic caching is defensible where a near-match answer is still acceptable, such as documentation lookup or a general explanation. It is indefensible where the answer turns on an entity, a number or a date. Those are exactly the distinctions an embedding threshold is built to blur.

Invalidation decides whether the cache was worth it

A cache is easy to populate and hard to expire, and the expiry design is where most of the value actually sits.

Three strategies exist and they compose. Time-based expiry is the crude default, and choosing the window is choosing how wrong you are willing to be: an hour of staleness on a documentation answer is fine, an hour on an account balance is not. Event-based invalidation keys off writes, so updating a document evicts everything grounded in it, which requires recording that dependency at write time. And revision-based keys avoid invalidation entirely, by making the old entry unreachable rather than deleting it.

Revision-based keying is what I would reach for first, because it degrades safely. A stale entry is never served. It is simply never matched again, and it ages out.

Whatever you choose, instrument it. Hit rate alone is a vanity metric. The pair worth watching is hit rate against the rate at which users correct a cached answer relative to a fresh one. If cached answers get corrected more often, the cache is buying latency with quality, and you now know the exchange rate.

A worked example, composited and labelled

The details here are composited from ordinary production shapes rather than one system, and the shape is exact.

An internal support assistant answers questions from a mixed corpus: public product documentation, tenant-specific configuration and customer contracts. The first cache keyed on the normalised question and reported a thirty-one per cent hit rate.

The rebuild split the cache by what the answer depended on. Questions answered purely from public documentation cache on the normalised question plus corpus revision plus model version, shared across all tenants, and that layer keeps a high hit rate honestly. Questions touching tenant configuration cache on the same key plus the tenant identifier. Questions touching contracts cache on the same key plus the principal identifier, because contract visibility varies by role.

They also added an intermediate cache that nobody had considered, holding parsed and chunked documents keyed by content hash. That one saved more compute than the answer cache ever had, because reprocessing the same 400-page contract had been the real expense.

Reported hit rate fell from thirty-one per cent to about nineteen. Cost fell further than before, and the cross-tenant class of bug became unreachable rather than unlikely.

The objection: this kills the hit rate and the savings with it

The objection is arithmetically correct, so it deserves arithmetic rather than a principle.

The first response is that the original number was not a savings figure. A hit rate inflated by keys that omit the principal measures how often the system serves the wrong person's answer. No cost model should count that as a saving.

The second is that the savings usually survive relocation. Move the caching down to the deterministic stages, meaning parsing, chunking, embedding and retrieval, where equivalence is easy to state and hits are frequent. You keep most of the compute saving and lose the correctness risk. The expensive part of many features is not the final generation.

The third is that hit rate is the wrong objective. What the business wants is cost per successful interaction from Chapter 7 and perceived latency from Chapter 8. A smaller honest cache in front of expensive deterministic work moves both. A larger cache that occasionally substitutes answers moves one and creates an incident class.

Chapter summary

A cache hit asserts that two requests are equivalent, and the performance benefit is what you get for being right. A wrong assertion does not degrade gracefully. It returns a confident answer to a question that was not asked. For AI features the equivalence is hard, because the answer depends on the asker, their permissions, the tenant, the policy in force and the corpus revision, none of which appears in the question text. The key is routinely shorter than the dependency. RFC 9111 supplies the pattern: the cache key is method plus target URI, extended by the request header fields the origin names in Vary, so identity is declared by the producer rather than guessed by the consumer. It also forbids a shared cache reusing a response to an authenticated request unless a directive explicitly permits it. Exact, normalised and semantic caching loosen the equivalence claim in turn, with normalisation rules quietly erasing negations and identifiers. The underrated option is caching intermediate deterministic stages such as parsed documents, chunks, embeddings and retrieval results, where equivalence is easy to state. The principal and tenant belong in the key, because a grounded answer encodes what that person was permitted to see, and honest scoping collapses the hit rate to its real value. Policy revision, model version and corpus revision belong there too, since an answer correct under last month's configuration is wrong today, and a cache that survives a model upgrade makes the upgrade unmeasurable. Semantic caching trades correctness for hit rate structurally, as Zhang and colleagues argue when they put locality and collision resistance in direct conflict, so it suits general explanation and not entity-specific answers. And revision-based keys beat time-based expiry, because a stale entry becomes unreachable rather than briefly wrong.

A cache recycles the system's own output for speed. The next chapter concerns recycling it for improvement, which has a subtler failure mode and a published name. Chapter 10 is Data Flywheels That Do Not Poison Themselves.