Memory: What to Write, What to Forget

Memory is usually added to a system as a feature and almost never as a policy. The storage question is easy and the write question is the one that decides whether the product improves or slowly fills with confident, stale, unattributable statements about your users.

Key takeaways

• Decide what may be written before choosing where to store it. Durable facts that a user would expect to be remembered qualify; inferences, moods, one-off mentions and transient state do not.
• Every record carries provenance, confidence and an expiry. A memory without a source cannot be checked, and a memory without an expiry becomes permanent by default.
• Prefer to forget. A smaller memory of verified facts outperforms a large memory of plausible ones, and one 2026 framework reports memory scaffolds degrading long-horizon performance in their tests.
• Contradictions need a resolution rule decided in advance. Latest wins is the usual default and it is wrong for facts a user set deliberately.
• Memory is a data store with none of the access controls of the systems the data came from, which makes retention and scope a security decision.

Read this beside Chapter 7, where memory competes for the same budget as evidence, and Chapter 9, which puts provenance and permissions on everything in the window. Chapter 10 treats memory as untrusted input, which it is.

A user mentions in passing that they are travelling in Berlin next week. Six months later, the assistant greets them with a suggestion about Berlin.

It felt like personalisation when it was written. It reads as unsettling now. The record has no expiry, no source, and no way for anyone to see why the system believes it.

The write policy is the design

Most memory implementations begin with a vector store and a summariser, and the question of what deserves to be written is answered implicitly by a prompt that says "extract important facts".

Invert that. Start with the classification, and let it decide storage.

Durable facts about a person or account belong in memory: a stated preference, a role, an account configuration, a constraint the user gave you. They change rarely and a user would be surprised to have to repeat them.

Task state does not. The current step, the file being edited, the pending confirmation are all run state, and Chapter 4 of the reliability argument in the long-horizon reliability book is where they belong.

Inferences deserve their own category and the strictest treatment. "The user seems frustrated" is a model's reading, not a fact, and once written it will be read back as though someone had established it.

And derived summaries of documents do not belong in memory at all. They belong in the index, where they can be re-derived when the document changes.

Every record needs five fields

A memory entry that is only text cannot be audited, corrected or expired.

The fact itself, stated as a short declarative sentence, in the user's terms rather than a summary of the conversation. The source, meaning the conversation and turn where it was established, so a human can read the original. The confidence, being whether the user stated it, confirmed it, or the system inferred it. The written date. And the expiry.

Expiry is the field that gets omitted and the one that prevents the Berlin problem. Give each class of fact a default lifetime: stated preferences long, account configuration until it is contradicted by the system of record, plans and travel short, anything inferred shortest of all.

Expiry should be enforced at read time as well as by a cleanup job. A record past its date is not returned, whether or not anything has deleted it yet, because the cleanup job is the part that breaks silently.

Write less than you can

The instinct is that more memory produces more personalisation. The evidence points the other way, and so does the arithmetic of Chapter 7's budget.

Khanal, Tao and Zhou, in their March 2026 reliability framework, report that memory scaffolds universally degraded long-horizon performance in their tests. That is one result on one set of tasks and it should not be over-read. It is also consistent with what Chroma measured in July 2025 about distractors: a single distractor lowers performance, and the effect grows with input length. A memory system that injects twenty marginally relevant facts into every window has manufactured twenty distractors.

So set a hard ceiling on how many memory records may enter a window, and make them compete on relevance to the current request rather than on recency alone. If only two facts are relevant, send two.

The stronger discipline is at write time. Require that a fact be stated by the user, or confirmed by them, or read from a system of record, before it is written at all. Everything else is a candidate that can be re-established by asking, and asking is cheap compared to being confidently wrong.

Who may write, and when

Two more decisions belong in the policy, and both are usually made by default.

Who writes. A model extracting facts from a conversation is one option, a deterministic rule triggered by a form submission or an account change is another, and a user editing their own profile is the third. Prefer the deterministic paths where they exist, because a fact captured from a settings change is exact and a fact extracted from prose is a guess with good manners.

When to write. Writing during the conversation is tempting and produces records from statements the user later retracts within the same session. Writing at the end of a session, after a short review pass over what was proposed, is cheaper and more accurate. Nothing about memory is urgent.

Add a confirmation step for anything consequential. When a fact would change future behaviour visibly, such as a delivery address or a working language, the cheapest correctness mechanism available is a single sentence asking the user to confirm before it is stored.

And log every write with its trigger. When a user asks why the system thinks something about them, the answer should be a row rather than an investigation.

Contradictions need a rule, decided in advance

Memory accumulates conflicting statements as a matter of course. People change their minds, correct themselves, and describe the same thing differently.

Latest wins is the common default and it is wrong in one important case: a deliberately set preference should outrank a casual later mention. If a user explicitly configured a notification setting in March and mentioned something in passing in July, the March statement is the one to trust, and the July one is a candidate for confirmation rather than an overwrite.

A workable rule set has three tiers. Facts from a system of record beat everything, and should be read live rather than remembered. Explicitly stated or confirmed preferences beat inferred ones regardless of age. Between two facts of equal standing, the newer one wins and the older one is retained as history rather than deleted.

Retaining history costs little and answers the question that arrives during an incident, which is when the system started believing something wrong.

Memory is a data store, with all that implies

The moment you write a durable record about a person, you have created a data store with its own retention, access and disclosure profile, and usually with none of the controls of the systems the data came from.

OWASP's 2025 entry on sensitive information disclosure covers the general risk, and the vector and embedding entry adds the specific one: cross-context leakage between users in shared stores, and inversion attacks that can recover source text from embeddings. A memory store keyed carelessly, or embedded into a shared index, is the clearest route to one user's facts appearing in another user's window.

Four controls cover most of it. Scope every record to a subject and an audience, and filter on both at read time rather than trusting the query. Redact secrets and credentials at write time. Set a retention period per class of fact, deliberately. And give users a way to see and delete what is remembered about them, which is both a legal requirement in many jurisdictions and the fastest way to discover what your write policy is actually storing.

Two shapes of memory, and when each is right

Storage follows the write policy, and two shapes cover nearly every product.

A keyed store of structured facts is the first. One row per fact, with the five fields from the previous section, looked up by subject and by category. It is boring, inspectable, easy to correct and easy to explain to a user. Most products need only this.

A semantic store of episodes is the second. Past conversations or events, embedded and retrieved by similarity when the current request resembles them. It is useful where the value is in recalling a situation rather than a fact, such as "how did we handle this last time".

The mistake is using the second for the first. Facts retrieved by similarity arrive inconsistently, because whether the system remembers your timezone should not depend on how you phrased today's question. Facts belong in the keyed store, retrieved deterministically for every request that could need them.

Where both exist, retrieve them separately and give them separate budgets in the window. They fail differently, and a shared budget means episodes crowd out facts on exactly the requests where the facts mattered.

Test memory by replaying conversations

Memory defects are hard to see in a single session and obvious across many, which suggests the test.

Take a set of real multi-session conversations, replay them through the write policy, and inspect what the system chose to remember. Read the resulting records as a person would. The ones that make you uncomfortable are the policy defects.

Then run the counterfactual: for each remembered fact, would the next session have gone worse without it. That question separates memory that earns its place from memory that merely exists, and it is usually surprising how few records pass.

Add two standing metrics. The write rate per conversation, which should be low and stable, and the read-and-use rate, meaning how often a retrieved memory actually appears in the answer. A high write rate with a low use rate is a store filling with material nobody needs, paid for in every window it enters.

Chapter summary

Memory is a write policy before it is a storage choice, so classify first: durable facts about a person or account belong in memory, run state belongs with the task, inferences deserve the strictest handling because they read back as established, and document summaries belong in the index where they can be re-derived. Every record needs five fields, being the fact in the user's terms, the source turn, the confidence, the written date and an expiry, with expiry enforced at read time rather than trusted to a cleanup job. Write less than you can, since one 2026 reliability framework found memory scaffolds degrading long-horizon performance and Chroma measured single distractors lowering it, so cap how many records may enter a window and require that a fact be stated, confirmed or read from a system of record before it is written at all. Decide the contradiction rule in advance, with systems of record beating memory, explicit preferences beating inferences regardless of age, and newer beating older among equals while the older is retained as history. Treat the store as the data store it is, with subject and audience scoping enforced at read time, secrets redacted on write, a deliberate retention period, and a user-facing view and delete, because OWASP's 2025 guidance names cross-context leakage and embedding inversion as real routes. And test by replaying real multi-session conversations, reading what was written as a person would, asking whether each fact changed the next session, and tracking write rate against read-and-use rate.

Every fact in memory and every chunk from retrieval arrives with the same three questions attached: where did this come from, how old is it, and who is allowed to see it. Chapter 9 is Provenance, Freshness, and Access Control Inside Context, which keeps those answers attached to the evidence all the way into the prompt.