Checkpoints: What an Agent Must Be Able to Resume From

Checkpoints are what turn a failed run into a resumed one. The word is familiar and the practice is usually wrong in a specific way: teams save the agent's conversation and lose the record of what it did to the world, which is the only part that cannot be regenerated.

Key takeaways

• A checkpoint is not a memory dump. Record what cannot be recomputed: the plan version, completed steps with results, effects emitted with their external references, and budget spent.
• Do not record derived context. A working summary rebuilt from step results is cheaper to recompute than to keep consistent, and a stale copy of it is a source of wrong state in its own right.
• The external reference is the field that matters most. Without the provider's identifier for the message you sent, resumption cannot tell whether it was sent, and cannot undo it either.
• Checkpoint after acceptance, not after execution. A step that ran but failed its check is not a state you want to resume into.
• The test is mechanical: kill the process, resume on a different machine, and compare the outcome. Anything that changes is state you were holding in memory and did not know about.

Read this beside Chapter 3, whose plan artifact is half the checkpoint, and Chapter 6, which needs the emitted-effects record in order to compensate. Chapter 7 puts the whole arrangement on a durable execution runtime.

The run dies at step twenty-six of forty. It might be a deploy, an out-of-memory kill, a rate limit that exhausted its retries, or a laptop lid closing.

Someone restarts it. The agent begins at step one, re-reads files it has already read, and re-sends two notifications that were already delivered. The second run costs as much as the first and leaves the world in a worse state than either.

What a checkpoint is actually for

Two different jobs hide behind one word, and conflating them produces designs that do neither well.

The first job is resumption: continue from where the run stopped without repeating work or repeating effects. The second is rollback: return to a known-good state after discovering that a step was wrong. Resumption moves forward from a boundary and rollback moves backward to one, and they need the same record.

That record is not the model's conversation. Conversation is derived: given the plan and the step results, a working context can be rebuilt. What cannot be rebuilt is the set of things that already happened outside the process, which is exactly what most implementations fail to record because it lives in someone else's system.

Chapter 2's three kinds of wrong state map onto this directly. In-flight state can be recomputed and does not need saving. Persisted internal state needs a boundary you can return to. Externalised effects need a record of what left the building.

Record what cannot be recomputed, and nothing else

The discipline is a single question asked per field: could this be rebuilt from what else is recorded. If yes, do not store it.

What to leave out is just as important. Derived summaries, retrieved documents that can be fetched again, intermediate reasoning, and any cached view of the world all belong on the recompute side. Each of them is a copy that can go stale, and a stale copy resumed into a live system is indistinguishable from a hallucination while being entirely the architecture's fault.

There is one exception worth allowing. Retrieved content that the run acted on should be recorded when the source is mutable, because the decision was made against what it said at the time. That is provenance rather than caching, and it belongs in the record for the same reason the external reference does.

The external reference is the field nobody adds first

When a step sends an email, creates a ticket, charges a card or posts a webhook, the provider returns an identifier. That identifier is the most valuable single item in the checkpoint.

Without it, a resumed run faces a question it cannot answer: did this already happen. The choices are to send again and risk a duplicate, or to skip and risk an omission, and neither is acceptable in a system handling money or customer contact.

With it, three things become possible. Resumption can verify the effect rather than guess. Compensation has a handle to act on, which is what Chapter 6 needs. And the run's audit trail becomes real, which matters the first time somebody asks what the agent actually did.

Store the identifier, the time, the target, and the request payload hash. The hash is what lets a resumed run detect that the same logical effect was already emitted with different arguments, which is a bug rather than a duplicate and should stop the run rather than continue it.

Checkpoint after acceptance, not after execution

The natural place to save is immediately after a step runs. It is the wrong place by one position.

A step that executed but failed its acceptance check has produced a state you do not want to resume into. Checkpointing there records a wrong intermediate state as though it were progress, which is the exact failure Chapter 2 describes, now persisted and authoritative.

So the order is: run the step, capture its effects and references, run the acceptance check, and checkpoint only if the check passes. When the check fails, record the failure and the effects separately, in a form the recovery path can read, and do not advance the resume point.

That distinction has a cost: effects emitted by a failed step still exist and are now recorded outside the resume point. Recording them is not optional. An effect that happened and was not written down is the worst outcome available, because compensation cannot see it and resumption will duplicate it.

Resumption is not the same as replay

Two mechanisms get confused here, and Temporal's model is the clearest way to separate them.

Replay, in the durable execution sense, reruns workflow code from the beginning and re-applies recorded history rather than restoring a memory snapshot. That requires the code to be deterministic, which is why Temporal's documentation warns against clocks, randomness and network calls outside activities: anything that could return a different value on the second pass causes the replay to diverge from the history it is reconstructing.

Resumption, as most agent systems implement it, loads a stored state object and continues from it. It is simpler and it tolerates non-deterministic code, at the cost of everything the checkpoint failed to include.

The two combine well, and the combination is what Chapter 7 recommends. Put the plan walk in deterministic code that can replay. Put every model call and tool call behind an activity boundary whose result is recorded. Then the non-deterministic parts are facts in the history rather than computations to repeat, and a resumed run reaches the same place the original one did.

How often to checkpoint

Frequency is a cost trade-off with an asymmetry worth exploiting.

Checkpoint at every step boundary where an external effect was emitted. That is not negotiable, and it is cheap relative to what those steps already cost.

Between purely internal steps, checkpoint by cost rather than by count: whenever the work that would be lost exceeds the cost of writing the record, which for model-driven steps is almost always immediately. A step that consumed a large inference call is worth persisting even if the result is small.

There is a rhythm consideration that shows up on long runs. METR's Time Horizon 1.1 measurements from January 2026 include tasks over eight hours, and a run of that length will meet a deployment, a token rotation or a dependency restart. Assume interruption rather than treating it as exceptional, and the checkpoint frequency question answers itself.

Where the record lives, and who may read it

A checkpoint is a durable copy of everything the run knows, which makes it a storage decision and a security decision at the same time.

Storage first. The record has to outlive the process, so in-memory structures and local files fail immediately on the machine-loss case. A row per run with an appended event log per step is enough, and the append-only shape is worth insisting on: it gives you the revision history from Chapter 3 for free, and it makes concurrent writers detectable rather than silently last-write-wins.

Security second, and it is the part that gets skipped. Step results routinely contain whatever the agent read, which on a real workload means customer records, credentials pulled from a secret store, and the contents of documents the requester may not be entitled to. That material is now sitting in a table with a different access model from the systems it came from, retained for as long as the run history is retained.

Three controls cover most of it. Record references rather than payloads where the source system can be re-read under the original permissions. Redact known secret shapes on write rather than on read, because a redaction applied at display time protects nobody with database access. And set a retention period for run records deliberately, rather than inheriting the default of forever, since a checkpoint store is an unusually rich target and its value to an attacker grows with every run.

The test that finds what you forgot

Design review will not find the missing field. A specific test will.

Start a run. Kill the process at a random step. Move the record to a different machine with a cold cache and no local files, resume, and let it finish. Then compare the outcome, the effects emitted, and the total cost against an uninterrupted run of the same input.

Anything that differs is state you were holding somewhere you did not intend. Run it at three different kill points, including one immediately after an external effect and before its checkpoint, because that is the interval where duplicates are born.

Make it a scheduled test rather than a one-off. Checkpoint completeness decays as features are added, and it decays silently, since nothing fails until a real interruption arrives. The reliability-science work from March 2026 makes the underlying point in a different context: what degrades on long horizons is not usually the thing that fails on short ones, so short-run tests will not catch this.

Chapter summary

A checkpoint serves resumption and rollback from the same record, and the record is not the model's conversation, because conversation is derivable while the effects that left the process are not. Store the plan version and its revision events, the completed steps with their results, every emitted effect with the external reference the provider returned, the idempotency keys issued, and the budget consumed. Leave out summaries, retrieved documents and intermediate reasoning that can be rebuilt, with one exception for content from a mutable source that the run acted on, which is provenance rather than caching. The external reference is the field teams add last and need most, because without it a resumed run cannot tell whether an effect already happened and compensation has no handle to act on. Checkpoint after the acceptance check rather than after execution, so a step that ran and failed its check never becomes a resume point, while still recording the effects that step emitted, because an unrecorded effect is worse than a failed one. Distinguish resumption from replay: replay reruns deterministic code against recorded history, which is why Temporal forbids clocks and randomness outside activities, and the productive combination puts the plan walk in replayable code with model and tool calls behind recorded activity boundaries. Checkpoint always after an external effect and otherwise by cost, assuming interruption on any run long enough to matter. Keep the record append-only and outside the process, and treat it as the sensitive store it is, holding references rather than payloads where the source can be re-read, redacting secrets on write, and setting a retention period on purpose. Then find the fields you forgot with a kill-and-resume test on a cold machine, run at several kill points, on a schedule.

The checkpoint tells you where you were. It does not tell you whether the step you just took was right. Chapter 5 is Verification Gates Between Steps: what a cheap check looks like when the thing being checked is a model's judgement, how to price a gate against the cost of the commitment it guards, and why self-assessment is the one form of verification that reliably fails.