Durable Execution: Workflows, Retries, Idempotency

Durable Execution is what makes the previous three chapters real. A plan, a checkpoint and a compensation are all instructions to a process, and a process is the least durable thing in the system. This chapter is about surviving its death without losing the run.

Key takeaways

• Split the system in two: deterministic orchestration that can be replayed, and non-deterministic activities whose results are recorded as facts. Every model call belongs on the activity side.
• Determinism is the price of replay. Clocks, random values and network calls inside orchestration code will diverge on the second pass and corrupt the reconstruction.
• Issue the idempotency key before the call, derive it from the step identity rather than from a timestamp, and persist it in the same write that records the intent.
• Retry transport, never judgement. A retry policy belongs on the activity that talks to a network, not on the step that decided what to say.
• Long runs meet deployments. On tasks measured in hours, the process restarting is a normal event to design for rather than an incident to be surprised by.

Read this beside Chapter 4, whose checkpoint record this chapter formalises, and Chapter 6, whose compensations run as activities like anything else. The applied version of this argument, outside book form, is in durable execution for long-running agents.

A deployment goes out at four in the afternoon. Somewhere in the fleet, a container holding an agent run at step twenty-six of forty receives a termination signal and has thirty seconds to exit.

The agent has no opinion about this. It has no callback. It has no shutdown hook that means anything, and its plan lives in a variable. Thirty seconds later the work is gone. Nobody logged what it had already done.

The two-sided split

Every durable system draws the same line, and it is worth stating in general terms before naming any product.

On one side sits orchestration: which step runs next, what its inputs are, what to do when it fails. This code must be replayable, which means running it again against a recorded history must produce the same sequence of decisions.

On the other side sit activities: everything that touches the world or produces an unpredictable value. Network calls, database writes, file operations, and every inference call. Activities are executed once, and their results are written into the history. On replay, the orchestration does not re-execute them; it reads what they returned.

Temporal describes exactly this shape, defining durable execution as the guarantee that an application runs to completion despite adverse conditions, with activities as the individual units of work that interact with the outside world and automatic retries configured per activity. The framing is worth adopting even if you implement it yourself with a database and a queue.

The critical placement decision for agents is the model call. It is non-deterministic by nature, so it is an activity, and its output is a recorded fact rather than something to be regenerated. Teams that put inference inside orchestration code discover the problem on their first replay, when the model returns a different plan and the reconstruction diverges from the history it was supposed to be rebuilding.

Determinism, and what it forbids

Replay means running the same code against the same history and expecting the same decisions. Anything that could return a different value on the second pass breaks it.

Temporal's documentation is explicit about the constraint, warning against relying on values not in the event history, such as direct calls to the current time, random numbers, or network calls made outside activities, because those can produce different results on replay and cause a workflow to diverge from its recorded history. Its replay model does not restore a memory snapshot: it starts the code from the beginning and re-applies the recorded events step by step.

The practical rules that follow are short. Read the current time through the runtime. Never the language's clock. Generate identifiers and random values inside activities, so each value becomes part of the history. Never call a service from orchestration code. And never branch on anything that is not an input or a recorded result. A local cache counts as neither.

There is one that catches agent teams specifically. Do not branch on a model's output that you did not record. If the orchestration asks a model whether to continue and uses the answer without writing it to history, the replay will ask again and may receive a different answer, silently taking a different path through a run that has already happened.

Idempotency is the part that gets skipped

Durability guarantees that a run continues. It does not guarantee that an external effect happens once, and those are different problems.

The dangerous interval is short and unavoidable. It sits between an activity emitting an effect and its result being recorded. A crash there leaves the world changed and the history unaware. On resume, the activity runs again. Without protection, the customer is charged twice.

An idempotency key closes it, and the details matter more than the concept. Derive the key from the identity of the step and the run rather than from a timestamp or a random value, so a resumed attempt produces the same key rather than a new one. Persist it before the call, in the same write that records the intent to act. Send it to the provider on every attempt, including retries. And keep the provider's response against that key, so a later attempt can distinguish a duplicate from a new request.

Where a provider offers no idempotency support, build the equivalent locally: record the intent with its key before acting, check for a completed record with the same key before every attempt, and accept that a narrow window remains. Then close that window by making the effect itself checkable, which usually means a read-back gate from Chapter 5 immediately after.

Assume at-least-once delivery everywhere, in both directions. Your calls may arrive twice, and so may the callbacks you receive. Deduplicate inbound events by the provider's event identifier before they touch any state, because a webhook delivered twice is the same class of defect as a payment sent twice, arriving from the other side of the wire.

Retry policy belongs to the activity, not the step

Chapter 2 argued that retrying a judgement failure produces the same wrong answer again. Durable execution makes that rule structural rather than advisory.

Retries are configured per activity, so a timeout on an HTTP call retries with backoff, while the step that decided what to send does not. If a model call fails because the provider returned a 503, that is transport and it retries. If the model returns a confident wrong plan, no retry policy in the runtime will help, and the mechanism that catches it is a gate.

Three settings do most of the work. Bound the attempts. Unbounded retries turn a provider outage into an unbounded bill. Use exponential backoff with jitter, so a fleet of agents does not synchronise its retries into a second outage. And keep a non-retryable error list, so a 400 or a validation failure fails fast instead of being attempted six times.

One more setting is worth naming because it is invisible until it hurts. Cap the total elapsed time of the retry sequence, not only the attempt count, so a policy of six attempts with backoff cannot quietly hold a step open for an hour while the run's wall-clock budget from Chapter 9 believes it is still early.

Set a per-activity timeout too, and set it deliberately. Model calls on long tasks can run for minutes, so a default of thirty seconds will produce spurious retries that each cost a full inference, which is the most expensive way to be wrong about a configuration value.

What the history buys you beyond recovery

The recorded history exists for replay, and it turns out to be the most valuable observability artifact in the system.

It gives you an exact account of what the agent did, in order, with inputs and outputs, which is what Chapter 12's on-call practice reads during an incident. It gives you the trajectory record that failure analysis needs, of the kind used by the 2026 diagnostic work across more than 3,100 trajectories to attribute failures to identifiable steps. It gives you the raw data for the horizon measurement in Chapter 11, since every run's duration, step count and outcome are already there.

Two disciplines keep it useful. Record inputs and outputs, not summaries. A summary is a lossy interpretation written before you knew what question you would ask. And keep the history queryable by step type, so that "how often does this tool call fail" is a query rather than a project.

The cost is storage and sensitivity, which is the same warning Chapter 4 gives about checkpoints. Histories accumulate, and they contain whatever the agent read.

When you do not need a durable runtime

Adopting a full durable execution platform is a substantial commitment, and it is not always the right first move.

Anthropic's December 2024 guidance to start simple and add complexity only when needed applies. If your tasks finish in under a minute, have no external effects, and can be re-run from scratch without cost, a queue with a retry is enough, and the machinery in this chapter is overhead.

The threshold is worth stating concretely. You need durability when any of four things is true: the run outlives a deployment window, it emits external effects, it costs enough that repeating it matters, or a partial run leaves the world in a state somebody would notice. METR's Time Horizon 1.1 measurements from January 2026 include tasks over eight hours, and any run of that length will cross a deployment, so the first condition is met by definition on genuinely long-horizon work.

There is a middle path many teams miss. You can build the two-sided split without adopting a platform: a table of runs, a table of steps with their recorded results, a worker that walks the plan, and idempotency keys on external calls. That is a few hundred lines. It captures most of the benefit. Adopt a platform when operating your own version stops being the smaller job, and not before.

Chapter summary

Durable execution divides the system into deterministic orchestration that can be replayed and activities whose results are recorded as facts, with every model call on the activity side because inference is non-deterministic and its output must be a recorded fact rather than a recomputation. Determinism is the price: no clocks, no random values, no network calls in orchestration code, and no branching on a model output that was not written to history, because Temporal's replay restarts the code and re-applies events rather than restoring a snapshot. Idempotency is a separate guarantee from durability and the one teams skip, so derive keys from step and run identity rather than from time, persist them before the call in the same write as the intent, send them on every attempt, and keep the provider's response against the key, assuming at-least-once delivery in both directions. Retry policy belongs to the activity rather than the step, with bounded attempts, backoff with jitter, a non-retryable error list and per-activity timeouts set for the real duration of model calls. The history then pays for itself twice, giving an exact account for incident response and the trajectory data for both failure analysis and the horizon measurement, provided inputs and outputs are recorded rather than summaries and the store is treated as sensitive. And none of it is needed for short, effect-free, cheaply repeatable work, with the threshold being any run that outlives a deployment, emits external effects, costs enough to matter, or leaves a partial state someone would notice.

The runtime now survives, the plan is recorded, and the effects are bounded. What the agent can actually do is decided by something else entirely. Chapter 8 is Tools Are the Real Interface, which is about designing an API for a caller that will sometimes get the arguments wrong, why error messages are the highest-leverage prompt in the system, and how tool surface area quietly sets your failure rate.