Online Evaluation and Guarded Rollouts

Online Evaluation and Guarded Rollouts is the half of the discipline that only exists in production. The suite measured the distribution you assembled; users send the one that actually occurs, and the difference between those two is what a guarded rollout is designed to survive.

Key takeaways

• Shadow traffic compares a candidate against the current system on real inputs without exposing anyone, and it is the cheapest online measurement available.
• Canary on a fraction of traffic, with the fraction chosen so a bad outcome is bounded, and with a stated duration rather than a vague soak.
• Write the rollback rule in numbers before the rollout starts. A rule agreed under pressure is a discussion, not a rule.
• The fastest signals are behavioural, not qualitative: refusal rate, escalation rate, retry rate, session abandonment, cost per successful outcome. Quality judgements arrive hours later.
• Separate the release of code from the release of behaviour with flags, so a rollback is a configuration change rather than a deploy.

Read this beside Chapter 9, whose gate this extends past the deploy, and Chapter 4, whose three metric families reappear as online signals. Chapter 12 is what happens when the rollback is too late.

The suite is green, the release goes out at eleven, and by two the support queue is unusual. Not broken, unusual: more follow-up questions than normal, and a few customers asking why the assistant has stopped answering something it used to answer.

Nobody can say whether that is the release, because nothing was measured in the two hours after it, and the deploy touched four things at once.

Shadow traffic: measure without exposing anyone

Shadow running sends real production inputs to both the current system and the candidate, serves only the current system's output, and records both.

It is the highest-value online technique because it carries no user risk and produces exactly the comparison you want: the same input, two systems, on the real distribution.

Three things to measure from a shadow run. The diff rate, meaning how often the two outputs differ materially rather than in wording. The direction of the diffs on a sample, judged with the apparatus from Chapters 6 and 7. And the system metrics, since the candidate's cost and latency are visible without any user waiting for it.

Two cautions specific to this class of system. Shadow runs double your inference bill for the sampled traffic, so sample rather than mirroring everything. And be careful with side effects: a candidate that writes, sends or charges must have those actions stubbed, or shadow running becomes a way to do everything twice.

Where a candidate cannot be safely shadowed because it acts, shadow the decision rather than the action. Record what it would have done and compare that to what the current system did.

Canary: expose a bounded fraction on purpose

A canary is the first real exposure, and its design question is how much damage a bad candidate can do before you notice.

Choose the fraction from the blast radius rather than from convention. If a bad answer costs a support ticket, one per cent for an hour is generous. If it can move money or send external communication, the first canary should be internal users only, then a named set of tolerant accounts, then a percentage.

Give it a stated duration and a stated volume. "Soak for a while" produces a rollout that expands when somebody remembers, and the useful form is a number of requests sufficient to detect the effect you care about, which for rare failures is larger than people expect.

Segment the canary deliberately. Random assignment is fine for broad quality, and it will under-sample the hard strata from Chapter 3. Where a specific stratum is the risk, route some of it into the canary on purpose.

Hodgson's feature toggle taxonomy is the right mechanism underneath all this, particularly permissioning toggles for the internal-first stage and ops toggles for the kill switch, with his warning about toggle inventory applying as soon as the rollout finishes.

Write the rollback rule in numbers, in advance

The single practice that distinguishes a guarded rollout from a hopeful one is a rule written before the traffic starts.

It has four parts. The signals to watch, named. The threshold on each, as a number relative to the pre-rollout baseline. The observation window. And the action, which should be automatic where possible and one action where not.

An example shape, with your own numbers: refusal rate more than three points above baseline, or escalation rate up by half, or cost per successful outcome up by a quarter, observed over thirty minutes at one per cent, triggers automatic rollback.

Write who may waive it, and make that a different person from whoever is shipping. In practice the waiver conversation is where rollouts go wrong, because the person with the most context is also the most invested.

Google's error budget framing supplies the policy above the rule: if the budget for the period is already spent, the answer to a marginal rollout is no, and that decision is arithmetic rather than a negotiation.

Watch the signals that move first

Quality signals are the ones you care about and the slowest to arrive. Behavioural signals move within minutes and correlate well enough to act on.

Refusal rate. A candidate that has become more cautious refuses more, and users notice immediately. This is usually the fastest mover after a prompt change.

Escalation and handoff rate, where the product has one. It is a direct measure of the system failing to complete work, reported by users implicitly rather than through a survey.

Retry and rephrase rate. A user rewording the same question within a minute is telling you the first answer was inadequate, and this signal needs no labelling at all.

Session abandonment, and its opposite, session length rising without task completion.

Cost per successful outcome, from Chapter 4, which catches the candidate that is more thorough, more expensive and no better.

Then the slower quality signals arrive: sampled judged scores on real traffic, thumbs and complaints, and the eventual measurement of whether tasks were completed. Use those to confirm or refute what the fast signals suggested, not to make the first decision.

The confound nobody controls for

One methodological warning, because it invalidates more online results than anything else.

Traffic is not stationary. Monday differs from Saturday, morning differs from evening, and a marketing campaign changes the question mix entirely. A before-and-after comparison across those boundaries measures the calendar as much as the change.

Three defences. Compare concurrent groups rather than consecutive periods, which is what canaries and interleaving give you. Where only a before-and-after is possible, compare the same hours on the same weekday. And record what else changed, since a model version update, a corpus refresh and a prompt edit in the same week make the result uninterpretable however carefully you measure.

Ship one behavioural change at a time during a rollout. It is slower, and it is the only way the numbers mean anything.

Sample real traffic into the offline suite

An underused property of online evaluation is that it feeds the offline one.

Sample production interactions continuously, judge a fraction of them with the calibrated judge from Chapter 6, and track the resulting score as a time series. That gives you a quality measurement on the real distribution, which the offline suite by construction cannot provide.

Then promote the failures. Any production case that the judge scores badly, or that a user complains about, becomes a labelled case in Chapter 3's dataset. That loop is what keeps the suite discriminative as Chapter 8 requires, and it costs a labelling session a week.

Two disciplines keep it honest. Sample randomly rather than only sampling complaints, or your time series measures complaint volume rather than quality. And keep the sampling rate stable, since a rate that changes silently makes the series uninterpretable.

Zheng and colleagues' bias findings apply here as much as offline. A judge scoring production traffic carries the same verbosity and self-enhancement biases, so a rising score with rising answer length deserves the probe from Chapter 7 rather than a celebration.

Comparing two systems on live traffic

Sometimes the question is not whether a candidate is safe but which of two candidates is better, and that needs a comparison design rather than a rollout.

Interleaving is the cheapest honest form where the product allows it. For each request, assign a candidate at random, record which was served, and compare outcomes across the two groups on the behavioural signals. Randomisation at the request level removes most of the confounds that plague before-and-after comparisons.

Assign consistently per user where the experience would be jarring otherwise. A user who gets a different assistant personality on alternate messages is a support ticket, so hash the user identifier and hold the assignment for the duration.

Size the comparison honestly. Detecting a small difference in a noisy behavioural signal needs more traffic than most teams assume, and running until the numbers look good is how a null result becomes a launch.

And keep the comparison short. Long-running splits accumulate operational cost, confuse support, and interact with other experiments, which is a cost worth paying for a significant decision and not for a small one.

Separate deploying code from releasing behaviour

The mechanical enabler for all of this is that shipping the code and enabling the behaviour are different events.

Ship the candidate dark, behind a flag. Enable it for internal users. Enable it for one per cent. Watch the rule. Expand. That sequence is only available if the enable is a configuration change, and it collapses into a series of deploys otherwise, each with its own risk and its own delay.

The rollback then costs seconds rather than a build. That difference decides whether a rollback happens at the first sign of trouble or after a discussion about whether it is worth the disruption, and the first is almost always correct.

DORA's failed deployment recovery time, in the current metric set updated in January 2026, is the measurement to watch here. A team that cannot reverse a behaviour change in minutes will not run guarded rollouts for long, because the guard is only useful if acting on it is cheap.

Retire the flags afterwards. A rollout that finished six months ago and left its toggle behind is a live risk with no owner.

Chapter summary

Online evaluation measures the distribution that actually arrives, and it starts with shadow traffic, which runs a candidate on real inputs without serving its output, giving a diff rate, a judged sample of the differences and honest cost and latency figures, provided side effects are stubbed and traffic is sampled rather than mirrored. Canaries then expose a bounded fraction chosen from the blast radius rather than by convention, internal users first where actions are involved, with a stated duration and volume and deliberate routing of the risky strata, built on feature toggles that also supply the kill switch. Write the rollback rule before the rollout, with named signals, numeric thresholds relative to baseline, an observation window and an action that is automatic where possible, and name a waiver holder who is not the person shipping, with the error budget deciding whether a marginal rollout happens at all. Watch the fast behavioural signals first, being refusal rate, escalation rate, retry and rephrase rate, abandonment and cost per successful outcome, and treat the slower judged and complaint-based signals as confirmation rather than as the trigger. Feed production back into the offline suite by sampling randomly at a stable rate, judging a fraction with the calibrated judge and promoting every failure into the labelled dataset, while remembering that judge biases apply to production sampling too. And separate deploying code from releasing behaviour with flags so that enabling is a configuration change and rollback costs seconds, which is what makes a guard worth having, then retire the toggles once the rollout is complete.

Guarded rollouts reduce the size of an incident. They do not prevent one, and the class of incident this book is about is unusual in that the system keeps working while being wrong. Chapter 11 is Tracing With OpenTelemetry GenAI Conventions, which is the instrumentation that makes such an incident explainable, and it starts with a note about where those conventions currently live.