Incident Response for Systems That Are Fluently Wrong

Incident Response for Systems That Are Fluently Wrong is the last chapter because it is where every earlier one is spent. The dataset, the judge, the gate and the traces exist so that when something goes wrong in a way nothing errors on, somebody can find it, stop it, explain it and prevent the next one.

Key takeaways

• The defining property is that nothing failed. No exception, no error rate, no latency change, and a green dashboard throughout, so detection has to come from behaviour and from users rather than from infrastructure.
• Containment has more than one setting. Narrow the scope, disable one capability, force a refusal on the affected class or fall back to a previous behaviour, before considering a full stop.
• Scope the blast radius early and in units a business recognises: how many users, which accounts, over what period, and whether anything left the system.
• Communicate the correction to the people who received the wrong answer, which is a step classical incident response rarely needs and this class of failure usually does.
• The review names a system defect, not the model. The output is a suite case, a gate rule, an owner and a date.

Read this beside Chapter 10, whose rollback is the fastest containment available, and Chapter 11, whose traces are what make explanation possible. Chapter 3's dataset is where the incident ends up.

A pricing change goes live on Tuesday. The assistant keeps answering pricing questions confidently, from the previous quarter's document, because the new one failed to index.

No error was raised. The retrieval returned results, the model used them, the answers were fluent and well-cited. Six days later a customer notices, and by then several hundred people have been told the wrong price.

Detection, when nothing errors

Conventional detectors are all negative in this class of incident, so detection has to be built rather than inherited.

Four sources, in order of speed. Behavioural signals from Chapter 10, being refusal rate, rephrase rate, escalation rate and abandonment, which move within an hour of a behaviour change. Continuous judged sampling of production traffic, which gives a quality series with a lag of hours. Freshness and pipeline monitors, which would have caught the failed index directly. And user reports, which are the slowest and the most certain.

The pipeline monitors deserve emphasis because they catch causes rather than symptoms. Document count in the index, age of the newest document, ingestion job success, and the distribution of retrieved document ages from a companion context volume. Any of those would have raised the pricing incident on Tuesday rather than the following Monday.

Set alerts on the derived signals rather than on the raw ones. Refusal rate is noisy by hour and stable week over week, so alert on a departure from the same hour last week rather than on an absolute threshold.

And keep one manual detector. Somebody reads twenty real interactions every day, which sounds unscalable and is twenty minutes, and it catches the class of problem no automated signal was designed for.

Containment has more than one setting

The reflex is to disable the feature, which is available, expensive and rarely the best first move.

Narrow the scope. Restrict the feature to the query types unaffected by the fault, which is possible when the fault is localised to a corpus area or a task family.

Force a refusal on the affected class. An assistant that declines to answer pricing questions and says why is better than one giving wrong prices and better than one that has vanished.

Fall back to the previous behaviour, using the flag from Chapter 10, which is the fastest option when the cause is a recent change.

Reduce autonomy. Where the system acts, route the affected class through human approval rather than stopping it, which keeps the work moving while removing the risk.

Full stop remains the right answer when the fault is not understood and the potential harm is high. Take it without embarrassment in those cases, and prefer the narrower options whenever the fault is localised.

Scope the blast radius in the units a business uses

The first question from outside engineering is always how bad it is, and the answer has a shape.

How many users received an affected answer, which requires the traces from Chapter 11 joined to the fault's signature. Which accounts, named where the relationship matters. Over what period, with a start time established from the change or the data rather than from the complaint. Whether any of it left the system, meaning emails sent, tickets created, records written, payments taken. And whether any of it is irreversible.

Get the period right, because the instinct is to start the clock at detection. In the pricing example the incident began on Tuesday and was detected on Monday, and the scope is six days rather than one hour.

Write the scope down as it firms up, with the confidence attached. "At least 240 users, likely under 400, between Tuesday 09:00 and Monday 14:00" is a usable statement. "Several hundred" is not, and it will be quoted back later.

Say something to the people who got the wrong answer

This is the step that distinguishes incident response for this class of system, and it is usually not an engineering decision.

A conventional outage inconveniences people and they know it happened. A fluently wrong answer misinforms people who do not know, may have acted on it, and will not find out unless somebody tells them.

So the response plan needs a communication path, decided in advance with whoever owns the customer relationship and, where the content is regulated, with legal. What triggers a proactive correction. Who writes it. What it says. How the affected list is produced from the traces.

Engineering's job in that path is narrow and important: produce an accurate, defensible list of who was affected and what they were told. That list is only available if Chapter 11's tracing exists, which is the strongest practical argument for it.

Explain it with the trace, not with a theory

The investigation for this class of incident has an order that saves hours.

Establish what the system was given before asking why it answered as it did. Which documents were retrieved, which model version ran, what the assembled window contained, which tools were called. Chapter 11's trace answers all four in one query.

Then check the pipeline before the model. Ingestion, indexing, permissions, freshness and configuration change more often than model behaviour does, and they change in ways that produce exactly this symptom.

Then reproduce, using the same inputs and the recorded configuration. A failure that cannot be reproduced from the trace is usually a trace gap rather than a mystery, and that gap is itself a finding.

Only then consider the model. Genuine model-behaviour incidents exist, particularly after a provider-side version change, and they are less common than the pipeline causes that present identically.

AgentLens' emphasis on trajectory reviews that produce readable explanations is the right standard for what the investigation should end with: a narrative of what happened, grounded in the record, rather than a plausible account.

The review names a system defect

Google's postmortem practice is the right frame, with one adaptation. Blameless here means refusing to blame the model as much as refusing to blame a person, because "the model hallucinated" ends an investigation at the point it should begin.

The productive questions are structural. Which suite case would have caught this, and why was it not in the dataset. Which gate rule would have blocked the change, and why did it not fire. Which monitor would have detected it in an hour rather than in six days. And which containment option was available and how long did it take to use.

Each of those has an owner and a date, which is the postmortem practice's core requirement: the output is an action, not a description.

Then close the loop into the suite. Every incident becomes at least one labelled case in Chapter 3's dataset, permanently. That is the mechanism by which a system gets more reliable over time, and it is the reason a two-year-old suite is worth more than a new one.

Who runs it, and what they need at hand

An incident in this class needs two roles that a conventional rota may not have.

An engineer who can read the traces and operate the containment switches, which is the usual on-call function with one addition: they must be able to disable a capability or force a refusal without a deploy.

Somebody who can judge whether an output was actually wrong. That is often not the engineer, particularly where the content is domain-specific, and their absence is why incidents sit unclassified for hours while people argue about whether the answer was acceptable.

Give both a short list of things kept ready: the query that produces the affected request list, the flags and what each one disables, the containment options in order of severity, and the communication path from earlier in this chapter.

Rehearse it once. Take a real past failure, walk it through detection, containment, scoping and communication with both roles present, and time each step. The rehearsal usually finds that one containment switch does not work, which is worth more than the exercise costs.

Measure the response, not just the incident

Four numbers describe how well this works, and they belong in the same monthly report as the gate's health from Chapter 9.

Time to detect, measured from the first affected request rather than from the first complaint. This is the number that shrinks when the detectors in this chapter are built, and it is often days at the start.

Time to contain, from detection to the affected class being safe, which the narrower containment options improve substantially.

Blast radius per incident, in users, which shows whether the guarded rollouts from Chapter 10 are doing their job.

And recurrence, meaning how many incidents were a repeat of something already reviewed. A non-zero recurrence rate says the reviews are producing descriptions rather than changes.

DORA's failed deployment recovery time, in the metric set updated in January 2026, remains the right measure for delivery-caused incidents, and these four sit beside it for the ones no deploy caused.

Chapter summary

An incident in a fluently wrong system has no outage, no error rate and no alert, so detection is built from behavioural signals that move within an hour, continuous judged sampling with a lag of hours, pipeline and freshness monitors that catch causes rather than symptoms, and user reports that are slowest and most certain, with alerts set on departures from the same hour last week rather than on absolute thresholds, and one human reading twenty interactions a day. Containment has more settings than stop: narrowing scope, forcing a refusal on the affected class, falling back to previous behaviour via a flag, or reducing autonomy to human approval, with a full stop reserved for faults that are not understood. Scope the blast radius in business units, being users, accounts, period and whether anything irreversible left the system, dating the start from the change rather than from the complaint and stating confidence bounds. Plan a communication path in advance, because people who received a confidently wrong answer will not otherwise learn of it, and engineering's part is producing an accurate affected list from the traces. Investigate in order: what the system was given, then the pipeline, then reproduction from the recorded configuration, then the model last, since ingestion, indexing, permissions and freshness change more often than model behaviour and present identically. Review blamelessly, which here means refusing to blame the model, and produce four structural answers with owners and dates covering the missing suite case, the missing gate rule, the missing monitor and the containment latency, then add the incident to the dataset permanently. And measure the response itself with time to detect, time to contain, blast radius and recurrence.

That is the book. Its argument in one line: a probabilistic system cannot be shipped on taste, so build the apparatus that can say no, which is a definition of done, a dataset from real traffic, three separate metric families, a judge you have tested, a gate with an owner, a guarded rollout and traces that explain an incident. The measure of whether it worked is not the score. It is whether the last regression was found by your suite rather than by a customer.