Failure Domains and the Blast Radius You Chose

Failure Domains and the Blast Radius You Chose is the design decision that answers a question every review eventually asks: when this breaks, how many customers notice. The number is not an outcome of your infrastructure, it is a choice you either make deliberately or inherit by default, and the default is all of them.

Key takeaways

• A failure domain is a set of components that fail together. Blast radius is the fraction of your customers inside the largest one, and if nobody chose it the answer is one hundred percent.
• AWS's cell guidance, published 20 September 2023, brings the fault isolation concepts it applies at Availability Zone and Region level down into a single workload, extending the Well-Architected bulkhead practice.
• You inherit boundaries whether you design them or not. The AWS Fault Isolation Boundaries whitepaper of 16 November 2022 notes that the dependencies a workload takes decide which boundary it actually gets.
• Cascading failure is positive feedback: Google's SRE book describes one component's failure raising the probability that the rest fail, through paths including a garbage collection death spiral and cache-hit collapse.
• A degradation path nobody exercises is not a degradation path. AWS states that degraded paths must be tested and simpler than the primary, and that fallback strategies should generally be avoided.

Read this after Chapter 8, which handled a single call failing, because this chapter handles a whole dependency, zone or tenant being gone. Chapter 7's shedding stops you dying under load and says nothing about how many customers a single fault may reach. Chapter 11 prices the containment this chapter chooses, and Chapter 13 records the number so the next engineer can tell it was deliberate.

A configuration change goes out at 14:10 and takes the shared cache cluster with it. Every tenant is affected, because there is one cluster.

Recovery takes eleven minutes and is competent. The postmortem is thorough about the change process and silent about the fact that eleven minutes reached one hundred percent of customers rather than five percent.

Nobody chose one hundred percent. It was implied by having one of everything.

A failure domain is a set that fails together

The definition is simple enough to apply in a review without a whiteboard. A failure domain is the set of components that stop working together when one of them does.

Blast radius is the measurable consequence: the fraction of customers, or requests, or revenue, inside the largest domain that a single fault can take out. That gives you a number between zero and one, and it can be written into a design document and argued about.

The reason this is worth formalising is that the number is almost never chosen. It emerges from having one database, one cache, one queue, one deployment. Each of those decisions was made for a good local reason and none of them was a decision about blast radius.

So the exercise is short and uncomfortable. List the singletons. For each one, write the percentage of customers it serves. That list is your blast radius profile and it usually contains a surprise.

One call failing is not one domain failing

Chapter 8's machinery is necessary and it solves a different problem, which is worth separating explicitly because teams conflate them.

A timeout with a bounded retry and an idempotency key handles the case where one call in one request does not come back. The system loses a little work, absorbs a little load, and the user retries. Nothing structural is required.

A domain failure is categorically different. The dependency is not slow, it is absent, for minutes. Every request that needs it fails, no retry policy helps because there is nothing to retry against, and the only variables left are how many customers are affected and what the system does instead.

Those two need separate design answers. Retry policy is a per-call setting. Blast radius is a topology decision, and no amount of client-side care changes it.

The boundaries you inherit against the ones you build

Before designing anything, find out what isolation you already have, because you are depending on it whether you know it or not.

The AWS Fault Isolation Boundaries whitepaper, published 16 November 2022, names the distinct boundaries the platform provides. Availability Zones, Regions, and the separation of control planes from data planes, each with a predictable scope of impact. Its most useful point for a designer is that the dependencies your workload takes determine which boundary it actually inherits. A workload calling a Region-wide control plane on the request path has a Region-sized failure domain, whatever its zone count.

That inversion is worth sitting with. Your blast radius is set by your dependencies rather than by your deployment diagram, which means a diagram showing three zones can be describing a single failure domain.

The audit is mechanical. For every dependency on the request path, name the boundary it sits inside. The widest one on the list is your real domain, and every narrower isolation you built is decorative until that dependency is removed from the path.

Cells, and the thin router in front of them

Cells are the technique for making a failure domain smaller than your customer base, and the guidance for them is published rather than folklore.

AWS's guidance, Reducing the Scope of Impact with Cell-Based Architecture, was published on 20 September 2023. It describes taking the same fault isolation concepts the provider applies at Availability Zone and Region level, and applying them inside your own workload architecture. It presents this as an expansion of the Well-Architected bulkhead practice. It claims three properties for the result: more fault isolation, more predictability and more testability.

Testability is the underrated one. A cell is small enough to be loaded to its limit in a test environment, which means you can know its capacity ceiling rather than estimating it, and Chapter 1's load model becomes verifiable per cell.

The router in front of the cells is the part to design conservatively, because it is the new shared fate. It should hold a routing table and nothing else: no business logic, no per-request database call, no dependency that is itself cell-spanning. A router that needs a central service to decide where a request goes has recreated the singleton it was built to remove.

The shared dependency nobody drew

Correlated failure is what breaks cell designs, and it hides in the components nobody thinks of as part of the architecture.

Look for these specifically. A shared secret store consulted on every request. One certificate with one expiry across every cell. A single identity provider, a shared feature flag service, one DNS zone. And one deployment pipeline that pushes to all cells at once. Each of those spans every domain you carefully separated.

The deployment pipeline is the most common and the most expensive, because it converts a bad release into a total outage in the time it takes to roll forward. The fix is to make cells deploy in sequence with a soak period between them, which is also the cheapest incident containment available.

Brewer's 2017 Spanner paper is a useful corrective on where to spend this effort. Its breakdown of causes of unavailability for that system reads 52.5 percent user, 13.3 percent bug, 12.1 percent cluster, 10.9 percent other, 7.6 percent network and 3.7 percent operator. Network is a small minority. The things most likely to take your system down are changes and defects, which is exactly what staged deployment across cells contains.

Cascading failure is positive feedback

A domain failure spreads if the system's response to failure increases load, and Google's SRE book is the clearest published treatment of the mechanism.

Its chapter on addressing cascading failures defines the shape as positive feedback: one component failing raises the probability that the rest fail. It names concrete paths. Resource exhaustion has several. One is a garbage collection death spiral, where memory pressure causes collection that consumes CPU that slows request handling that increases memory pressure. Cache-hit collapse is another. When a cache tier degrades, backend load rises sharply, which is the stampede arithmetic from Chapter 6 arriving as an incident.

The design responses are the ones already established. Bounded queues from Chapter 7, so a degraded stage cannot accumulate unbounded work. Retry budgets from Chapter 8, so failure does not multiply load. And cells, so the feedback loop is confined to a fraction of the system.

The chapter also gives a testing instruction that belongs in the design phase rather than the operations phase. Load a component until it breaks, and then watch it recover. A component whose recovery has never been observed is a component whose recovery behaviour is a guess.

A degradation path is only real if it is exercised

Graceful degradation is the good answer and it comes with a warning that most write-ups omit.

The AWS reliability practice REL05-BP01 says degraded paths must be tested and must be simpler than the primary path, and it states that fallback strategies should generally be avoided. That sounds contradictory until you separate two things. Serving stale, alternate or no data while still performing the core function is degradation. That is good. Switching to an entirely different code path under stress is a fallback. It is risky, because that path is the least tested code in the system running at the worst possible moment.

Google's cascading failure chapter makes the same point from the other direction, warning that lightly used degradation paths are themselves a risk. The path that only runs during an incident is the path with no production evidence behind it.

The resolution is to prefer degradation you can exercise continuously. Serve stale from the cache contract you wrote in Chapter 6, on a small fraction of traffic, every day. Disable a non-core feature during a routine drill rather than only during an incident. If a path cannot be exercised in normal operation, treat it as untested and price it accordingly.

A worked example, composited and labelled

The details here are composited from ordinary production shapes rather than one system, and the shape is exact.

A business-to-business product serves 4,000 customer organisations from one stack. Its blast radius is 100 percent and its largest customers have contractual expectations about availability.

Cells are proposed and the arithmetic decides the count. Eight cells put roughly 500 organisations each, so a single-cell failure reaches 12.5 percent of customers. Twenty cells reach 5 percent. That sounds better until the fixed cost per cell is counted. Each cell needs its own capacity headroom, its own baseline infrastructure and its own place in a sequential deployment. Twenty sequential deploys with a soak period is a release that takes a day.

The chosen design is eight cells, with the three largest organisations placed in their own cells rather than sharing. The recorded constraint is that no single fault may affect more than fifteen percent of organisations, which eight cells satisfies with margin. The reversal signal is customer count per cell exceeding 700, or any new shared dependency appearing on the request path, which would silently restore the old blast radius.

The objection: cells cost more and we are not Amazon

Both halves of that objection are true and neither is a reason to skip the decision.

Cells do cost more. Fixed cost per cell, more infrastructure to operate, a router to keep boring, and a deployment process that takes longer on purpose. That is a real bill and Chapter 11 is where it belongs, priced against the cost of an outage that reaches every customer.

The second half is the more interesting one. You are not operating at that scale, and the technique degrades gracefully to smaller sizes. Two cells is not sophisticated and it halves your blast radius. Separating your three largest customers onto their own stack is one afternoon of infrastructure work and removes the case where a small customer's traffic spike takes down your biggest account.

So the decision is not whether to build cells. It is to write down the blast radius number you have, the number you want, and the cheapest step between them. A team that records fifty percent as an accepted risk has made a decision. A team that has never computed the number has not.

Chapter summary

A failure domain is the set of components that stop working together, and blast radius is the fraction of customers inside the largest one, which is one hundred percent by default because singletons are chosen for local reasons and never as blast radius decisions. Chapter 8's timeouts and budgets handle one call failing, not a dependency absent for minutes, which is a topology question. The AWS Fault Isolation Boundaries whitepaper of 16 November 2022 describes Availability Zones, Regions and the control plane and data plane split as boundaries with predictable scope, and makes the point that the dependencies a workload takes decide which boundary it inherits, so a three-zone diagram can describe a single domain. AWS's cell guidance of 20 September 2023 brings those isolation concepts down into one workload as an expansion of the bulkhead practice, claiming better fault isolation, predictability and testability, with the router kept to a routing table so it does not become the new singleton. Correlated failure hides in shared secret stores, certificates, identity providers, flag services, DNS and above all the deployment pipeline, and Brewer's 2017 figures put network at only 7.6 percent of Spanner's unavailability against 52.5 percent user and 13.3 percent bug. Google's cascading failure chapter supplies the positive feedback mechanism and the instruction to load a component to failure and watch it recover, and the chapter's decision is a recorded blast radius target with the shared dependency that would reverse it.

The blast radius is chosen and the containment is designed. None of it is verifiable unless the system emits the signals that would show a domain degrading, and those signals have to be decided while the design is still cheap to change. Chapter 10 is Observability as a Design Input.