Start From the Load, Not the Diagram

Start From the Load, Not the Diagram is the ordering rule that makes every later decision in this book arguable. The load model is the first artifact of a design, and the diagram is a consequence of it, because a box drawn before the arrival rate is known is a box whose size nobody can defend.

Key takeaways

• A load model has five parts: arrival rate, service time, the shape of demand over a day and a week, the percentile you will be judged on, and the growth multiplier you are designing for.
• Concurrency is derived, not chosen. Little's 1961 result gives the mean number in the system as arrival rate times mean time in system, so two of those three numbers fix the third.
• Size from the peak, not the mean. Two systems with an identical mean can differ by an order of magnitude at the hour that matters.
• Averages hide the failure. Google's SRE book points out that at a 100 ms average and 1,000 requests per second, one percent of requests might easily take five seconds.
• Fan-out converts a rare slow call into a common slow request, which is why Dean and Barroso argue that tail latency, not mean latency, decides user-visible responsiveness.

Read this beside Chapter 2, which takes the load model and asks where state has to live to absorb it, and Chapter 11, which turns the same model into a bill. This chapter names no currency at all, deliberately. Money is chapter 11's subject and mixing the two is how load models end up being argued about on price rather than on shape.

A design review opens with a diagram. Three boxes, two arrows, a database cylinder, and a queue drawn as a rectangle with a wavy edge.

Somebody asks how many requests per second the middle box handles. The room goes quiet, because nobody has that number, and the diagram was drawn as though it did not need one.

That is the failure this chapter fixes. Not the drawing. The order.

A load model is five numbers and a shape

A load model is a written statement of the demand a system must absorb, expressed in units somebody can check. It has five parts and none of them is optional.

Arrival rate is the first: requests, messages or jobs per second, per class of work. Service time is the second: how long one unit of work occupies a worker, measured rather than guessed. Shape is the third, and it is the part teams skip: the profile of arrivals across a day, a week and whatever cycle your business actually has.

The fourth is the percentile you will be judged on. The fifth is the growth multiplier, which is a factor applied to the shape rather than to a single number.

Write those five down and the diagram draws itself. Skip them and you will draw a diagram that is right in topology and wrong in every dimension.

Concurrency is derived, not chosen

The most common sizing error is treating concurrency as a knob. It is not a knob. It is an output.

Little's Law, published by J. D. C. Little in Operations Research in 1961 as a proof of the queueing formula L = λW, states that the mean number of units in a system equals the arrival rate multiplied by the mean time a unit spends in the system, given finite means and a stationary process. The consequence for a design is immediate. Fix any two of arrival rate, latency and concurrency, and the third is already decided.

So a pool size is not a preference. If work arrives at 200 requests per second and each request occupies a worker for 250 milliseconds, the mean number in the system is 200 times 0.25, which is 50. Fifty workers is the floor for steady state, before any variance, any burst and any headroom.

That arithmetic is the reason a thread pool of 20 behind a 200 per second arrival rate does not degrade gracefully. It cannot. The queue in front of it grows without bound until something rejects work, which is chapter 7's subject.

The mean is the number that hides the problem

Averages are seductive because they are one number and they are always available. They are also the number least connected to what a user experiences.

Google's SRE book makes the point with arithmetic in its monitoring chapter. At an average latency of 100 milliseconds and 1,000 requests per second, one percent of requests might easily take five seconds, and the average will not move. The book's recommendation follows from that: collect request counts bucketed by latency with exponentially distributed boundaries, rather than averages computed for you.

Amazon's Dynamo paper, published at SOSP in October 2007, states the same discipline as an internal requirement. Its services express latency targets at the 99.9th percentile of the distribution, and the paper is explicit about why: the aim is to bound the experience of every customer rather than the experience of the median one.

Pick your percentile before you pick your hardware. Write it into the load model as a number with a threshold, such as p99 under 300 milliseconds for the read path. That sentence is what makes a later capacity argument settleable.

Fan-out turns a rare slow call into a common one

Percentiles at the component level do not stay at the component level. They compose, and they compose badly.

Dean and Barroso set this out in The Tail at Scale, published in Communications of the ACM in 2013. Their argument is that variability at the individual component level is amplified at scale, so in a system where one user request fans out to many servers, the tail of the component distribution governs the user-visible response rather than the mean. Their named mitigations are hedged requests and tied requests, both of which spend extra work to cut the tail.

The arithmetic is worth doing once, with round illustrative numbers. If a request calls ten independent services in parallel and each has a one percent chance of exceeding 100 milliseconds, then the chance that at least one exceeds it is one minus 0.99 to the tenth power, which is about 9.6 percent. A one-in-a-hundred component event has become a one-in-ten user event.

Raise the fan-out to a hundred and the same one percent component tail produces a slow response for roughly 63 percent of requests. Nothing got slower. The topology did the damage.

Growth multiplies a shape, not a number

Growth is usually recorded as a single figure: three times the traffic next year. That figure is almost useless on its own, because it does not say what happens to the shape.

Three times the mean with the same peak-to-mean ratio is one design. Three times the mean with a peak that has become sharper, because you added a marketing send or a scheduled batch, is a different design with different failure modes. The second one breaks earlier and in a place the first one never touches.

So record growth as a multiplier applied to each part of the model. Arrival rate times three. Peak-to-mean ratio held or changed, stated explicitly. Service time held constant only if you can say why, since a larger dataset frequently makes service time worse.

The useful question is not what the traffic will be. It is which of the five numbers moves first, and what breaks when it does.

A worked load model, composited and labelled

The details here are composited from ordinary production shapes rather than one system, and the shape is exact. The inputs are round illustrative figures chosen so the arithmetic is easy to redo.

A document-collaboration product has two classes of work. Interactive reads arrive at a mean of 400 per second with a weekday peak of 1,200 per second at 09:00 local time in its largest market, and each takes 40 milliseconds of server time. Export jobs arrive at a mean of 2 per second, peak 30 per second, and each takes 8 seconds.

Apply Little's Law to each class separately, which is the step teams miss. Reads at peak need 1,200 times 0.04, which is 48 concurrent workers. Exports at peak need 30 times 8, which is 240 concurrent workers. The small, slow, unglamorous class needs five times the concurrency of the class everyone is looking at.

That single comparison changes the design. It says exports cannot share a pool with reads, because a burst of exports will consume every worker and the read path will fail for reasons nobody can see in the read metrics. It also says the read tier and the export tier grow on different curves.

Now add the percentile. If reads are judged at p99 under 300 milliseconds and service time is 40 milliseconds, the queueing budget is 260 milliseconds, which is what makes a bounded queue and a rejection policy a design requirement rather than a nicety.

What the top of the market actually looks like

It helps to have one calibration point that is first-party and dated, so the numbers in your own model feel like the right kind of thing.

AWS published Prime Day 2025 figures in its news blog on 26 August 2025. Among them: DynamoDB peaked at 151 million requests per second, SQS set a record of 166 million messages per second, EBS peaked at 20.3 trillion I/O operations, and Lambda handled over 1.7 trillion invocations per day.

Those are an existence proof and nothing else. They are not a benchmark, not a target, and not a comparison for your system. The only useful thing to take from them is that the top of the market states load as a rate with a date attached, which is exactly what your load model should do at your own scale.

Resist the urge to design for a number like that. Design for your peak times your stated growth multiplier, and write down the number at which the design has to be revisited.

The decision this chapter settles

Every chapter of this book ends in one decision with a recorded constraint and a signal that would reverse it. Here is this chapter's.

The decision is the load model itself: five numbers per class of work, written into the design document before any component is chosen. The constraint that forces it is arithmetic rather than taste, because Little's Law fixes concurrency once arrival rate and service time are known, and no amount of diagramming changes that.

The reversal signal is a threshold somebody can observe. Name the observed peak arrival rate, or the observed service time, at which the model is wrong enough to redo. Something like: revisit when peak reads exceed 2,000 per second, or when p99 read service time exceeds 80 milliseconds for a full week.

Both numbers go in the record that chapter 13 formalises. A load model without a reversal threshold is a guess with better formatting.

The objection: we do not have traffic yet

This is the honest objection and it deserves a real answer rather than a lecture about premature optimisation.

The answer is that a load model for a system with no traffic is cheaper and more useful than one for a system with traffic, because you are not defending a number, you are stating an assumption. Write the model in terms of business units you can reason about: active accounts, actions per account per day, and the fraction of those that land in the worst hour. Then convert to a rate.

That conversion is the whole exercise. Ten thousand accounts, twenty actions each per working day, with a third of the day's volume in one hour, gives roughly 18 requests per second in the peak hour. Now the model is falsifiable, and the first week of real traffic either confirms it or corrects it.

The cost of skipping this is not a slow system. It is an undefendable one. Eighteen months later somebody asks why the queue is where it is and the only available answer is that it seemed sensible at the time.

Chapter summary

A design starts with a load model rather than a diagram, and the model has five parts: arrival rate per class of work, measured service time, the shape of demand across a day and a week, the percentile the system will be judged on, and a growth multiplier applied to the shape rather than to a single figure. Concurrency is then derived and not chosen, because Little's 1961 proof of the queueing formula fixes the mean number in the system as arrival rate times mean time in system, so a pool of fifty is arithmetic rather than preference when work arrives at 200 per second and occupies a worker for 250 milliseconds. The mean is the number that hides the problem: Google's SRE monitoring chapter shows that a 100 millisecond average at 1,000 requests per second is compatible with one percent of requests taking five seconds, and Amazon's Dynamo paper states latency targets at the 99.9th percentile precisely to bound every customer's experience rather than the median one. Fan-out then amplifies component variability into user-visible slowness, which is Dean and Barroso's argument in The Tail at Scale, and the arithmetic shows a one percent component tail becoming a 9.6 percent user event across ten parallel calls. Applying the law per class of work is what separates a read tier from an export tier, and the chapter's decision is the model itself, recorded with the observed threshold at which it must be redone.

The load model says how much work arrives and how fast it has to finish. It says nothing about where the durable part of that work lives, which is the decision that constrains everything after it. Chapter 2 is State Is the Hard Part.