Latency Budgets and Perceived Speed

Latency Budgets and Perceived Speed is the practice of deciding in advance how many milliseconds each part of an interaction may consume, then designing the interface so the unavoidable waiting reads as progress rather than absence. It is a budget, so it is finite. Every step that overspends takes the difference from another step.

Key takeaways

• A latency budget is allocated before you build, per step, with a stated total. Measuring afterwards produces a complaint rather than a design.
• Nielsen's three response time limits, 0.1 second for instantaneous, 1 second for uninterrupted thought and 10 seconds for holding attention, still set the thresholds, and Google's Core Web Vitals guidance puts good interaction responsiveness at or below 200 milliseconds.
• Streaming changes perceived duration, not actual duration. It helps a draft that a user reads as it arrives and does nothing for a suggestion, because half a suggestion is not usable.
• Optimistic state is a promise. Show the result before it is confirmed only where you can honestly and visibly take it back, which is Chapter 4's reversibility spent on speed.
• Work that cannot be fast should stop pretending to be synchronous. Convert it to a job with a notification, and the budget stops mattering.

Read this after Chapter 7, which priced the same interaction in money, and before Chapter 9, which buys latency back with a cache and can hand back the wrong answer while doing it. Chapter 3's ladder set the tolerance for each shape, and this chapter spends it.

The team measures the feature at a median of 2.1 seconds and considers that acceptable. Support tickets say it is slow.

They look again with the median broken out. Retrieval is 400 milliseconds, the model's first token arrives at 1.6 seconds, and generation finishes at 2.1.

The user is looking at a spinner for 1.6 seconds and then reading for half a second. Nothing in the measurement was wrong. The measurement was answering a question nobody asked.

Latency is a budget you allocate, not a number you observe

Most teams treat latency as an outcome. It is a constraint, and constraints are allocated.

The allocated version looks like this. The interaction has a total. The total is divided across the steps, each step gets a number, and exceeding it is a defect against that step rather than a general regret. Teams already handle page weight and error budgets this way. Neither is controversial.

Doing it in advance changes design decisions rather than just monitoring. A 300 millisecond retrieval budget rules out a reranking pass unless something else gives. A 500 millisecond verification budget makes a second model call impossible and a code check obvious. Those are architecture decisions and they should be made with the number in hand.

Doing it afterwards produces the conversation nobody wins. The feature is slow, every step looks individually reasonable, and there is no agreed basis for naming the one at fault. So nothing changes.

Three thresholds still decide the design

The relevant human thresholds have been stable for three decades, and they are worth quoting from the source rather than paraphrasing from memory.

Jakob Nielsen's response time limits, published by the Nielsen Norman Group and drawn from his 1993 book Usability Engineering, name three. 0.1 second is the limit for the user feeling the system reacts instantaneously. 1 second is the limit for their flow of thought to stay uninterrupted, even though they notice the delay. 10 seconds is the limit for keeping their attention on the dialogue. Beyond that, people go and do something else.

The modern web performance equivalent is more specific about responsiveness. Google's web.dev documentation for Interaction to Next Paint puts an INP at or below 200 milliseconds as good, above 200 and up to 500 as needing improvement, and above 500 as poor, assessed at the 75th percentile of experiences rather than the median.

Those two sets of numbers do different jobs. Nielsen's limits describe how long a human will tolerate waiting for the outcome. The INP threshold describes how long the interface may take to acknowledge that anything happened, and no language model latency excuses missing it.

That distinction is the useful one. The button must respond within 200 milliseconds. The answer may take four seconds if the intervening time is legibly occupied.

Where the time actually goes

Breaking the interaction into its real steps is what makes a budget assignable, and there are more of them than the code suggests.

Ahead of the model there is request handling, authentication, authorisation and any policy check, which should be tens of milliseconds. Then context assembly: retrieval, reranking, template rendering and token counting. Retrieval is usually the largest pre-model cost and the one most often unbudgeted.

Then the model itself, which splits into two numbers that behave differently. Time to first token is dominated by queueing and prompt processing, so it grows with prompt length. Generation time after that is roughly proportional to output length, so it is governed by how much you asked for. Those are two separate levers: shorten the prompt to improve the first, shorten the answer to improve the second.

After the model there is verification, parsing, policy evaluation and the commit. Chapter 5 put a gate here and Chapter 4 put an audit record here, and both cost time that belongs in the budget rather than in a footnote.

The reason the split matters is that only some of these can happen while the user reads. A check that must complete before the first token appears is expensive in perceived terms. The same check running against a streamed answer before the commit is nearly free.

The budget differs by shape, by an order of magnitude

Chapter 3's ladder is the right axis, because the user's attention is in a different place on each rung.

The first row is the one that gets designed wrong. An inline suggestion is racing the user's own hands, so a suggestion that arrives after they have typed the thing is worse than no suggestion, because it is noise on a page they have moved past. Cancel aggressively and show nothing.

The third row is the one that gets wasted. An unattended action has time nobody is spending, and Chapter 3 made the point that a feature answering in 400 milliseconds when nobody is watching has converted its main advantage into nothing.

Streaming changes perception, not duration

Streaming is the most commonly reached-for latency fix, and understanding precisely what it buys prevents applying it where it does not.

What streaming changes is when the user's wait ends. Total time to the complete answer is the same, or slightly worse. Time until they have something to read drops to time to first token, which is often a quarter of the total, and reading occupies the remainder.

That is why it helps a draft enormously and a classification not at all. A partially arrived paragraph is readable. A partially arrived category is a fragment of a word, and a partially arrived JSON object is unparseable. Streaming structured output to a user buys nothing. Streaming it to code buys a bug.

Streaming has real costs that belong in the decision. Errors arrive mid-stream, after the user has started reading, which means the interface needs a way to retract or amend a partial answer. Any verification that would have blocked a bad answer now runs against text already on screen, so either the commit is separate from the display, or you have chosen to show unverified content.

The clean pattern is to stream the display and gate the commit. The user reads immediately, the check still happens, and nothing takes effect until it passes.

Optimistic state is a promise you have to keep

The other perceived-speed technique is showing the outcome before it is confirmed, and it is Chapter 4's reversibility being spent to buy speed.

Optimistic state works when three things hold. The success rate is high enough that the optimistic display is usually right. The reversal is visible and honest when it is not. And the effect has not left your boundary, so an undo genuinely undoes it.

Where those hold, it is excellent: mark the message as filed, show the label applied, move the item to the new column. Where they do not, it is a lie with a short fuse. Showing an email as sent before the send succeeded means the user believes something false about the world. The correction arrives too late.

The specific hazard for AI features is applying optimism to the model's judgement rather than to the transport. Optimistically assuming the write succeeded is fine. Optimistically assuming the extraction was correct is a different claim and a much worse bet.

Work that cannot be fast should stop pretending

Some work genuinely takes a minute, and the honest move is to change its shape rather than to decorate the wait.

The conversion is well understood. The request creates a job, returns immediately with an identifier, and the user gets a notification when it completes. Nothing about a language model makes this harder than it is for video encoding or report generation.

Two details separate a good job from an annoying one. Progress has to be meaningful, meaning named stages rather than a percentage somebody invented. A stage list says what is happening. A fake percentage bar teaches the user to distrust it. And the result has to be durable and addressable, so they can close the tab and come back.

This is also where the latency budget stops being the binding constraint and the cost budget takes over. A job with no user waiting can afford three verification passes, and Chapter 7's arithmetic is what decides whether it should.

A worked example, composited and labelled

The details here are composited from ordinary production shapes rather than one system, and the shape is exact.

A product adds AI search across an internal knowledge base. The first version retrieves, reranks, generates an answer, checks the answer against the retrieved passages, and renders. Median 4.3 seconds, all of it behind a spinner.

The budget rewrite made four changes and touched no model. The search results themselves render immediately from retrieval, at around 500 milliseconds, so the user has something true to read while the summary is produced. The generated summary streams into a panel above them. The reranking pass moved behind a flag, because it cost 600 milliseconds and its measured contribution to answer quality was small. And the check against retrieved passages now runs against the streamed text and gates the citation badges rather than the display.

Total time got marginally worse. Perceived time went from 4.3 seconds of nothing to half a second to first content. The complaints stopped.

The lesson is about ordering. Ship the true, cheap, fast part first, and let the expensive part arrive into a page that is already useful.

The objection: users will wait for a good answer

They will, sometimes, and the objection is right often enough to be dangerous.

It holds where the user has no alternative and the value is high: a legal search that saves two hours, a reconciliation that would otherwise be manual. People wait for those, and Nielsen's 10 second limit is about attention on a dialogue rather than about willingness to run a long job they asked for.

It fails in three places. It fails where an alternative exists, because a user who can get a mediocre answer instantly will usually take it. It fails for anything in a workflow the user repeats fifty times a day, where four seconds is three and a half minutes of their day. And it fails when the wait is unexplained, since tolerance depends on knowing what is happening far more than on the number.

So the reasonable position is not that speed always wins. It is that the wait must be earned, visible and bounded, and that a slow feature with no acknowledgement, no progress and no cancellation is not a slow feature, it is a broken one.

Chapter summary

A latency budget is allocated across the steps of an interaction before it is built, with a stated total, so that overspending is a defect against a named step rather than a general regret, and so that decisions like whether a reranking pass or a second verification call fits are made with the number in hand. Jakob Nielsen's limits from Usability Engineering still set the human thresholds at 0.1 second for apparent instantaneity, 1 second for uninterrupted thought and 10 seconds for held attention. Google's web.dev guidance puts good Interaction to Next Paint at or below 200 milliseconds at the 75th percentile. Those two do different jobs: the interface must acknowledge within 200 milliseconds even when the answer takes four seconds. The steps that consume the budget include request handling and policy checks, then context assembly where retrieval is usually the largest and least budgeted pre-model cost, then time to first token which grows with prompt length, then generation time which grows with output length, then verification, parsing and the commit. The crucial distinction is which of those can run while the user reads. Budgets differ by interaction shape by an order of magnitude: an inline suggestion races the user's own hands and is best cancelled silently, while an unattended action holds time it should spend on verification rather than waste. Streaming changes when the wait ends rather than how long the work takes, which helps a readable draft, does nothing for a classification and actively harms a structured output. Stream the display and gate the commit. Optimistic state is reversibility spent on speed and requires a high success rate, an honest visible reversal and an effect that has not left your boundary. Never apply it to the model's judgement. And work that cannot be fast should become a job with named stages and a durable result.

A cache is the other way to make an interaction fast, and it is the one that can make the system wrong faster. Chapter 9 is Caching Without Serving the Wrong Answer Faster.