Routing: Capability, Cost, Fallback

Routing: Capability, Cost, Fallback is the decision to stop treating the model as a single dependency. A feature with four model calls has four independent choices: which model runs, what happens when that model is unavailable, and what the product does when nothing is. Answering all three per step is what makes a heterogeneous system rather than a wrapper.

Key takeaways

• Routing is per step, not per feature. A pipeline with classify, extract, draft and check stages has four routing decisions, and using one model for all four is a default rather than a choice.
• A route is a triple of step, primary and fallback. The fallback is the element teams omit, which is why a single provider incident becomes a product outage.
• Build the table from measured per-step results. Instinct reliably picks the wrong step to spend capability on, because the visible step is not the hard one.
• Fallback is not retry. Retry assumes the same call will work next time; fallback assumes it will not and changes something about the call.
• Degrading beats failing and both beat guessing. Decide, per step, what a reduced answer looks like and whether the product can honestly serve one.

Read this after Chapter 5, which handled the case where work leaves the system for a human, and before Chapter 7, which prices the table you build here. The version of this argument that concerns long-running work with many steps is the long-horizon reliability argument.

The feature has one environment variable naming a model, read in one place, used by every call. It has been that way since the prototype.

A provider incident lasting ninety minutes takes the whole feature offline. Not degraded. Off.

In the review afterwards somebody points out that three of the four calls in the pipeline are classification against a fixed list of categories, and that the team had a rule-based classifier for that until eighteen months ago. It still exists in the repository.

One model for everything is a default, not a decision

Almost every feature starts with a single model reference because that is how prototypes are built, and nothing ever forces the question again.

The reason it survives is that it never looks wrong. Quality is acceptable, the code is simple, and the bill is a single line item. The costs are all deferred: they arrive as an outage, as a cost review, or as a latency complaint about the step that did not need a frontier model.

What makes the default expensive is that model calls inside one feature are not alike. Classify a message into one of forty categories. Extract six fields from a document. Draft a paragraph in the company's voice. Check that paragraph against a policy. Those are four tasks with different capability requirements, different volumes and different consequences for being wrong.

Volume decides the bill. The cheap steps are usually the frequent ones, so routing them to your most capable model means paying the highest unit price on the largest count. That is the arithmetic Chapter 7 makes explicit.

A route is a triple, and the third element is the one people omit

Being precise about what a route is prevents the discussion turning into model preference.

A route is a step, a primary model and a fallback. The step is a named unit of work with an input contract and an output contract. The primary is the model that normally serves it. The fallback is what serves it when the primary will not, whether that is a different model, a different provider hosting a similar model, a deterministic implementation, or an explicit degraded behaviour.

Most routing tables in production have the first two columns filled in. The third is empty, or holds the word retry, which is not a fallback.

There is a fourth element worth recording even though it is not part of the mechanism: the reason. Write down why this step is on this model, in one sentence, with the date. Six months later that sentence is the only thing standing between you and re-litigating the whole table from memory.

Capability, cost and failure behaviour are three separate axes

Teams collapse routing into a single quality ranking, and the three axes genuinely diverge.

The last row is the one that separates an architecture from a configuration. Capability, cost and latency are properties you can look up or measure once. Failure behaviour only exists if somebody built it and exercised it, and the honest way to know it works is to turn the primary off in a staging environment during working hours.

Note that the axes disagree in useful ways. The most capable model is rarely the fastest, the cheapest is rarely the most capable, and the most reliable provider on your list may be neither. A table forces you to state the trade per step instead of settling it once for everything.

Build the table from measured per-step results

The instinct-driven version of this table is wrong in a predictable direction, and the correction is measurement rather than argument.

Instinct routes capability to the step that looks hard to a human. Drafting prose looks hard, so drafting gets the frontier model. In the features I have looked at, the step that actually needs capability is usually the checking step, because checking requires holding a policy and an output in mind at once and noticing a subtle conflict, and it is the step whose failure is silent.

Getting this right needs per-step evaluation rather than end-to-end scores. An end-to-end accuracy number tells you the pipeline works and nothing about which stage to spend on. So the prerequisite for a routing table is a small evaluation set per step, with its own inputs and its own pass criteria.

There is published work on the mechanical version of this. Ong and colleagues, in a paper first posted to arXiv in June 2024, trained routers that choose between a stronger and a weaker model per query using human preference data. They report cost reductions of more than two times in certain cases on their benchmarks, without a quality drop they could measure. Read that as evidence about their tasks and their models.

An arXiv position paper published in June 2025 and revised in September 2025 argues a related point more strongly. It holds that small language models are sufficiently powerful, inherently more suitable and necessarily more economical for many invocations in agentic systems, because those systems perform a small number of specialised tasks repetitively and with little variation. Where general conversation genuinely matters, it argues for a heterogeneous system invoking several kinds of model. That is a position rather than a benchmark result. It is one the volume arithmetic supports.

Fallback is not retry

These get conflated constantly, and the difference is what the system believes about why the call failed.

Retry assumes the failure was transient and the same call will succeed shortly. It is correct for a timeout, a 429 and a connection reset, and it needs a backoff, a cap and a jitter, exactly as it does for any other network dependency.

Fallback assumes the call will not succeed and changes something. A different provider. A smaller model with a shorter prompt. A deterministic implementation. A different decomposition of the task.

Two hazards follow, and both are specific to probabilistic components. A fallback model has a different error profile, so falling back silently changes the quality of the feature without telling anybody, which means the fallback path needs its own monitoring and its own evaluation rather than inheriting the primary's. And retrying a model call is not idempotent, as Chapter 1 noted, so a retry after a partially applied action is a second decision rather than a repeat.

The rule I use is short. Retry the transport. Fall back on the model. Never retry across a commit point without a check that the first attempt did nothing.

Degrading beats failing, and both beat guessing

When primary and fallback are both gone, the product still has to do something, and the choice should be made in advance.

Google's Site Reliability Engineering book, in its chapter on handling overload, describes degraded responses as ones that are not as accurate as, or contain less data than, normal responses, but are easier to compute. It notes that a service pushed far enough may have no immediate option but to serve errors. That framing transfers directly, with one addition. For an AI feature, a degraded response must not be a less reliable response presented as a normal one.

So a degraded state has to be honest and visible. Classification falls back to the rules and the result is labelled as a rules-based guess. Drafting is unavailable and the compose box opens empty with a note. Summarising is replaced by the first two sentences of the source, which is worse and true.

The unacceptable degradation is the one that keeps the interface identical and quietly lowers accuracy. Users calibrate their trust against what they have seen, so a feature that silently gets worse spends credibility it earned under different conditions.

Chapter 11 designs what those states look like on screen. What belongs in the routing table is the decision itself, one line per step, written before the incident rather than during it.

What routing costs you, honestly

Heterogeneity is not free, and a chapter that only sold it would be useless.

You maintain several prompts, because a prompt tuned for one model is rarely optimal on another, and drift between the primary and fallback paths is a real burden. You maintain an evaluation matrix rather than a list, since every step times every model is a cell somebody has to fill. And you accept two error profiles per routed step.

Version pinning gets harder too. Providers deprecate and update model versions, so a table with five models has five deprecation clocks. The table is the natural place to record which version each row is pinned to and when it was last re-evaluated.

Routing trades simplicity for cost, resilience and fit. It is worth it when volume makes cost matter, or when the availability requirement makes a single provider unacceptable. Below those thresholds, one model and a written fallback plan is defensible.

A worked example, composited and labelled

The details here are composited from ordinary production shapes rather than one system, and the shape is exact.

A document-processing feature has four steps: decide the document type, extract fields, draft a summary for the reviewer, and check the extracted fields against the client's business rules. Everything ran on one frontier model.

Per-step evaluation changed three rows. Document typing turned out to be a twelve-way classification that a small model handled at the same accuracy on their evaluation set, and it was ninety per cent of the call volume. Extraction stayed on a capable model, because field accuracy was the product. Drafting moved down a tier, because reviewers edited the summary anyway. Checking moved up, because that was the step whose misses reached the client.

The fallback column took a day to fill. Typing falls back to the old rules with a visible label. Extraction falls back to a second provider, with its own accuracy tracked separately. Drafting degrades to no summary. Checking has no fallback and no degraded state, because an unchecked document must not be released, so it routes to the human queue from Chapter 5 instead.

Cost fell because the highest-volume step got cheap, not because anything got worse. That is the usual shape of the win.

The objection: this is premature optimisation

The objection is that a routing table is infrastructure for a problem you do not have yet, and for many features that is correct.

The honest threshold is that routing earns its complexity when one of three things is true. The bill is large enough that a factor of two matters. The availability requirement makes a single provider unacceptable. Or a specific step has a measured quality problem that a different model would fix.

Below that, the right amount of routing is architectural rather than operational: put every model call behind one interface, keep the model choice as data rather than scattered constants, and write the fallback decision per step down even if it is not implemented. That costs a day and converts a rewrite into a configuration change.

What is not premature is the failure behaviour. Deciding what the product does when the provider is down is not an optimisation, it is a requirement, and it is the row that turns out to matter first.

Chapter summary

Routing is a per-step decision rather than a per-feature one, and the single model reference inherited from a prototype is a default nobody chose. The four calls inside one pipeline differ in capability requirement, volume, latency share and consequence of error. Because the cheap steps are usually the frequent ones, sending them to your most capable model pays the highest unit price on the largest count. A route is a triple of step, primary and fallback, with the fallback column usually blank or filled with the word retry, and a fourth entry worth recording is the dated one-sentence reason the row exists. Capability, cost, latency and failure behaviour are four axes that disagree, and only the last requires somebody to build something and exercise it, which means turning the primary off in staging rather than writing a document. Build the table from per-step evaluation rather than instinct. Instinct spends capability on the step that looks hard to a human, when the step that needs it is usually the silent checking step. Published work supports the mechanical version: Ong and colleagues report cost reductions of over two times in certain cases from routing between a stronger and a weaker model on their benchmarks, and an arXiv position paper argues that small models are sufficiently powerful and more economical for the repetitive specialised invocations agentic systems mostly make. Retry assumes a transient failure and belongs to the transport. Fallback assumes the call will not work and changes the model, the prompt or the implementation, carrying a different error profile that needs separate monitoring. Degraded states must be honest and visible rather than an identical interface at lower accuracy. And heterogeneity costs prompt maintenance, an evaluation matrix and several deprecation clocks, which is worth paying above a volume or availability threshold and not below it.

The table you have just built has a price attached to every row, and most teams have never added it up. Chapter 7 is The Unit Economics of a Feature, which computes cost per successful interaction rather than cost per call.