A Component That Can Be Wrong

A Component That Can Be Wrong is the constraint the rest of this book is designed around. Every architectural habit you have is built on components that are either correct or visibly broken, and a language model is a third thing: right at some rate, wrong at some rate, and equally fluent in both cases.

Key takeaways

• Conventional components fail loudly. They throw, time out, or return an error code, and every pattern you know for handling failure depends on that signal existing.
• A probabilistic component fails silently and fluently, so the error rate becomes a design input you build around rather than a defect you wait to have fixed.
• The rate is not the whole story. What matters is the cost of a wrong answer at the point it lands, which is a product question rather than a model question.
• MIT's Project NANDA report puts about 95% of enterprise generative AI pilots at no measurable P&L impact, attributing the gap to integration and organisational learning rather than to model quality.
• Waiting for a better model is a strategy with no completion criterion, because a lower error rate changes the arithmetic of these decisions and never removes them.

Read this before Chapter 2, which is the test for whether the model belongs in the feature at all, and Chapter 3, which turns this constraint into the one decision everything else follows from. The reliability argument for long-running work is Agents That Finish.

The feature demos beautifully. It reads a customer's message, works out what they want, and drafts a reply that a support lead describes as better than what the team writes on a busy afternoon.

It ships. Three weeks later somebody notices that roughly one reply in twenty confidently states a refund policy the company does not have.

Nobody wrote that policy. Nobody can find where it came from. And nothing in the system logged an error, because from the system's point of view nothing went wrong.

Determinism is an assumption, and it is load-bearing

Software architecture is built on a promise that is so reliable nobody says it out loud. Components either work or they announce that they have not.

A database returns rows or raises. An HTTP call returns a status code. A parser either produces a tree or throws. Every pattern you have for resilience, meaning retries, circuit breakers, fallbacks, timeouts and health checks, depends on failure being detectable by the caller.

That is what a language model breaks. It returns a well-formed, plausible, confident answer whether or not the answer is right, and there is no field in the response that distinguishes the two. The call succeeded. The content may be fiction.

So the resilience machinery you already own does not fire. Your retry logic retries timeouts, not falsehoods. Your circuit breaker opens on error rates it can measure, and it cannot measure this one.

The error rate is an input, not a bug report

The habit worth breaking early is treating wrongness as a defect with a fix. It is a property of the component, like latency or cost, and it belongs in the design the same way.

Stated properly, the input has three parts. A rate, meaning how often the output is unacceptable. A distribution, meaning which kinds of input produce the bad cases. And a severity, meaning what a bad case costs when it reaches whoever is downstream.

Teams routinely have none of those numbers, which is the actual problem rather than the rate itself. A feature designed against an unmeasured error rate is a feature designed against an assumed one, and the assumption is always optimistic.

Measuring it is the subject of a companion volume, and the evaluation argument is that you cannot ship a probabilistic system on taste. This book takes the number as given and asks what to build around it.

Confidently wrong is the shape of the failure

Among ways to be wrong, this one is specifically hostile to human review, and the design has to account for that rather than assume vigilance.

A component that fails obviously gets caught. Garbled output, an empty response, a timeout: those are visible to a user and to a monitor. A component that produces a polished paragraph containing one invented fact is not caught, because the polish is what reviewers use as a proxy for correctness.

That proxy is broken here, and it was reasonable everywhere else. Humans learned to judge text quality by fluency because, for text written by humans, fluency correlated with care. The correlation does not hold for generated text, and no amount of telling reviewers to be careful restores it.

The design consequence is that review cannot be the only control. Whatever a reviewer is asked to check has to be small, specific and checkable, which is a constraint on the shape of the output rather than an instruction to the reviewer.

What the downstream code stops being able to assume

Working outward from the model, each assumption that fails costs a specific piece of engineering.

Schema is the first. A component that usually returns valid JSON returns invalid JSON sometimes, so parsing needs a real failure path rather than an optimistic one. Semantics are the second, and harder: a validly shaped response can carry a value that is well-typed and wrong, such as a date in the correct format that is not the date on the document.

Idempotence is the third, and it is the one that causes incidents. Calling the same component twice with the same input can produce two different plans, so a retry is not a repeat, it is a second decision. Any code that retries a model call and then acts on the result needs to know that.

Referential integrity is the fourth. A model asked for an identifier will supply one that looks correct, and it may name a record that does not exist. Anything the model produces that is used as a key has to be resolved against a system of record before it is used.

The four questions this forces

Before designing anything, four questions have to have answers, and a team that cannot answer them is not ready to build.

What does a wrong answer cost, at the point where it lands. Who or what notices that it was wrong, and how long does that take. Can the effect be undone, and by whom. And what does the user see while the system is unsure.

None of those four is a machine learning question. All four are product and architecture questions, and the answers differ wildly between a feature that suggests a tag on a document and one that issues a payment.

That is the whole reason this book exists in the order it does. Chapter 2 screens the feature, Chapter 3 picks its shape, and Chapters 4 through 11 are the four answers worked out in detail.

Most of these features fail for non-model reasons

There is a published figure worth putting in front of anyone who believes the model is the project's main risk, and it should be quoted with its method rather than waved around.

MIT's Project NANDA report, The GenAI Divide: State of AI in Business 2025, reports that about 95% of enterprise generative AI pilots produced no measurable profit and loss impact, against enterprise investment the report puts at 30 to 40 billion dollars. Its method was 52 executive interviews, surveys of 153 leaders and analysis of 300 public deployments.

The part that matters for an architect is the attribution rather than the headline. The report locates the gap in integration and organisational learning rather than in model capability, which is to say the models were good enough and the systems around them were not.

Read as an engineering statement, that is a claim about the four questions above. A pilot with an impressive demo and no undo, no escalation path, no cost ceiling and no measured error rate is a pilot that cannot be put in front of real work, whatever the model scores.

A worked example, composited and labelled

The details here are composited from ordinary product shapes rather than taken from one system, and the shape is exact.

A team builds invoice data extraction. The model reads a PDF and returns supplier, date, amount and line items, and the extraction is good, at something like 94% of fields correct on their own sample.

Shipped as an automation that posts entries directly to the ledger, 94% is a disaster. Six fields in a hundred are wrong, the errors are silent, they compound across a month, and correcting a posted ledger entry is an accounting task rather than a click.

Shipped as a draft that a clerk confirms, the same 94% is a strong feature. It moves the clerk from typing to checking, the errors surface at review, and a rejected field costs seconds.

The model did not change between those two products. The interaction shape did, which is the argument of Chapter 3 arriving early because it is unavoidable.

The objection: the next model will fix this

This is the most common response to everything in this chapter, and it deserves a real answer rather than a lecture about hype.

Models genuinely do improve, and the improvement is not marginal. Epoch AI's analysis of inference prices, published in March 2025, found that the price of reaching a given capability milestone fell between 9x and 900x per year depending on the milestone, with a median around 50x, and a median near 200x for milestones first reached after January 2024. Capability per pound moved fast.

None of that removes the design problem, for two reasons. A lower error rate changes the arithmetic of the four questions and never makes them go away, because a system acting autonomously at a 1% error rate still needs to know what a wrong action costs and who undoes it. And Epoch's own note is that the persistence of the fastest declines is uncertain, so a feature whose economics work only at next year's prices has no economics today.

There is also a simpler point. Waiting has no completion criterion. No release note will say that hallucination is solved, so a plan that begins with waiting is a plan with no start date.

Chapter summary

Conventional components are either correct or loudly broken, and every resilience pattern in ordinary architecture depends on failure being detectable by the caller, which is precisely what a language model does not offer: it returns a fluent, confident, well-formed answer whether the content is right or invented, so retries, circuit breakers and health checks never fire. The error rate therefore becomes a design input with three parts, being a rate, a distribution over which inputs produce bad cases, and a severity measured by what a bad case costs where it lands, and most teams hold none of those numbers, which is a larger problem than the rate. Confident wrongness is specifically hostile to human review, because reviewers use fluency as a proxy for care and that correlation does not survive generated text, so whatever a reviewer checks must be small and specific rather than a polished paragraph. Downstream code loses four assumptions in turn: valid schema, correct semantics inside a valid schema, idempotence, since two identical calls are two decisions rather than one repeat, and referential integrity, since any identifier the model produces must be resolved against a system of record. Four questions follow, none of them about machine learning, covering what a wrong answer costs, who notices and how fast, whether the effect can be undone, and what the user sees while the system is unsure. MIT's Project NANDA report puts about 95% of enterprise pilots at no measurable P&L impact and attributes that to integration and organisational learning rather than model quality, which is a claim about exactly those four answers. And the same extraction model at 94% field accuracy is a disaster as an automation and a strong feature as a draft, which is the interaction shape deciding the outcome rather than the model.

Before choosing a shape there is a prior question, which is whether this feature should contain a model at all. Chapter 2 is Deciding Where AI Belongs and Where It Does Not, and it is a screening test with two gates that most proposed features fail.