Deciding Where AI Belongs and Where It Does Not

Deciding Where AI Belongs and Where It Does Not is a screening test that runs before any design work. It has two gates, verifiability and reversibility, and where a feature sits against them decides whether it can ship at all and, if it can, which shapes remain available to it.

Key takeaways

• The question is not whether a model could do the task. It is whether the task tolerates being done wrong at some rate, which is a different question with a different answer.
• Gate one is verifiability: can the output be checked by something cheaper than what produced it, meaning code, a system of record, or a human glance rather than a second model.
• Gate two is reversibility: can the effect be undone, by whom, and inside what window.
• A feature that fails both gates should not ship with a model in it, in any shape. That is the finding teams most often overrule and most often regret.
• Where deterministic code can do the job, it should, because it is cheaper, faster, testable and right every time. A model earns its place by handling variety that rules cannot enumerate.

Read this after Chapter 1, which established that wrongness is a property rather than a defect, and before Chapter 3, which chooses among the shapes this test leaves open. Chapter 12 is the same test applied later, when the feature already exists and the evidence says stop.

A planning meeting produces a list of eleven places to add AI to the product. Everyone is enthusiastic and the list is genuinely creative.

Nine of the eleven are string matching, a lookup, or a rule with four branches. They are described with a model because that is the language the meeting was held in.

The two that genuinely need one are the two nobody argued about, because they involve reading unstructured text somebody outside the company wrote.

Most proposals do not need a model

Before the gates, there is a cheaper filter, and applying it first saves the gates for the features that deserve them.

Ask what the feature does when you describe it without naming a technology. Match this description to a category. Extract three fields from a form. Sort these items by urgency. Decide whether this text mentions a competitor. Several of those have deterministic implementations that are faster, cheaper, testable and correct every time.

The tell is enumerability. If the set of valid outputs is small and known, and the mapping from input to output can be written down, then it should be written down. A rule that handles eleven cases is a rule, not a limitation.

A model earns its place where the input space is genuinely open, meaning natural language somebody else wrote, documents in formats you do not control, or a mapping too varied to enumerate without a rule per customer. That is a real category and it is smaller than the meeting thinks.

The arXiv position paper on small language models makes an adjacent point worth borrowing. It argues that agentic systems mostly perform a small number of specialised tasks repetitively, so a large general model is often the wrong instrument for the job. If the task is narrow enough for a small model, it is worth asking whether it is narrow enough for no model.

Gate one: can the output be checked

The first gate asks whether something cheaper than the generator can tell you the output is acceptable. Cheaper matters, because a check that costs as much as the work does not scale.

Some outputs check themselves. An extracted invoice total can be compared against the sum of its line items. A generated identifier can be resolved against the database. A produced SQL query can be run against a copy and its row count sanity-checked. Those are strong positions, because verification is code.

Some outputs are checkable by a human at a glance, which is weaker but real. A suggested category is checkable in a second. A three-line summary of a document the reader knows is checkable in ten. The cost is bounded and the reviewer is not being asked to audit a paragraph for invented facts.

Some outputs are not checkable at acceptable cost. A summary of a fifty-page contract the reviewer has not read, a claim about market conditions, a piece of legal or medical reasoning: verifying those requires doing the work, which removes the point of the feature.

Unverifiable is not automatically disqualifying. It means the feature cannot rely on verification as its control, so it has to rely entirely on gate two.

Gate two: can the effect be undone

The second gate asks what happens after a wrong output is accepted, and the honest answer is often worse than the team assumes.

Three properties matter. Whether an undo exists at all. Who can perform it, since an undo that needs an engineer and a database session is not an undo from the product's point of view. And the window, because a mistake caught in ten seconds and the same mistake caught in ten days are different incidents.

Some effects are cleanly reversible: a draft not yet sent, a tag, a draft order, a change under soft deletion. Some are reversible with cost and friction, such as a posted ledger entry that needs a correcting entry, or a shipped package that needs a return.

Some are not reversible in any useful sense. A message a customer has already read. A payment that has cleared. A file deleted from a system without versioning. A decision communicated to a regulator. For those, the product's only remaining control is preventing the wrong output from being accepted, which forces a shape.

Chapter 4 is about manufacturing reversibility where it does not exist naturally, which is frequently possible and is a product decision rather than a technical one.

The four positions, and what each permits

The two gates produce four positions, and each one leaves a different set of shapes available.

Verifiable and reversible is the comfortable quadrant. Errors are caught cheaply and undone cheaply, so any shape works, including autonomous action, and this is where a first AI feature should live if the organisation has a choice.

Verifiable but irreversible permits drafting with a confirmation step, because the verification is what makes the confirmation meaningful. The user checks the specific checkable thing, then commits.

Reversible but unverifiable permits action, provided the undo is real, discoverable and used. The design effort goes into making reversal a normal operation rather than an incident.

Neither verifiable nor irreversible is the quadrant to refuse. There is no control left: you cannot tell whether the output is wrong and you cannot undo it once it lands. Features here get promoted anyway, usually because the demo was persuasive, and they are a plausible mechanism behind the integration failures MIT's Project NANDA report describes when it puts about 95% of enterprise pilots at no measurable P&L impact and locates the cause outside model quality.

Where deterministic code wins outright

It is worth being specific about the cases where a model is the wrong choice even though it would work, because these are the ones that get lost.

Anything with a legal or contractual answer belongs in code. Tax rates, pricing rules, eligibility criteria, retention periods: those have correct answers that somebody is accountable for, and they need to be auditable line by line rather than probable.

Anything an auditor will ask you to reproduce belongs in code, because reproducing a model's decision from six months ago requires you to have pinned the version, the prompt and the context, which is a real project rather than an assumption.

Anything that runs at very high volume on a narrow task usually belongs in code, on cost grounds alone. Chapter 7 does the arithmetic, and the arithmetic is unkind at scale.

And anything where a wrong answer is silently absorbed belongs in code, because silent absorption means nobody ever finds out, so the error rate is unbounded in practice however low it is in principle.

Where the model is genuinely load-bearing

The positive case deserves as much precision, because a book that only warns is not useful.

Open-ended input is the strongest case. Text a customer wrote, a document in an unpredictable format, a request phrased in a way nobody anticipated: rules do not enumerate this and the model handles it well.

Tolerant-output tasks are the second. Drafting, summarising, rephrasing, suggesting, clustering: work where several answers are acceptable and the user is going to shape it anyway.

Cheap-error, high-volume triage is the third, where being right most of the time saves real effort and being wrong costs a correction. Routing a ticket to the wrong queue is annoying and survivable, and doing it manually for ten thousand tickets is expensive.

Interfaces to the unstructured are the fourth: turning a described intent into a structured request that deterministic code then validates and executes. The model does the interpretation and the code keeps the authority, which is the pattern most of the good features in this category share.

A worked example, composited and labelled

The details are composited from ordinary product shapes rather than one system, and the shape is exact.

A logistics product proposes three AI features. Automatically reschedule deliveries when a driver runs late. Summarise each day's exceptions for the operations lead. And classify inbound customer emails by intent.

Run the gates. Rescheduling is verifiable, since the new plan can be checked against capacity and constraints in code, and it is reversible only until a customer notification goes out. Summarising is barely verifiable, since the reader has not read the underlying exceptions, and it is fully reversible because it is a document nobody acts on directly. Classification is verifiable by a human in a second and fully reversible.

So classification ships as an autonomous action with a visible correction. Rescheduling ships as a draft plan that a dispatcher confirms, with the notification as the commit point. And summarising ships, carefully, with the constraint that it links to every exception it mentions, which converts an unverifiable claim into a checkable one.

That third move is the general trick worth remembering. When an output fails the verifiability gate, changing the output format is often cheaper than changing the model.

The objection: our competitors ship AI in everything

They do, and some of them are wrong, and being wrong in company is still being wrong. That said, the objection is usually about market perception rather than architecture, and it should be answered on those terms.

The competitive question is whether customers switch because of the feature, and for undifferentiated AI additions the answer has generally been no. A feature that is wrong one time in fifteen and cannot be checked produces support load and distrust, which competes against you rather than for you.

The stronger position is to ship fewer of these, with the undo and the honesty designed in, and to be able to say what the system does when it is unsure. That is a claim a competitor who bolted a model onto eleven surfaces cannot make.

There is also a sequencing argument that satisfies most of the pressure. Ship the two features that pass both gates first, learn the operational reality of running them, and use that experience on the harder quadrants. That is a faster route to a credible AI product than eleven pilots, and it is roughly the opposite of what the failure statistics describe.

Chapter summary

The screening question is not whether a model could perform the task but whether the task tolerates being performed wrong at some rate, and a cheaper filter runs first: describe the feature without naming a technology, and if the valid outputs are enumerable and the mapping can be written down, write it down, because deterministic code is cheaper, faster, testable and correct every time. The first gate is verifiability, meaning whether something cheaper than the generator can confirm the output, which is strongest when verification is code, such as an extracted total checked against its line items or an identifier resolved against the database, weaker but real when a human can check at a glance, and absent when confirming requires doing the work again. The second gate is reversibility, meaning whether an undo exists, who can perform it without an engineer, and inside what window, with sent messages, cleared payments and unversioned deletions failing it outright. Those gates produce four positions: verifiable and reversible permits any shape and is where a first feature belongs; verifiable but irreversible permits a draft with a meaningful confirmation; reversible but unverifiable permits action provided the undo is real and used; and neither should not ship with a model in it, which is the finding teams overrule most often. Deterministic code wins outright for legal and contractual answers, anything an auditor will ask you to reproduce, narrow work at very high volume, and anything whose errors are silently absorbed. Models are genuinely load-bearing for open-ended input, tolerant outputs, cheap-error high-volume triage, and as an interface that turns described intent into a structured request which code then validates and executes. And when an output fails the verifiability gate, changing its format, such as requiring a summary to link every exception it mentions, is usually cheaper than changing the model.

The gates leave a set of shapes available. Choosing among them is the single most consequential decision in the whole design, because it fixes reversibility, latency, cost and review load simultaneously. Chapter 3 is Suggest, Draft, or Act.