Deciding to Cut the Feature

Deciding to Cut the Feature is the decision this book has been building the instruments for. A probabilistic feature can be measured honestly, so whether it should continue to exist has an evidential answer. What usually stands between a team and that answer is that nobody wrote the threshold down while it was still cheap to write.

Key takeaways

• A feature with no kill criterion never gets cut. By the time the evidence is overwhelming, the decision has become a question about whose judgement was wrong.
• Five signals carry the decision: correction-free acceptance, reversal rate, cost per successful interaction, abstention rate and review minutes per interaction.
• Thresholds have to be written before launch, with a date, a number and a named owner, because a threshold set afterwards is a negotiation dressed as a measurement.
• Arkes and Blumer named the sunk cost effect in 1985: a greater tendency to continue an endeavour once money, effort or time has been invested, justified by not wanting to appear wasteful.
• Cutting usually means demoting down Chapter 3's ladder or narrowing the scope, not deleting the code. The reversible move is almost always available and almost never considered.

Read this after Chapter 11, which built the surface most of these signals come from, and alongside Chapter 2, whose screening test this is, applied a year later with real data. Chapter 4's reversal rate and Chapter 7's cost per successful interaction are two of the five numbers.

Twelve people are in a room about a feature everyone privately knows is not working. The engineering lead has a graph. The product lead has a plan for the next iteration.

Nobody says cut it, because there is no agreed basis for saying it. The graph is bad, but bad compared to what. The plan might work, and nobody can prove it will not.

So the meeting ends with a fourth iteration and a review in six weeks. That has happened three times, and the review keeps getting scheduled by the people whose work is under review.

A feature with no kill criterion never gets cut

The failure is structural rather than a failure of nerve, and understanding that is what makes it fixable.

Without a pre-agreed threshold, the case for cutting has to be made in the room, against people who built the thing, using evidence whose interpretation is itself the argument. Every number can be contextualised, every trend has a reason, and the person proposing to stop is implicitly proposing that somebody's judgement was wrong. That is a political conversation, and political conversations default to continuing.

With a pre-agreed threshold, the conversation changes in kind. The number crossed the line. The line was set in February by the person who now has to act on it, so the decision is administrative. Nobody's competence is on trial. A commitment is being honoured.

That is the entire mechanism, and it is the same one behind error budgets and stop-loss orders. Move the judgement to a moment when nothing is at stake, and the later decision becomes arithmetic.

What a cut decision needs to be, precisely

A kill criterion is not a vague intention to review, so the definition needs enough parts to be checkable.

It has five: a named signal, a threshold value, an evaluation window, a decision owner, and the specific action taken when the threshold is crossed. Remove any one and it stops working. A signal with no threshold is a dashboard. A threshold with no owner is a suggestion. An action left unspecified becomes another meeting.

The action deserves particular care because it is where most criteria go soft. Escalate to leadership is not an action. Demote this action from act to draft, or withdraw the feature from the two customer segments where it fails, are actions, because they can be performed by the owner without convening anybody.

Write it as a sentence with a date on it. If correction-free acceptance is below sixty per cent over any trailing four weeks after the first of June, the product lead demotes the drafting step to a suggestion. That sentence takes ninety seconds to write and it is worth more than any amount of subsequent discussion.

Five signals carry the decision

These are the numbers this book has been asking you to collect, and each one fails in a different direction, which is why you need all five.

Correction-free acceptance is the first: the share of outputs a user committed without changing anything. It measures whether the feature does the work or merely starts it. It is more honest than acceptance alone, because Chapter 11 showed that an interface with expensive corrections manufactures acceptances.

Reversal rate is the second, and Chapter 4 called it the cheapest ungameable quality signal the product produces. Users reversing an AI action are telling you it was wrong, at their own expense, in a form nobody had to design a survey for.

Cost per successful interaction is the third, computed as Chapter 7 described, including review minutes. Compare it to the value of the interaction rather than to last month, because a feature can get cheaper every month and still be uneconomic.

Abstention rate is the fourth, from Chapter 5. A rising rate means the input distribution has moved away from what the system handles. A feature declining a third of the time is a rules engine with a large bill.

Review minutes per interaction is the fifth, and it decides whether the feature can scale. If the human time per item is not falling, the feature is a fixed-cost service rather than leverage. Keeping it is then a staffing decision.

Write the thresholds before you launch

The table is the artifact. Fill it in at design time, in the same document as the interaction shape decision from Chapter 3.

Two rules keep the table honest. Set the numbers before you have any data, because a threshold set after you know your performance is a threshold set to accommodate it. And give each row an owner who has the authority to take the action alone, since a threshold requiring a committee is a threshold that will not fire.

One more habit is worth adopting. Review the table on a fixed calendar rather than when somebody is worried, because the reviews that get triggered by concern are the ones that get postponed by optimism.

Sunk cost is the mechanism, and it was named in 1985

The reason all of this needs machinery, rather than good judgement, is a well-documented feature of how people evaluate commitments.

Arkes and Blumer, writing in Organizational Behavior and Human Decision Processes in 1985, described the sunk cost effect as a greater tendency to continue an endeavour once an investment of money, effort or time has been made. They located its justification in a desire not to appear wasteful. Their paper is where the term comes from, and the effect it names is not a failure of intelligence. It is the default.

The AI version carries extra weight. The investment is usually visible, recent and attached to a strategic narrative, so stopping reads as a statement about the company's whole position on AI rather than about one feature's acceptance rate.

Recognising this changes what you build rather than how you think. You cannot reason your way out of a bias you are inside. So the response is a pre-commitment made when the investment was zero, which is exactly what the threshold table is.

There is a second-order effect worth naming. Teams that have cut a feature once find it dramatically easier the second time, because the first cut establishes that stopping is a normal operation with a normal procedure. That is the same argument Chapter 4 made about undo paths.

Cutting usually means demoting, not deleting

The word cut implies deletion, and deletion is rarely the right move, which is one reason the conversation is harder than it needs to be.

Demotion down Chapter 3's ladder is the first option and the most useful. A feature acting autonomously and reversing too often becomes a drafting feature, which keeps the work the model does well and returns the decision to a person. A drafting feature nobody accepts becomes a suggestion. Each step down reduces risk and review cost while keeping whatever value exists.

Narrowing the scope is the second. Most failing AI features fail on a subset: two document formats, one customer segment, one language, one category of question. Measure by segment and the average often turns out to be hiding a feature that works well for most traffic and badly for the rest. That is a scoping decision, not a cancellation.

Replacing the model with code is the third, and it is the outcome Chapter 2 predicted for the proposals that never needed a model. A year of production data tells you which cases actually occur, and a rule covering the eleven real cases is often better than a model covering the infinite hypothetical ones.

Removal is the fourth and it is legitimate. Where the feature is uneconomic at any shape, where the cost of a wrong answer is unbounded, or where nobody uses it, take it out. Say why in the release notes. A product that removes things is a product somebody is editing.

A worked example, composited and labelled

The details here are composited from ordinary production shapes rather than one system, and the shape is exact.

A sales product ships AI-generated follow-up emails that send automatically after a call. Six months later the numbers are ambiguous and everybody has an opinion.

The threshold table, written before launch, is unambiguous. Correction-free acceptance is not applicable because the feature acts. Reversal rate, meaning follow-ups the rep cancelled during the delay window from Chapter 4, is at eleven per cent against a ceiling of five. Cost per successful interaction is fine. Abstention is fine. Review minutes are near zero by design.

One threshold crossed, one action specified: demote from act to draft. The email is now composed and waits for the rep to send it. Reversal becomes irrelevant. The cancel has been replaced by a decision.

Segment analysis found the rest of the story. Cancellations were concentrated in one team selling a product line with different terminology, and after that corpus was added the drafting acceptance was high enough to argue for promoting the feature back for the other teams. The evidence produced a narrower, better feature rather than a cancellation.

The objection: cutting the feature reads as the AI bet failing

This is the real reason features survive their own evidence, and it is worth answering at the level it is actually held.

MIT's Project NANDA report, The GenAI Divide: State of AI in Business 2025, puts about 95% of enterprise generative AI pilots at no measurable profit and loss impact, against enterprise investment it puts at 30 to 40 billion dollars. Its method was 52 executive interviews, surveys of 153 leaders and analysis of 300 public deployments. It attributes the gap to integration and organisational learning rather than to model quality.

Read that as the cost of not cutting. A portfolio of pilots nobody stopped is exactly what produces a large investment with no measurable return. Organisations that look like they are failing at AI are frequently organisations failing to stop.

The credible position is the opposite of the fear. Being able to say that you ran eight AI features, kept three, demoted two and removed three, with the evidence for each, is a stronger claim about competence than eight features still described as promising. It is also how the three that work get the attention they need.

There is a resourcing argument that usually settles it. Every retained feature consumes evaluation, monitoring, review capacity and on-call attention indefinitely, and that budget is finite. Keeping a weak feature is not neutral. It spends the capacity the strong one needs.

Chapter summary

A feature with no kill criterion never gets cut. Without a pre-agreed threshold the case for stopping must be argued in a room against the people who built it, using evidence whose interpretation is itself the argument, which makes it political and therefore biased towards continuing. A kill criterion needs five parts: a named signal, a threshold value, an evaluation window, an owner with authority to act alone, and the specific action taken. A signal without a threshold is a dashboard. A threshold without an owner is a suggestion. Five signals carry the decision, each failing in a different direction. Correction-free acceptance, which is more honest than acceptance because expensive corrections manufacture acceptances. Reversal rate, the cheapest ungameable quality signal a product produces. Cost per successful interaction, compared against the value of the interaction rather than against last month. Abstention rate, whose rise means the input distribution has moved. And review minutes per interaction, which decides whether the feature is leverage or a fixed-cost service. Thresholds must be set before any data exists, on a fixed review calendar rather than when somebody becomes worried. Arkes and Blumer named the mechanism in 1985, describing the sunk cost effect as a greater tendency to continue once money, effort or time has been invested, justified by not wanting to appear wasteful. That is why the answer is pre-commitment rather than better judgement. And cutting usually means demoting a rung, narrowing to the segment where the feature works, or replacing the model with a rule covering the cases production actually produced, with removal reserved for features that are uneconomic at every shape.

That is the book. Its argument in one line: the error rate of a confidently wrong component is a design input rather than a defect, so the product is screened on verifiability and reversibility, shaped as suggest, draft or act before anything else is decided, given a real undo and an explicit commit point, gated on signals from outside the model, routed per step with a fallback and an honest degraded state, priced per successful interaction including review minutes, budgeted for latency before it is built, cached on a key holding the principal and the corpus revision, fed by corrections rather than by its own output, and shipped with the wrong answer designed and the stopping number already written down. The measure of whether it worked is not that the model was never wrong. It is that when it was, somebody had already decided what the product would do about it.