Sequencing Repairs Without Freezing Delivery

Sequencing Repairs Without Freezing Delivery is the scheduling problem that decides whether a rescue is survivable inside a working business. The constraint is not effort. It is that two streams of change have to enter the same codebase every week without colliding, while the codebase is the thing being repaired.

Key takeaways

• Order repairs so each one makes the next cheaper. The deployment path and the tested perimeter come first because every later repair is priced against them.
• Batch size is the variable that decides whether repair and delivery can share a week. DORA's guidance on trunk-based development is three or fewer active branches, merged at least daily, with branches living hours rather than weeks.
• A repair that needs a long-lived branch is not a repair, it is a second project. Restructure it into a sequence of merges behind a flag, or move it to the bottom of the list.
• Release and deploy are separate events. Shipping dark, then enabling for a fraction of traffic, converts a risky repair into an observation.
• Treat flags as inventory with a carrying cost. Every one needs an owner and a removal date, or the repair leaves behind a second system of conditionals nobody can reason about.

Read this beside Chapter 5, which set the two-lane week, and Chapter 6, which built the perimeter this chapter spends. Chapter 10 takes the deployment path itself from workable to reliable.

Two weeks after the reset is agreed, the collision happens. A delivery change and a repair touch the same module, the merge takes two days, and the resulting build fails in a way neither change causes alone. Somebody proposes that repair work move to its own branch until it is done.

That proposal is the single most reliable way to end a rescue quietly. The branch will live for six weeks. It will merge in one afternoon nobody enjoys, and the delivery work it collides with will be blamed for the collision.

Order the repairs so each one buys the next

Sequencing by severity feels right and produces the wrong order. The worst problem in a stalled system is usually structural, expensive, and impossible to do safely until several cheaper things are true.

Order by leverage instead. A repair earns its place by how much it reduces the cost of the repairs that follow it, which produces a fairly consistent sequence across different systems. First the deployment path, enough that a change can reach production the same day it is written, because every later repair is priced against the cost of shipping it. Then the tested perimeter from Chapter 6 around the paths the repairs will touch. Then the seams that let those paths be changed without surgery. Then the structural work, which is now unremarkable rather than heroic.

There is a second ordering test that catches the exception. Ask what happens if this repair is abandoned halfway. Repairs that are safe to abandon halfway can start now. Repairs that leave the system worse in the middle need the perimeter finished first and a plan for the intermediate state, which is what Fowler's strangler fig provides at the architectural level: the new grows beside the old, and each increment is a complete, shippable position rather than a step in an unfinished migration.

Write the sequence down as a list of positions rather than a list of tasks. A position is a state the system can sit in indefinitely without embarrassment. A sequence made of positions can be paused at any point, which is the property that survives contact with a customer escalation.

Batch size decides whether the two lanes can coexist

The collision in week two is a batch-size problem wearing a scheduling costume.

DORA's guidance on trunk-based development is specific: three or fewer active branches, merging to trunk at least once a day, with branches that typically last no more than a few hours, and no code freeze periods. Teams that work this way show higher software delivery performance. On a rescue the reasoning is more direct than the correlation. Two streams of change can only share a codebase if neither stream ever accumulates enough divergence to make a merge into an event.

So repair work gets decomposed until each piece is mergeable today. Not designed today. Merged today. Extracting one call into a method is a merge. Adding one characterisation test is a merge. Introducing an interface with a single implementation, still unused, is a merge. The structural change you are actually after arrives as the fortieth of those, at which point it is a rename and a routing switch.

Teams resist this, and the objection is legitimate: forty small merges take longer in total than one large one. That is true and it is the wrong comparison. The right comparison includes the merge conflicts avoided, the delivery work that did not block, the ability to stop at any point, and the diagnosis time on the failures that did occur, which is where Tornhill and Borg's finding about long maximum cycle times in low quality code does its damage. Small batches are how you keep the tail short in a codebase that produces long tails.

Release dark, then enable slowly

Repair and delivery can share a week without sharing a fate, and the mechanism is separating the deploy from the release.

Pete Hodgson's taxonomy from October 2017 is the working vocabulary here. Release toggles let an incomplete change ship dormant. Ops toggles let a behaviour be turned off in production without a deploy, which matters on a system whose deploys are still slow. Permissioning toggles let a change reach one customer or one internal user first. Experiment toggles are the fourth kind and rarely earn their place during a rescue.

The pattern that suits repair work best is a comparison rather than a switch. Run the new implementation alongside the old, send it the same input, compare the outputs, log the differences, and serve the old result. That converts the scariest category of repair, which is replacing logic nobody fully understands, into an observation you can watch for a week before anyone depends on it. The characterisation net from Chapter 6 gives you the expected shapes; production traffic gives you the ones you did not think of.

Then enable in fractions rather than all at once. Decide the abort condition before the first fraction, in numbers. Not "if it looks bad". A rate, a threshold and a name.

Flags are inventory, and inventory has a carrying cost

The failure mode of the previous section is a codebase carrying ninety flags, forty of them permanently true, six of them interacting.

Hodgson's framing is the one to hold: teams treat toggles as inventory with a carrying cost, and work to keep that inventory low. Each flag adds a branch to reason about and a combination to test. On a rescue the risk compounds, because the flags are being added to code whose behaviour was already partly unknown.

Three rules keep it bounded. Every flag gets an owner and a removal date at the moment it is created, recorded where the team looks rather than in a comment. Every flag that survives its date becomes an agenda item rather than an accepted fact. And removal is scheduled as work in the same week that the enable is completed, because a flag removed the same week is a five-minute change and a flag removed six months later is an archaeology exercise.

There is one exception worth keeping. An ops toggle around a genuinely unstable third-party dependency is legitimate long-lived inventory, and it should be documented as a control rather than treated as debt.

A practical note on where the flags live. On a system with a slow deploy, a flag that requires a deploy to change is not an ops control at all, and reading its value from configuration the application can reload is worth the afternoon it takes. Keep the mechanism boring. A table, a cached read, and a default that matches current behaviour will outperform a platform you have to learn during an incident.

The other question the team will ask is who may flip a flag in production. Answer it before the first flag exists, in writing, with names. A control that anyone can change and nobody owns produces the worst incident of the rescue: a behaviour that changed, at a time nobody recorded, for a reason nobody remembers.

Protect the perimeter with a rule the team can apply alone

Sequencing collapses if every decision has to route through the person holding the plan. The team needs a rule they can apply on a Tuesday without asking.

The rule that works is stated in terms of the perimeter from Chapter 6. Inside the perimeter, a change ships only with the characterisation net green and a flag if it alters observable behaviour. Outside the perimeter, a change ships on judgement. Anything that would move the perimeter, meaning new code paths that touch money, data or obligations, requires a test before it merges rather than after.

That single rule handles most of the traffic, keeps the two lanes from negotiating with each other, and makes the exceptions visible. Exceptions should be rare and logged, because the log is the diagnostic. If the same exception appears three weeks running, the perimeter is drawn in the wrong place and the plan needs to change rather than the team needing more discipline.

Watch the two numbers that say whether it is working

Sequencing is the part of a rescue most likely to feel productive while achieving nothing, so it needs external instrumentation rather than a sense of progress.

Two of DORA's current metrics carry it. Deployment frequency should rise as batches shrink, and change fail rate should fall as the perimeter closes. If deployments are rising while the change fail rate rises with them, the batches got smaller without the net getting stronger, and the correct response is to slow the repair lane rather than to add process. If neither number moves for a month, the work being sequenced is not on the critical path, and the sequence itself is wrong.

Deployment rework rate is the third number worth watching here, because it exposes the failure this chapter is designed to prevent. Unplanned deployments caused by production incidents are the signature of repair work landing in batches too large to verify, and it usually appears in the data a week before anyone in the room notices the pattern.

Chapter summary

Repair and delivery share a week or the rescue stops, and whether they can share it is decided by batch size rather than by discipline. Order repairs by leverage rather than severity, so that the deployment path comes first, then the tested perimeter, then the seams, then the structural work that is now ordinary, and use the abandonment test to catch the exceptions: a repair that is unsafe to stop halfway needs the perimeter finished before it starts. Write the sequence as a list of positions the system can sit in indefinitely, not as a list of tasks, because positions survive escalations. Keep three or fewer active branches, merge daily, and decompose repair work until every piece is mergeable today, accepting that forty small merges cost more in total and buy the short tail that a low quality codebase otherwise denies you. Separate deploy from release: ship dark, run new and old side by side comparing outputs on real traffic before anything depends on the new path, then enable in fractions with a numeric abort condition agreed in advance. Treat flags as inventory with an owner, a removal date and removal scheduled in the same week as the enable, with long-lived ops toggles around unstable dependencies as the one documented exception. Give the team a perimeter rule they can apply without asking, and log the exceptions, because three of the same exception means the perimeter is wrong. Then watch deployment frequency, change fail rate and deployment rework rate, which will tell you whether the sequence is working before the room does.

The repairs are landing and delivery has not stopped. What is still slow, and has been slow since the first week, is the act of putting anything into production at all. Chapter 10 is Restoring Deployability: how to get from a release that takes a day and a person who knows the incantations to one you would perform on a Tuesday afternoon, and why rollback deserves as much design as the deploy itself.