Restoring Deployability

Restoring Deployability is the repair that makes every other repair deliverable. A system whose releases are rare, manual and frightening cannot be improved incrementally, which means every improvement grows until it is too large to verify. This chapter is about making the release itself uninteresting.

Key takeaways

• The test of deployability is social, not technical. Would the least senior person available run this deploy at two on a Tuesday, alone, without a checklist held by somebody else.
• Reproducibility comes first. A release that cannot be rebuilt from a commit, with its configuration and its migrations, is a release you cannot reason about after it fails.
• Rollback deserves the same design effort as the deploy, and the same practice. A rollback path that has never been exercised this month is a hypothesis rather than a control.
• Schema migrations are the part that does not reverse cleanly, so they get separated from code releases and made backward compatible in both directions for at least one release.
• Deployment frequency and failed deployment recovery time are the two numbers that say whether this worked, and both are measured in production rather than claimed in a runbook.

Read this beside Chapter 5, which baselined these numbers, and Chapter 9, whose small batches are only possible once this chapter is done. Chapter 11 hands the result to the team that will live with it.

Week one asked whether the system could be deployed and got a qualified yes: it can be deployed by one person, on a good day, in about four hours, with a document that is mostly accurate. Everyone has been treating that as a nuisance rather than as a defect.

It is the defect. Almost every other pathology in this book is downstream of it.

Why the release path is the highest-leverage repair

A slow, risky release changes team behaviour in ways that look like unrelated problems.

Changes get batched, because shipping twice costs twice. Batches make failures harder to diagnose, because a failed release now contains fourteen changes rather than one. Diagnosis time rises, so releases become events that need scheduling, which makes them rarer, which makes them larger. The loop closes and tightens.

DORA's current guidance names the two ends of that loop directly. Deployment frequency is the number of deployments over a period, and failed deployment recovery time is how long it takes to recover from a deployment that fails and needs immediate intervention. A team with a four-hour manual release has both numbers bad for the same reason, and the second one is the one that hurts during an incident.

The trunk-based guidance from DORA puts the constraint in a form that is easy to test against your own system: branches lasting hours rather than weeks, merging at least daily, and no code freeze periods. None of that is available to a team that cannot ship on demand. Deployability is the precondition for the way of working, not a consequence of it.

Reproducible before automated

The instinct is to automate the existing release. Do that second.

Automating an unreproducible release produces a faster way to reach an unknown state. First establish that a given commit, plus a known configuration, plus a known migration state, produces the artifact running in production. That sounds obvious. On a stalled delivery it is frequently false, and the falseness is where the mysterious failures come from.

Three checks find most of it. Build the current production version from the tagged commit and compare it with what is deployed, byte for byte where you can, dependency tree at minimum. Enumerate every configuration value the running system reads and find each one's source of truth, expecting to discover two or three that exist only in a console somebody edited. Then check whether the migration state in production matches the migration history in the repository, because a schema changed by hand under incident pressure is the most common single cause of an environment that cannot be reproduced.

Write down what you find as a list of divergences with owners. That list is the work, and it is usually a week of it. Automation applied afterwards is straightforward. Automation applied before is the reason the second attempt at deployability fails after the first one already did.

Rollback is a designed path, not an emergency

Ask a team what happens if a release goes wrong and you will usually get a description of a decision, not of a mechanism. Someone will be called, the situation will be assessed, and then either a fix will be rolled forward or the previous version will be restored by hand.

That is not a rollback. It is an improvisation with a name.

A designed rollback has four properties. It is one command or one button. It is performed by the same person who deployed, without permission. It restores a known version rather than reversing a change. And it is practised, on a schedule, in production, so that the first real use is not the first use.

Practising sounds indulgent to a business that has been waiting six weeks for its roadmap. It is cheaper than the alternative in one particular way that is worth naming in the argument: a team that trusts its rollback stops treating deploys as decisions, and a team that stops treating deploys as decisions ships smaller batches, which is the mechanism this whole chapter exists to enable.

Flags do part of this job better than a redeploy can. Hodgson's ops toggles let a behaviour be switched off without shipping anything, which is the fastest available form of reversal on a system whose deploys are still slow. Use them for the paths you have most recently repaired, and remove them once the repair has held for a month, as Chapter 9 requires.

The migration is the part that does not reverse

Code rolls back. Data does not, and pretending otherwise is how a routine failure becomes a recovery project.

Separate the two. Ship the migration in its own release, before the code that needs it, written so the current code keeps working unchanged. Then ship the code. Then, a release later, remove what the old code needed. That is three deploys instead of one, and each of them is individually reversible, which is the point.

Two rules make the pattern hold in practice. Additive changes only during the repair phase: add a column, backfill it, read from it, and drop the old one weeks later once nothing reads it. And no destructive statement runs inside a release window at all, so that a dropped column can never coincide with a failing deploy at the moment attention is scarcest.

Backfills of any size get their own treatment, because a migration that locks a large table is an outage regardless of how correct it is. Run them in batches, resumably, outside the release path, with progress recorded somewhere a person can read at three in the morning.

The environment that does not match is the one you will test in

Most stalled deliveries have a staging environment. Few have one that predicts anything, and a staging environment nobody trusts is worse than none, because it absorbs effort and returns false confidence.

The divergences are always the same three. Data, where staging holds a thousand synthetic rows and production holds four million real ones with a decade of edge cases. Configuration, where staging points at test credentials and sandbox endpoints, so the code paths that touch a real provider never run. Scale, where a single instance hides every problem that only appears when two of them race.

You cannot close all three on a rescue budget, and you should not try. Decide instead what staging is for and state it. If it exists to catch schema and deployment mechanics, keep it small, keep it current, and stop asserting it proves behaviour. If it needs to predict behaviour, it needs masked production data from Chapter 6 and real integrations in sandbox mode, and that is a project with a budget rather than an afternoon.

Then move the remaining confidence into production, where it belongs. Dark releases, fractional enables and the comparison pattern from Chapter 9 give you evidence from the only environment that is real, which is why they outperform a staging environment nobody quite believes.

Make the deploy observable while it is happening

Most teams find out a release failed from a customer. That is a monitoring gap that looks like a deployment gap.

The deploy should announce itself into whatever the team already watches, with the version, the commit and the person who ran it. Then a small number of checks should run automatically after it and be visible without anyone opening a dashboard: does the application answer, does one representative write path work end to end, are the error rates and latencies within a stated band compared with the ten minutes before.

The comparison window matters more than the absolute thresholds on a system you are still learning. You do not yet know what normal is. You do know what it was ten minutes ago, and a change that appears within a minute of a deploy is attributable in a way that no absolute threshold can match.

Google's error budget framing supplies the decision rule to attach to this: when the budget is being spent faster than planned, releases slow down until reliability recovers. On a rescue, use the objective set in Chapter 5 from the team's own recent history, and let the deploy checks feed it rather than feeding a separate opinion.

Prove it with the least senior person in the room

The completion test for this chapter is deliberately a social one, because every technical test can be satisfied by a system only one person can operate.

Have the newest team member perform a real release, unaided, in the afternoon, while you watch and say nothing. Anything they have to ask about is an unfinished part of the repair. Anything they get wrong is a missing guardrail rather than a training issue. Anything that requires a credential only one person holds is a bus factor finding that belongs in Chapter 11's handover, not in a document nobody reads.

Then measure rather than declare. Deployment frequency should be rising, failed deployment recovery time falling, and the deployment rework rate from Chapter 5 should be visibly lower than it was in the baseline. If the numbers have not moved after three weeks of this work, the release path was not the constraint, and the honest move is to say so and go back to the sequence in Chapter 9.

Chapter summary

A slow, risky release is not a nuisance downstream of the real problems, it is upstream of most of them, because it forces batching, and batching lengthens diagnosis, which makes releases rarer and larger in a loop that tightens on itself. Restore reproducibility before automating anything: rebuild the deployed version from its commit, enumerate every configuration value and find its source of truth, and reconcile the production migration state with the repository, recording the divergences as owned work. Design the rollback with the same care as the deploy, so that it is one action, permissionless, restoring a known version, and practised on a schedule, with ops toggles carrying the fastest reversals on paths repaired most recently. Treat migrations as the part that does not reverse: separate them from code releases, make them additive during the repair phase, keep destructive statements out of release windows entirely, and run large backfills resumably outside the path. Make the deploy observable while it happens, announcing itself with version and author, followed by automatic checks that compare against the ten minutes before rather than against thresholds you have not earned yet, feeding the error budget set in Chapter 5. Then prove it socially, by having the newest person release unaided while you watch, and confirm it numerically with deployment frequency, failed deployment recovery time and deployment rework rate. If those numbers do not move, the release path was not the constraint and the sequence needs revisiting.

The system can now be changed, verified and released by the people who own it. That sentence contains the whole remaining problem, because ownership is not yet where it needs to be. Chapter 11 is Handing Back Ownership and Documentation: what a team must be able to do unaided before you leave, which documents are worth writing and which are theatre, and why the handover starts in week two rather than in the final week.