Migrating a Running System

Migrating a Running System is the chapter where design stops being greenfield, and it is the one that interview material has no reason to contain. A migration is not a change from one state to another; it is a sequence of intermediate states, each of which serves live traffic for days or weeks, and each of which therefore has to be correct on its own terms.

Key takeaways

• Every intermediate state is production. Design each phase to be safe alone, because you will be in at least one of them longer than planned.
• The pattern is expand, migrate, contract. Sato's 2014 write-up gives the shape: augment the interface to support old and new, move every client, then remove the old version.
• Compare before you trust. Stripe's 2017 account of the pattern reads both paths and compares results before either becomes authoritative.
• Intermediate versions are unavoidable at scale. Google's F1 paper constrains servers to be no more than one schema version apart and replaces unsafe changes with a sequence of safe ones.
• Throttle the backfill on a signal the primary produces. GitHub's gh-ost tails the binary log instead of using triggers, which makes the migration dynamically throttleable and testable against a replica.

Read this after Chapter 11, whose cost model tells you what the migration is worth, and after Chapter 4, since most migrations exist because a data model stopped matching its questions. Chapter 13 turns the outcome into a record, which matters here more than anywhere: a migration is the most expensive consequence of an undocumented decision, and this chapter is what that consequence costs.

A schema change is written, reviewed and merged. It renames a column and updates every query that referenced it, which is thorough work.

It deploys during a rolling release. For four minutes, half the instances run the old code against the new schema, and every request they serve fails.

The change was correct. The intermediate state was never designed, because nobody wrote it down as a state.

A migration is a sequence of safe states

The definition that prevents most migration incidents is that there is no such thing as a change. There are only states.

Between the old design and the new one there are intermediate configurations where two schema versions exist, or two code versions, or two stores. Each one runs production traffic. Each one must be independently correct, because your rollout takes minutes at best and your backfill takes days.

That reframing produces the design instruction. Enumerate the intermediate states before writing any code, name what is authoritative in each, and name the rollback available from each. If a state has no rollback, it is a point of no return and it deserves to be marked as one.

Most migration failures are a state nobody enumerated. Not a bug in the new code.

Expand, migrate, contract

The general form is small enough to memorise, and it applies to schemas, APIs, message formats and configuration equally.

Sato's ParallelChange entry on martinfowler.com, dated 13 May 2014, gives the three steps. Expand, meaning augment the interface to support both the old and the new version. Migrate, meaning move every client to the new version. Contract, meaning remove the old version once nothing uses it.

The discipline is the ordering. And the patience. Expand is additive and therefore safe: add the column, keep the old one, write both. Migrate is where the time goes, because it is bounded by the slowest client rather than by your deploy. Contract is the step teams skip, and skipping it leaves a permanent tax of dead columns, dual writes and compatibility branches nobody dares remove.

Write the contract step into the plan with an owner and a date. Otherwise it does not happen.

Compare both paths before trusting either

The most valuable phase is the one that produces no user-visible change at all, and it is the one most often cut for time.

Stripe's engineering write-up on online migrations at scale, published 2 February 2017, describes a four-phase pattern. Dual write to the old and new stores. Backfill existing data into the new store. Change reads to come from both, comparing the results before trusting either. Then move reads to the new path, and finally writes.

The comparison phase is the argument. It runs in production, on real traffic, with real data shapes, and it finds the cases your test fixtures did not contain: the null nobody expected, the encoding difference, the rounding, the record written by a version of the code that no longer exists.

The same write-up gives the arithmetic that forces the backfill to be parallel. Processing a hundred million objects sequentially at one second each would take over three years. Redo that with your own row count before agreeing to a timeline.

Servers must tolerate more than one version

There is a formal result behind the intermediate-state discipline, and it comes from a system that had no choice about it.

The F1 paper, by Rae, Rollins, Shute, Sodhi and Vingralek at VLDB 2013, addresses schema change in a globally distributed database with stateless servers and no global membership. Its finding is that such a change cannot be applied atomically. Servers must be allowed to run different schema versions, and the protocol constrains them to be no more than one version apart. Changes that would cause corruption are replaced by a sequence of individually safe changes.

Every part of that transfers to an application with more than one instance. Your rolling deploy is a period of two code versions. Your migration is a period of two schema versions. The one-version-apart constraint is the practical rule: never require a jump of two, because the intermediate is what production actually runs.

This is why a column rename is three deployments rather than one. Add, dual write and read preferring the new, then remove.

Throttle on a signal the primary produces

A backfill is a large amount of work aimed at the same database serving your users, and its throttle is what stops the migration becoming an incident.

GitHub's post on gh-ost, published 1 August 2016, makes the mechanism concrete. Its argument against trigger-based online schema change is that triggers compete for locks with the application's own writes and cannot truly be paused. Tailing the binary log instead decouples the migration from the primary's workload, which makes it dynamically throttleable on replication lag or load, and it allows the migration to be tested against a replica before it touches a primary.

Take three things from that even if you never use the tool. Drive the backfill from a log or a cursor rather than from a mechanism inside the write path. Throttle on a signal the database already produces, such as replication lag, rather than on a fixed rate you guessed. And rehearse the migration on a replica or a restored copy first.

The last one is the cheapest insurance here. Rehearse it once. A backfill that has run on a copy of production data is a backfill with a known duration.

The five phases, and the rollback at each

This table is the artifact to lift into your own plan. Fill in the last column honestly, because an empty cell there is a point of no return.

The fourth row is where most of the risk actually sits, because the two stores are now authoritative for different operations. Keep that phase short. Keep the old store writable until the comparison has been clean for longer than your longest reporting cycle.

Incremental replacement beats rewrite

The largest version of this decision is whether to migrate at all or to rebuild, and the argument against rebuilding is empirical rather than aesthetic.

Fowler's StranglerFigApplication entry, updated 22 August 2024, gives three reasons incremental replacement wins. The details of existing behaviour are hard to specify. Users cannot wait for a long rebuild. And much of the old behaviour is not actually wanted, so specifying it fully wastes effort. The pattern proceeds by finding seams and running both systems side by side.

That third reason is the one people miss. A rewrite forces you to decide what every existing behaviour was for, including the ones that exist because of a bug in 2019 that a customer now depends on. Incremental replacement lets you discover which of those matter by observing which paths are still used.

The connection to this chapter's mechanics is direct. A strangler fig is expand, migrate, contract applied at the level of a whole system, with the comparison phase run per seam rather than per table.

What actually blocks migrations, as reported

The failure mode teams expect is technical, and survey evidence points somewhere less interesting and more expensive.

Flexera's 2026 State of the Cloud Report, published 18 March 2026, reports the top migration challenges as understanding application dependencies at 54 percent, assessing technical feasibility at 44 percent, and comparing on-premises with cloud cost at 43 percent. Dependency mapping leads. It sits ahead of anything about the migration mechanism itself.

That should reorder your plan. The first artifact is not a schema diff. It is a list of every consumer of the thing you are moving, including the report somebody built in a spreadsheet, the integration a customer wrote against an undocumented field, and the job that reads the table directly rather than through your service.

On the current enthusiasm for AI-assisted migration, two 2026 vendor posts are worth reading as commentary rather than as evidence. Tembo's June 2026 piece on legacy code modernisation with agents and Abstracta's June 2026 piece asking who validates that the business still works are both first-party marketing rather than measurement. The second one names the right question, which is the comparison phase, and that question is answered by the mechanics in this chapter rather than by the tooling.

The objection: dual writes are too much work

They are genuinely a lot of work, and the objection is usually a preference for a maintenance window instead.

Sometimes a window is correct. A small dataset, a product with a genuine quiet period, a customer base in one timezone, and a change that fits in twenty minutes: take the window, write down the rollback, and skip the rest of this chapter. That is a legitimate decision and it should be recorded as one.

The window stops being available at a size that arrives sooner than teams expect. The Stripe arithmetic is the boundary: once your backfill is measured in days, no window exists, and the only remaining options are the phased pattern or an outage.

The cost comparison is also worth stating plainly. Dual writes cost engineering time that you schedule. The alternative costs an unplanned outage plus a data reconciliation, at a time you do not choose, with a rollback you have not tested. The first cost is larger on paper and smaller in practice, which is the same asymmetry every chapter in this book has been pointing at.

Chapter summary

A migration of a running system is a sequence of intermediate states rather than a change, and each state serves production traffic for days or weeks, so each must be independently correct with a named rollback. Sato's ParallelChange of 13 May 2014 gives the general form of expand, migrate, contract, with contract being the step teams skip and pay for permanently. Stripe's 2 February 2017 write-up supplies the four-phase version with the crucial comparison step, reading both paths and comparing results before either is trusted, and its arithmetic that a hundred million objects processed sequentially at one second each would take over three years is why the backfill must be parallel. The F1 paper from VLDB 2013 establishes why intermediate versions are unavoidable, since a schema change in a distributed system with stateless servers cannot be atomic, servers must be allowed to differ and are constrained to be no more than one version apart, which is why a column rename is three deployments. GitHub's gh-ost post of 1 August 2016 argues against triggers because they compete for locks and cannot pause, and for tailing the binary log so the migration is dynamically throttleable on replication lag and testable on a replica first. Fowler's strangler fig entry, updated 22 August 2024, gives the reasons incremental replacement beats rewrite, and Flexera's 18 March 2026 survey puts understanding application dependencies as the top migration challenge at 54 percent. The chapter's decision is a phase plan with the rollback named at every step.

The migration is the bill for a decision nobody recorded. The final chapter is the artifact that prevents the next one, and it is short enough that there is no excuse for its absence. Chapter 13 is Writing the Decision Record.