Measuring Delivery When DORA Stops Working

Measuring Delivery When DORA Stops Working is the chapter that arrives late on purpose. You cannot judge which metrics still discriminate until you hold the gates, queues and owners that generate them, which is what Chapters 5 through 10 built.

Key takeaways

• DORA's 2025 report presents software delivery performance as two factors and five metrics: throughput as lead time for changes, deployment frequency and failed deployment recovery time, and instability as change fail rate and rework rate.
• DORA states the limit in its own words: while these five metrics provide a vital snapshot of performance, they are ultimately outcomes, telling you what is happening but not explaining why.
• Two credible instruments disagree. DORA's survey reports AI positively related to throughput; Faros AI's telemetry reports lead time up 480.4% in the roughly 10% of its dataset that instruments deployment, and contests DORA's method by name.
• DORA warns that logs-based metrics are not automatically objective, because instrumentation approaches vary, errors create inaccuracies and interpretation is subject to bias.
• Never publish a delivery metric per engineer. DX's own guidance says speed and throughput metrics used in isolation often incite fear and counterproductive behaviours, and that diffs per engineer requires caution.

Read this after Chapter 10, because the ownership and gate records are what makes any of these numbers attributable, and before Chapter 12, which prices the two cost lines this measurement exposes. Chapter 1's three starter numbers are the subset to instrument first.

The board asks a reasonable question. What did the tool spend buy.

The dashboard shows deployment frequency up and lead time roughly flat, which supports no narrative at all. Somebody produces a vendor report showing throughput up 59%, and somebody else produces one showing lead time up 480%.

Both numbers are real. Neither is about this company, and neither was measured the way the person reading it assumes.

The framework changed and most dashboards did not

DORA restructured the thing most engineering organisations still call the four keys. Its State of AI-assisted Software Development 2025 was published 24 September 2025, fielded from 13 June to 21 July 2025, with 4,867 respondents and 78 in-depth interviews.

Software delivery performance is now presented as two factors over five metrics. Throughput comprises lead time for changes, deployment frequency, and failed deployment recovery time. Instability comprises change fail rate and rework rate, where rework rate is the ratio of deployments that are unplanned but happen as a result of a production incident.

The regrouping is not cosmetic. It separates the speed of the pipe from the damage it does, which is the distinction this book has made since Chapter 1. It also puts recovery time on the throughput side rather than treating it as a quality measure.

If your dashboard still shows four tiles in the old grouping, it is measuring a framework its authors have moved on from. Check the grouping before you argue about the numbers.

What the five metrics can and cannot tell you

DORA is explicit about the ceiling, and the sentence is worth quoting rather than paraphrasing. While these five metrics provide a vital snapshot of performance, they are ultimately outcomes. They tell you what is happening, but they don't explain why.

That is the honest limitation and it has a practical consequence. Every one of the five can move for reasons that have nothing to do with the thing you changed, so a delivery metric is a prompt for an investigation rather than an answer.

DORA's own illustration is that a low deployment frequency might be caused by technical debt, bureaucratic processes, or team constraints. Its response is a cluster analysis producing seven team profiles with stated shares: foundational challenges 10%, legacy bottleneck 11%, constrained by process 17%, high impact low cadence 7%, stable and methodical 15%, pragmatic performers 20%, and harmonious high-achiever 20%.

The useful move for a leader is to hold the five outcome metrics and one explanatory instrument beside them. Which explanatory instrument is the argument in this chapter.

Which metrics stopped discriminating

Some of the five degraded specifically because the unit of change got smaller and more numerous. That is the mechanism from Chapter 1 arriving in the measurement layer.

Two rows deserve emphasis. Deployment frequency is the metric most likely to be quoted at you and least likely to mean what the quoter thinks, because it rises mechanically when changes get smaller. And the instability pair held up best, which is why DORA's regrouping put them together.

The 2024 figure, and why it is history

One number circulates as though it were current and it is not. DORA's 2024 work estimated a 1.5% reduction in throughput and a 7.2% increase in instability for every 25% increase in AI adoption.

The 2025 report restates that as history rather than as its own finding. Its 2025 result is a reversal on the throughput side: AI adoption now shows a positive relationship with software delivery throughput and with product performance, while continuing to show a negative relationship with software delivery stability, and no relationship with friction or burnout.

Quoting the 2024 figure today is a dated-evidence error, and it is one I have seen made in both directions. If you are going to cite DORA on this, cite the year.

The stability half did not reverse. That asymmetry is the finding, and it is consistent with everything in Chapters 7 and 8.

Two credible instruments disagree

This is the part no page on this subject handles honestly, so it gets its own section rather than a footnote.

DORA's instrument is a survey. Self-reported, 4,867 respondents, estimated standardised effects with 89% credible intervals, fielded mid-2025. Faros AI's instrument is telemetry. Two years of data across 22,000 developers and more than 4,000 teams, per-company standardised metrics, Spearman rank correlation at p below 0.05. Its comparison sets each team's two lowest AI-adoption quarters against its two highest, analysed as of March 2026.

They disagree. DORA reports AI positively related to throughput. Faros AI reports lead time from commit to production up 480.4%, with its own caveat that this rests on roughly 10% of its dataset and should be treated directionally. Faros AI also reports that high pre-AI engineering maturity offered no protection in its data, which sits against DORA's framing of AI as an amplifier of existing strengths.

Faros AI contests the method directly, writing that DORA's findings are based on surveys, that what surveys cannot capture is what happens downstream, and that perception lags reality while telemetry does not. Faros AI sells engineering-productivity analytics, and that conclusion supports what it sells, which is worth holding in mind without dismissing the argument. DORA has not published a reply.

I am not going to pick a winner for you. The rule I would apply is procedural. Measure with logs where you have the instrumentation, with surveys where you do not, publish which instrument each number came from, and never let a single instrument decide a headcount question.

How a delivery figure gets distorted in transmission

There is a live example of this worth learning from, and I will describe the mechanism without naming the publisher, because the point is the mechanism.

CircleCI's report of 18 February 2026 states that average daily workflow runs rose 59% year over year, while the median team saw only about 4%, the top 5% saw 97%, and the bottom quartile showed no measurable increase. It separately reports that for the median team, feature-branch throughput rose 15% while default-branch throughput fell 7%.

A widely read summary rendered the first of those as feature-branch throughput up 59% year over year. Two distinct errors are stacked in one phrase. An average was reported as though it described a typical team, when the median was around 4% and the distribution was extremely skewed. And a count of workflow runs was relabelled as branch throughput, which is a different quantity measured a different way.

The result is a number that is four times too large for the median team and about a thing nobody measured. It then appears in slide decks.

The defence is a habit rather than a policy. For any figure you are about to repeat, ask three questions. Average or median. Of what population. Measuring exactly what unit.

What to add, and what DORA says about adding it

The temptation at this point is to throw the framework out and buy a new one. DORA advises against it, and its guidance is specific enough to act on.

Instead of throwing out the entire framework, DORA writes, you can use existing measures as a baseline to help identify how a paradigm shift is changing the developer experience. It suggests adding metrics on the acceptance rates of AI suggestions, model quality or trust, while keeping existing measures such as perceived productivity and time spent reviewing code.

Note that second clause. Time spent reviewing code is on DORA's own keep list, which is the metric Chapter 5 is built on and the one most organisations do not collect.

DORA also punctures the assumption that instrumenting your way out of this is automatically more rigorous. It is a common misconception that logs-based metrics are objective, DORA writes, because instrumentation approaches vary, errors can create inaccuracies, and interpretation is subject to bias. Two structural frameworks are worth knowing here. The SPACE paper by Forsgren, Storey, Maddila, Zimmermann, Houck and Butler, published in ACM Queue in February 2021 and therefore older than eighteen months, states that developer productivity cannot be measured by a single metric or dimension. DX's Core 4 presents four dimensions, speed, effectiveness, quality and impact, as a unification of DORA, SPACE and DevEx, and its own AI measurement framework proposes utilisation, impact and cost.

Never at the individual level

One rule in this chapter is absolute, and the vendor whose framework is most often used to justify breaking it says so itself.

DX's own guidance warns that speed and throughput metrics used in isolation often incite fear and counterproductive behaviours from developers. It says diffs per engineer requires caution, and lists three preconditions. Counterbalance it with an opposing metric. Set no targets or rewards on it. And roll it out carefully.

I would go further and simply not publish it. The failure is not that engineers game it, though they will. It is that generated volume makes the number nearly meaningless while making it look more precise, so it produces confident wrong conclusions about people.

Team level, with an explanatory instrument beside it, is where these numbers belong. Anything finer becomes a performance conversation, and a performance conversation built on a corrupted proxy costs you the trust you need for everything in Chapter 14.

The objection: if you cannot measure it you cannot manage it

The objection is that all this caution amounts to refusing to answer the board, and it deserves a better response than a caveat.

METR's work is the cautionary tale I would use, because it turned its own result over in public. Its randomised trial of July 2025 found experienced open-source developers took 19% longer with AI tools while estimating afterwards that they had been sped up by 20%. That page now carries a notice saying the results are out of date. Its update of 24 February 2026 reports a confidence interval on the original of +2% to +39%. The follow-up estimate was -18% for the original cohort, with an interval of -38% to +9%, and -4% for newly recruited developers. METR now states that developers are more sped up from AI tools in early 2026 than its early-2025 estimates suggested. It also says selection effects make the data only very weak evidence for the size of that increase, noting that 30% to 50% of developers declined to submit tasks they did not want to do without AI.

Then a second METR result on the other side of the same question. Its survey of 11 May 2026, from 349 technical workers, reports a median 1.4x to 2x change in the value of their work and a median 3x speed change. The authors themselves state there are reasons to be sceptical of the magnitude. It is self-reported. Say so whenever you cite it.

So the answer to the board is not a number. It is a method: two instruments, both dated, both named, with their disagreement reported rather than averaged. That is a more defensible answer than a single confident figure, and it is the only kind of answer the current evidence supports.

Chapter summary

DORA's report of 24 September 2025 restructured software delivery performance into two factors and five metrics. Throughput is lead time for changes, deployment frequency and failed deployment recovery time. Instability is change fail rate and rework rate. The regrouping separates the speed of the pipe from the damage it does. DORA states the ceiling plainly: the five are ultimately outcomes, telling you what is happening but not why, which makes each one a prompt for investigation rather than an answer, and its cluster analysis produces seven team profiles with stated shares. Deployment frequency is the metric most degraded by the shift, because it rises mechanically as the unit of change shrinks, while the instability pair held up best. The 2024 estimate of a 1.5% throughput reduction and 7.2% instability increase per 25% adoption increase is restated as history in the 2025 report, whose own finding reverses the throughput sign while keeping the negative relationship with stability. Two credible instruments then disagree. DORA's survey of 4,867 respondents with 89% credible intervals sits against Faros AI's telemetry across 22,000 developers, which reports lead time up 480.4% on roughly 10% of its dataset, finds no protection from pre-AI maturity, and contests DORA's method by name while selling analytics that its conclusion supports. Figures also degrade in transmission, as when an average daily workflow-run increase of 59% was repeated as feature-branch throughput for a median team that actually saw about 4%. DORA advises keeping the framework, adding acceptance rate, model quality and trust measures, retaining perceived productivity and time spent reviewing code, and remembering that logs-based metrics are not automatically objective. Never publish these per engineer.

Measurement exposes two costs that never appear on an invoice. Chapter 12 is The Two Cost Lines Nobody Counts, which prices review hours and incident cost against the seats and tokens finance already tracks.