Measuring a Reliability Horizon for Your Own Workload

Measuring a Reliability Horizon for Your Own Workload is the chapter that makes the rest of the book empirical. Every recommendation so far has a cost, and none of them can be justified without a number that says how unreliable your system currently is at the lengths of work you care about.

Key takeaways

• Three inputs: representative tasks labelled with the time a competent person takes, a success definition a machine can check, and repeated runs so you measure a distribution rather than a draw.
• Bucket by human duration, not by step count. Step count is a property of your decomposition and changes when you refactor; human duration is a property of the work.
• Measure at the success rate you actually need. A 50% horizon is a research statistic, and the operational question is where the curve crosses 90% or 99%.
• Report intervals. METR's own figures carry ranges spanning factors of four, and a point estimate from twenty runs will be less certain than that.
• Re-measure on a schedule and after any change to models, tools or prompts, because the horizon is a property of the whole system rather than of the model in it.

Read this beside Chapter 1, which introduced the horizon as an external measurement, and Chapter 5, whose gate results supply much of the data. Chapter 12 puts the resulting numbers on a dashboard somebody watches.

Someone asks whether the agent can be trusted with the quarterly reconciliation. It takes an analyst most of a day.

The honest answer is not yes or no. Nobody has measured anything of that length. The longest task the system has been evaluated on takes twenty minutes. That gap is the entire question.

What you are measuring, and what you are not

A reliability horizon is the relationship between how long a task takes a competent person and how often your system completes it correctly. It is a property of a pipeline rather than of a model: your prompts, your tools, your gates, your retries and your model together.

That distinction is what makes the measurement worth doing at all. METR's Time Horizon 1.1, published 29 January 2026, measures models on a fixed task suite. Their figures are the right reference for tracking the field: a post-2023 doubling time of 131 days, with Claude Opus 4.5 at 320 minutes for the 50% horizon and an interval from 170 to 729 minutes. What those numbers cannot tell you is how your system behaves on your work, because your tools are not their tasks.

The March 2026 reliability framework makes the same point from the failure side, with domain-specific decay: software engineering degrading from 0.90 to 0.44 across the horizon while document processing stayed nearly flat. If decay depends that strongly on domain, an inherited number is not evidence about yours.

You are also not measuring capability. You are measuring completion under your constraints, which is the only thing that predicts what happens when you deploy.

Build the task set from what you actually run

Twenty to fifty tasks is enough to start, and where they come from matters more than how many there are.

Take them from production. Sample real requests across the range of lengths you care about, including the awkward ones: the request with missing data, the one with an unusual account configuration, the one that failed last month. A task set assembled from clean examples measures a system nobody operates.

Then label each with human duration. Not your estimate. Time somebody competent doing it, or reconstruct it from records if the work is already tracked. This is the expensive part of the exercise. It is also the axis the whole measurement hangs on. Ten carefully timed tasks beat fifty guessed ones.

Bucket by duration rather than by step count. Under five minutes, five to thirty, thirty minutes to two hours, two to eight hours, and beyond if your work goes there. Step count is tempting because it is free, and it moves whenever you change the decomposition, which makes historical comparison meaningless.

Keep the set stable and version it. When you add tasks, record which version of the set produced which measurement, because a curve that improved because the tasks got easier is the most common self-deception in this exercise.

Define success so a machine can check it

A measurement that needs a human to judge each run will be done once. Then never again.

Write a checker per task, in the spirit of the acceptance conditions from Chapter 5: the final state of the records, the presence and content of the artifacts produced, the absence of unintended changes measured by diff. Where the output is text, check the structural properties that matter, such as required fields present and figures matching source, rather than similarity to a reference answer.

Three failure classes need separating in the result, because they lead to different work. Wrong output, meaning the run finished and the answer is incorrect. Incomplete, meaning the run stopped or escalated before finishing. And damaging, meaning the run produced side effects it should not have, which is a category no benchmark reports and your business cares about most.

Anthropic's September 2025 tool guidance recommends evaluating with realistic tasks and iterating from the results, including letting agents analyse the results to improve the tools. That loop is available here too, and the honest constraint is that the checker must not be written by the same process that produces the answer.

Run it enough times to see the spread

One run per task gives you a draw from a distribution. Long-horizon distributions are wide.

Run each task at least five times, more for the longest bucket, and record the distribution rather than the mean. The variance amplification described in the 2026 reliability work means the spread grows with task length, so the longest bucket needs the most repetitions to say anything at all.

Report the result as a rate with an interval per bucket. Five successes in five runs is not a 100% success rate, it is a small sample consistent with anything above roughly 55%, and stating that plainly is what keeps the exercise honest. METR's own published intervals span factors of four on the longest tasks, and their sample sizes are larger than yours will be.

Also record what each run cost and how long it took in wall-clock terms. A system that succeeds 95% of the time at four times the price of a person is a measurement result too, and it belongs in the same table as the success rate.

What it costs to run, and how to make it affordable

The objection to all of this is cost, and it is a fair one. Fifty tasks at five runs each is two hundred and fifty runs, and if the longest bucket takes hours, that is a substantial bill.

Three moves keep it affordable. Stratify the sampling: run the short buckets many times because they are cheap, and the long buckets fewer times while reporting wider intervals for them, rather than dropping them entirely. Run the full set on a schedule and a reduced smoke set on every change, so daily feedback is cheap and quarterly measurement is thorough. And harvest production: every real run already produces a history, and if you can label outcomes from the gate results in Chapter 5, a large part of the curve comes free from work you were doing anyway.

Production harvesting has a bias worth naming. Real traffic is not evenly distributed across durations, so the long buckets will be thin, and the tasks people send to an agent are the ones they already believe it can do. Keep the curated set for the lengths production does not reach.

There is one cost that is not optional. The longest bucket is the one everybody wants an answer about and the one nobody measures, because it is expensive and slow. A horizon measured only up to twenty minutes cannot answer a question about a seven-hour task, and saying so plainly is more useful than extrapolating.

Read the curve, then use it

The output is a curve. Three readings come off it immediately.

Where it crosses your required success rate is your usable horizon, which is the number to quote when someone asks what the agent can be trusted with. Where it falls steeply is where the architecture in this book pays: gates before the drop, checkpoints across it, escalation after it. And how it compares with your last measurement tells you whether the changes you made helped, which is the only way to know whether the gate you added last month was worth its latency.

Attribute the failures as well as counting them. The April 2026 diagnostic work across four domains and more than 3,100 trajectories demonstrates that long-horizon failures can be located at specific steps and causes, and your run histories from Chapter 7 contain the same material. Group failures by step and by cause, and the top two usually account for most of them.

Then convert the curve into a policy. Google's error budget framing supplies the shape: state the reliability you intend to provide for a task family, measure against it, and let the gap govern what you ship. A task family below its target gets architecture work rather than a wider deployment, and one comfortably above it can have a gate removed and the latency returned to users.

Re-measure, because the horizon moves

A horizon is a measurement of a system at a moment. Every part of that system changes.

Re-measure on a schedule, quarterly at minimum, and additionally after any model version change, any tool change, any prompt change large enough to be reviewed, and any change to the gates. Model updates are the case teams handle worst. A newer model is assumed to be better, and the assumption is untested on the axis that matters. The 2026 reliability work found rankings shifting at longer horizons, so a model that wins your short evaluation can lose your long one.

Keep the history. A table of horizon by quarter, by task family, is the most informative artifact your team will own about its own agents, and it answers the question executives ask most often, which is whether this is getting better.

One caution about drift in the other direction. Task sets rot: the production distribution shifts, and a set assembled a year ago may no longer represent the work. Refresh a portion of the set each cycle, keep a stable core for comparability, and record which is which.

Chapter summary

A reliability horizon relates how long a task takes a competent person to how often your pipeline completes it correctly. It is a property of the whole system rather than of the model. That is why METR's published figures track the field and cannot describe your workload, particularly given how strongly the 2026 reliability work found decay varying by domain. Build the task set from production, including the awkward cases. Label each task with a measured human duration rather than an estimate. Bucket by duration rather than by step count, because step count moves when you refactor. Version the set, so improvements cannot come from easier tasks. Define success so a machine can check it, separating wrong output from incomplete runs and from damaging runs, and keep the checker independent of the process being checked. Run each task several times because long-horizon distributions are wide, report rates with intervals rather than as point estimates, and record cost and wall-clock time alongside success. Read the curve for your usable horizon at the success rate you actually require, for where the steep drop is so the architecture can be placed there, and against your last measurement so changes can be judged, attributing failures to steps and causes from the run histories. Then convert it into an error-budget-style policy that governs deployment. And re-measure quarterly and after every model, tool, prompt or gate change, keeping the history and refreshing part of the task set each cycle while holding a stable core.

You now have an architecture and a number that says whether it is working. What remains is the day-to-day: the alerts, the rota, the kill switch and the review. Chapter 12 is On-Call for Agents, which is about operating software that acts on its own, and about the one question an on-call engineer must always be able to answer within a minute of being paged.