Wiring Evals Into CI as a Release Gate

Wiring Evals Into CI as a Release Gate is where an evaluation programme either becomes part of how the team ships or becomes a dashboard nobody opens. The technical work is straightforward. The rules and the ownership are what decide the outcome.

Key takeaways

• Block on hard requirements and on regressions against a baseline. Do not block on a graded score that moved inside its error bar, because that produces false alarms and false alarms end gates.
• Run the suite on every change to prompts, retrieval, tools, models and the assembler, since all five change behaviour and only some of them look like code.
• Give the gate a named owner and a logged override with an expiry. An override that is a Slack message is an override with no memory.
• Separate the fast gate from the full run. Minutes on every commit, the complete suite nightly, and the held-out set on a schedule.
• Track the gate's own metrics: how often it fires, how often it was right, and how often it was overridden. A gate nobody trusts is worse than no gate.

Read this beside Chapter 7, whose error bar decides what may block, and Chapter 4, whose three families behave differently at the gate. Chapter 10 extends the same rules into production with shadow traffic and canaries.

The suite is written, it runs, and the numbers are on a dashboard. Three months later somebody notices the dashboard has been red since a change in week two.

Nothing enforced it. Nobody owned it. The suite was accurate the whole time, and it changed nothing, which is the most common failure of an evaluation programme and it is organisational rather than technical.

What a gate blocks on

Precision here prevents most of the pain later. Three rules cover a working gate.

Any hard requirement failing is red. These are the binary criteria from Chapter 2: invented figures, missing citations, answering when the case required a refusal, permission violations. One failure is enough, because these are correctness failures rather than quality movements.

A regression beyond the band on any stratum is red. Not the overall average. A system that improves on common lookups and degrades on refusals has a serious problem that an average hides, and Chapter 3's strata exist for this.

A graded score moving inside its error bar is a pass. Chapter 7's rule is load-bearing here: on fifty cases, two cases is not a result, and a gate that blocks on noise will be disabled by the end of the quarter.

Add a fourth rule if your system takes actions: any trajectory invariant from Chapter 5 failing is red, regardless of the outcome. A run that produced the right answer while violating a prohibition is a defect.

Run it on the changes that actually change behaviour

The gate has to trigger on the right set of events, and in this class of system that set is wider than the code.

Five categories change behaviour. Prompt changes, including system prompts, tool descriptions and judge prompts. Retrieval configuration, including chunking, indexes, top-k and rerankers. Tool definitions and their descriptions. Model version, including provider-side updates you did not initiate. And the assembler, meaning anything that decides what goes into the window.

Three of those are usually not in a normal code review path. Prompts get edited in consoles, model versions change under aliases, and retrieval parameters live in configuration. Move all of them into version control first, or the gate simply will not fire when it matters.

Provider-side model updates deserve their own mechanism, since no commit accompanies them. Pin versions where the provider allows it, and run the full suite on a schedule so an unannounced change is caught by the nightly rather than by a customer.

Split fast and full

A gate that adds twenty minutes to every commit will be routed around. A gate that runs only nightly does not block anything.

Run both. The fast gate is the mechanical checks plus a judged sample, sized to finish in a few minutes, on every commit and every pull request. The full run is the whole suite including all judged criteria, nightly, plus on demand before a release.

That split follows the cost structure from Chapter 6: mechanical checks are free and stable, judged criteria cost money per case. Sampling the judged half keeps the fast gate cheap without giving up coverage, because the nightly closes the gap.

Then the held-out set, on its own schedule, monthly or per release rather than per commit. Chapter 3's seal only holds if the set is rarely touched, and running it in CI would destroy it within weeks.

AgentLens' authors describe nightly pipelines for exactly this shape of regression detection, which is a useful precedent when arguing for the compute.

Thresholds come from the baseline, not from ambition

Absolute thresholds are the most common design error in a gate.

"Faithfulness must exceed 90%" sounds rigorous and is arbitrary. It will either sit far below current performance, in which case it never fires, or far above it, in which case everything is blocked and the rule is suspended on day one.

Set thresholds relative to a recorded baseline instead. The baseline is the last release that shipped and was not rolled back, stored with its dataset version, model version and scores. The rule is then a comparison: no stratum may fall more than the error bar below its baseline.

Update the baseline deliberately, on release, rather than automatically on every green run. Automatic baselines ratchet downward through a series of individually acceptable regressions, and six weeks later the system is measurably worse with no rule ever having fired.

Google's error budget framing gives the policy layer above this. State the quality you intend to hold, spend the difference deliberately, and make the trade explicit rather than letting it happen through threshold drift.

The override is the most important design decision

Every gate needs an override, because sometimes shipping is correct despite a red run. The question is what shape the override takes.

Four properties make it work. It is named, so a specific role may authorise it rather than anyone with commit access. It is logged, with the reason, the failing cases and the person, in a place that can be reviewed later. It expires, so an override applies to one deploy rather than becoming a permanent exemption. And it is counted, because the override rate is the health metric for the whole programme.

An override rate that is climbing means one of three things. The gate is producing false alarms and needs tightening in the statistical sense. The suite has drifted from what the business considers important. Or the team is shipping known regressions under delivery pressure, which is a management conversation rather than an engineering one.

All three are worth knowing and none is visible without the log. In my experience the first is most common in the first quarter and the third is most common after that.

Make a red run actionable in one click

Gates die from friction as often as from false alarms.

A red run should link straight to the failing case identifiers from Chapter 3, the diff between this run and the baseline, the actual outputs, and the judge's stated reason where a judged criterion failed. The engineer who broke it should be able to see what broke in under a minute.

Two details make a disproportionate difference. Show the previous passing output beside the new failing one, because most regressions are obvious side by side and invisible in isolation. And name the change that most likely caused it, which is usually derivable from what was committed.

Keep the output stable across runs so it can be diffed. A gate whose report is a fresh narrative each time cannot be compared, and comparison is the whole point.

Introduce the gate in stages

A gate switched on at full strength on a Monday blocks the week and loses the argument. Three stages avoid that.

Stage one is advisory. The suite runs on every change and reports, and it blocks nothing. Two or three weeks of this establishes the false-alarm rate and lets the team fix the obviously broken cases without pressure.

Stage two blocks on hard requirements only. These are the criteria nobody disputes: invented figures, missing citations, permission violations. Blocking on them is uncontroversial precisely because a failure is unambiguous, and it establishes that a red run means something.

Stage three adds regression rules on the strata, once the baseline has been stable long enough to be meaningful and the error bars are known. This is the stage that needs the override policy in place first, because it is the stage that will occasionally block a release somebody wants.

Announce each stage before it starts and say what will change. A gate that gains authority quietly is experienced as an obstacle, and a gate that gains it openly is experienced as a decision the team made.

Report the gate's own health

The gate is a system and it needs its own metrics, reported monthly.

Fire rate, meaning how often it blocked. True positive rate, meaning how often a block corresponded to a real defect, established afterwards. Override rate and reasons. Mean time to green after a block. And flakiness, meaning how often two runs of the same commit disagree.

Flakiness above a low threshold is an emergency for the gate's credibility. Non-zero temperature, provider variability and judged criteria all contribute, and the fixes are known: fix seeds where the provider allows it, run judged criteria with deterministic settings, and treat any case that flips without a change as a suite defect to be fixed or removed.

Publish the health report to the same audience that sees the quality numbers, because a gate whose credibility is deteriorating is a problem the whole team owns rather than a maintenance detail. Keep these separate from DORA's delivery metrics, which measure how well you ship rather than what you shipped. Both matter, and conflating them produces a dashboard that answers neither question.

Chapter summary

A suite becomes a gate when it runs automatically, blocks on rules agreed in advance, and has a named override. Block on any hard requirement failing, on a regression beyond the error bar on any stratum rather than on the overall average, and on any trajectory invariant violation, while explicitly not blocking on graded movement inside the band, since false alarms are what get gates disabled. Trigger it on the five things that change behaviour, being prompts, retrieval configuration, tool definitions, model version and the assembler, which means moving prompts and configuration into version control first and running a scheduled full pass to catch provider-side model changes that arrive with no commit. Split the fast gate, mechanical checks plus a judged sample in minutes on every commit, from the nightly full run and the rarely opened held-out set, following the cost structure of mechanical versus judged scoring. Set thresholds relative to a recorded baseline from the last shipped release rather than to an absolute ambition, and update the baseline deliberately on release so it cannot ratchet downward through individually acceptable regressions. Design the override as named, logged with reasons and failing cases, expiring after one deploy, and counted, because the override rate distinguishes a noisy gate from a drifted suite from a team shipping known regressions under pressure. Make a red run actionable in one click with case identifiers, the baseline diff, the outputs side by side and the judge's reason. And report the gate's own health monthly, including fire rate, true positives, overrides, time to green and flakiness, treating a flipping case as a suite defect rather than as background noise.

The gate protects what you can measure before shipping. Chapter 10 is Online Evaluation and Guarded Rollouts, which covers the part that only exists in production: shadow traffic, canaries, the signals worth watching in the first hour, and the rule that triggers a rollback without a meeting.