Metrics That Matter

The Dashboard That Lied

A company celebrates engineering improvement. Deployment frequency is up. Cycle time is down. Pull request volume is healthy. The team looks more productive.

At the same time, customer activation is falling.

The team is moving faster, but not toward the right outcome.

Metrics did not fail. The metric stack was incomplete.

Output metrics versus outcome metrics

Output metrics tell you what was produced. Outcome metrics tell you what changed.

Output examples:

• Tickets completed.
• Pull requests merged.
• Deployments shipped.
• Story points completed.
• Features released.

Outcome examples:

• Activation improved.
• Churn reduced.
• Conversion increased.
• Support volume decreased.
• Incident recovery improved.
• Time to value shortened.

You need both. Output without outcome creates motion theatre. Outcome without delivery visibility creates mystery.

Product metrics engineers should understand

Engineers do not need to own every product metric, but they should understand the ones their work affects:

• Activation: are users reaching first value?
• Retention: do users keep coming back?
• Conversion: do users move through important steps?
• Engagement: are users using the core workflow?
• Expansion: are accounts adopting more value?
• Support volume: are users getting stuck?
• Time to value: how long until the user gets the promised payoff?

These metrics help engineers connect code to behavior.

Engineering delivery metrics

Delivery metrics help teams improve flow:

• Lead time for changes.
• Deployment frequency.
• Change failure rate.
• Mean time to recovery.
• Review time.
• Work in progress.
• Blocked time.

These metrics should improve the system, not surveil individuals. If a metric makes people hide work or avoid hard tasks, it is damaging the culture.

Reliability metrics

Reliability metrics connect system health to user trust:

• Availability.
• Error rate.
• Latency.
• Incident count.
• Incident severity.
• Recovery time.
• Queue delay.
• Data freshness.

The right reliability metric depends on the user journey. A slow report may be acceptable in one product and disastrous in another.

Developer productivity without surveillance

Measuring developers like factory workers creates bad behavior. Lines of code, raw ticket counts, and individual velocity often punish thoughtful work.

Better questions:

• Where does work get stuck?
• What causes rework?
• Which systems slow delivery?
• Which reviews wait too long?
• Which incidents repeat?
• Where do developers lack self-service paths?
• What interrupts deep work?

The goal is to improve the environment, not rank humans by shallow metrics.

The Builder Metrics Stack

The Builder Metrics Stack has six layers, each one built on the layer below it:

1. Team health
2. System health
3. Delivery health
4. Product behavior
5. Business impact
6. User outcome

Measure what helps you decide. A healthy team does not optimize one layer while ignoring the others.

Dashboards that create decisions

A dashboard is useful only if it improves decisions.

For every dashboard, ask:

• Who uses it?
• What decision does it support?
• How often is it reviewed?
• What action follows a change?
• Which metric can be removed?
• Which metric needs context?

Metric theatre happens when dashboards are created for visibility but not decision-making.

Founder lens

Leaders need evidence, but evidence must be connected. A founder should be able to see whether engineering speed, product behavior, system reliability, and business outcomes are moving together.

When metrics conflict, the team has something important to learn.

Developer lens

For your product area, create a small scorecard:

• One user outcome metric.
• One business impact metric.
• One product behavior metric.
• One delivery health metric.
• One system health metric.

Review it after releases. Ask what changed and why.

AI-era lens

AI-assisted development needs evaluation. Do not measure only AI usage. Measure whether AI improves lead time, review quality, test coverage, defect rate, documentation usefulness, or learning speed.

For AI product features, measure quality, fallback rate, user correction, task success, latency, and cost.

Common mistakes

• Measuring output and assuming value.
• Using individual productivity metrics carelessly.
• Creating dashboards nobody uses for decisions.
• Ignoring product metrics as "not engineering."
• Optimizing delivery speed while reliability declines.
• Measuring AI adoption instead of AI outcomes.

Builder checklist

• I know the outcome metric my work supports.
• I can distinguish output from outcome.
• I use delivery metrics to improve flow.
• I connect reliability metrics to user journeys.
• I avoid surveillance-style productivity thinking.
• I can design a decision-oriented dashboard.
• I evaluate AI workflows by results.

Exercise: build a five-metric scorecard

Choose one product area and define:

• One user metric.
• One business metric.
• One product behavior metric.
• One delivery metric.
• One system health metric.

For each, write what decision it helps the team make.

Closing thought

Measure what helps you decide, not what helps you pretend.

Key takeaways

• Output metrics (tickets, PRs, deployments) and outcome metrics (activation, churn, conversion) answer different questions; a team can improve one while the other quietly gets worse.
• Delivery and reliability metrics exist to improve the system, not to surveil individuals: a metric that makes people hide work is damaging the culture it is meant to protect.
• A dashboard only earns its place if it supports a named, recurring decision; otherwise it is metric theatre.
• For AI-assisted work, measure outcomes (lead time, defect rate, task success) rather than adoption alone.