Observability as a Design Input

Observability as a Design Input is the argument that some of what you will need to see cannot be added after the fact, so the decision belongs in the design phase rather than in the operations phase. An identifier nobody propagated cannot be attached to requests that already ran, and a percentile computed before aggregation cannot be recombined into a correct answer.

Key takeaways

• Decide what you will need to see while the design is cheap. Propagation, identifiers and bucket boundaries are structural choices, not dashboard settings.
• Averages mislead. Google's SRE book notes that with a 100 millisecond average at 1,000 requests per second, one percent of requests might easily take five seconds.
• Record distributions rather than pre-computed percentiles. The OpenTelemetry metrics data model states that legacy Summary points cannot always be merged in a meaningful way.
• W3C Trace Context is a Recommendation dated 23 November 2021, with traceparent carrying version, a 32 hex digit trace-id, a 16 hex digit parent-id and trace-flags, so propagation is a contract on every hop.
• An SLI specification is the outcome you believe matters to users, and its implementation is how you measure it. The same specification supports several implementations at different cost.

Read this after Chapter 9, whose containment design you cannot verify without signals, and before Chapter 11, whose cost model consumes exactly the measurements this chapter decides to emit. That ordering is deliberate: unit cost is arithmetic over instrumented quantities, so observability decisions constrain what the bill can be explained in terms of. The companion argument about measurement with the authority to stop a release is what turns these signals into a gate.

An incident lasts fifty minutes. Twelve minutes are spent fixing it and thirty-eight are spent establishing which tenant was affected.

The traces exist. They cover every hop. None of them carries the tenant identifier, because the identifier was resolved at the edge, used for authorisation, and never attached to a span.

Adding it takes twenty minutes of work. It does not help, because it cannot be applied to traffic that has already happened.

Observability is a design input, not a purchase

The useful definition here is narrow and operational. Observability is a design input when the decision has to be made before the code exists. Making it later costs a migration, or is simply impossible.

Three things fall into that category. Propagation, meaning which identifiers travel on every call. Cardinality, meaning which attributes you attach and therefore which questions you can ask. And aggregation shape, meaning whether you keep distributions or throw them away at the edge.

Everything else is a purchase. Dashboards, alerting rules, retention windows, which vendor stores the data. All of those change on a Tuesday afternoon with no code change.

So the design review question is not what tooling we will use. It is which questions we will need to answer during an incident, and whether the data to answer them is being produced.

The four golden signals, and why the average lies

Google's SRE monitoring chapter provides the shortest defensible starting list, and one warning that changes how you instrument.

The four golden signals are latency, traffic, errors and saturation. The chapter is specific that latency for successful and failed requests must be tracked separately. A fast error and a slow success average into a number that describes neither. A flood of quick 500 responses can improve your latency figure while the service is failing.

The warning is arithmetic. At an average latency of 100 milliseconds and 1,000 requests per second, one percent of requests might easily take five seconds. Nothing in the average reveals it. The chapter's recommendation follows directly: collect request counts bucketed by latency with exponentially distributed boundaries, rather than averages computed for you.

That is a design instruction rather than a dashboard preference. Bucket boundaries are chosen in the instrumentation, and a boundary set too coarse is a question you cannot ask later.

Distributions merge, percentiles do not

There is a specific technical reason not to compute percentiles at the edge, and it is worth knowing precisely because it decides an implementation detail everywhere.

The OpenTelemetry metrics data model states that legacy Summary points cannot always be merged in a meaningful way. That is the whole problem with pre-computed percentiles. If each instance reports its own p99, no valid arithmetic combines twenty of those into a fleet p99. The number on your dashboard is then an average of percentiles, which is not a percentile of anything.

Histograms fix it. They carry bucket counts along with a count and a sum, so buckets add across instances. The same data model describes exponential histograms that support downscaling without error, so resolution can be reduced after collection rather than fixed at instrumentation time.

Exemplars are the third piece and they are underused. The data model provides for exemplars carrying trace and span identifiers. A point on a latency histogram therefore links to the specific trace that produced it. That turns a slow bucket into a clickable example rather than a search problem.

Trace context is a contract on every hop

Propagation is the part of observability that behaves like an architectural constraint, because it has to be honoured by every component or the trace breaks.

W3C Trace Context is a Recommendation, Level 1, dated 23 November 2021. Its traceparent header carries four fields: a version, a 32 hexadecimal digit trace-id, a 16 hexadecimal digit parent-id, and trace-flags. Its tracestate header carries up to 32 vendor key-value members. That is a small, fixed, unambiguous contract.

The design consequence is that every hop must forward it, including the ones you did not write. A queue that does not carry trace context in message metadata severs the trace at the asynchronous boundary. So does a third-party gateway, a batch job that reads rows rather than messages, and a webhook receiver.

Audit the hops before building, not after. For each boundary in Chapter 5's design, name how trace context crosses it. The boundaries where the answer is unclear are exactly the boundaries where your incidents will be slow.

The attribute you cannot add to yesterday

Cardinality decisions look like a data volume question and they are actually a question about which incidents you can diagnose.

The OpenTelemetry semantic conventions are versioned separately from the specification, at v1.43.0 as recorded in the research for this book. They cover general, database, HTTP, messaging, RPC, system, FaaS, CI/CD, cloud provider, feature flag and GraphQL domains. Use the standard attribute names rather than inventing your own. That is what makes a query portable across services and across tools.

The harder decision is the domain-specific attributes. Tenant identifier, cell identifier, plan tier, request criticality from Chapter 7, and whichever key your data is partitioned by. Each one is cheap at instrumentation time and impossible to retrofit onto traffic that has already passed.

Pick a small set and require it. Four or five attributes present on every span, enforced in a shared library rather than by convention, is the difference between answering which tenant and guessing.

Sample deliberately, instrument centrally

Coverage and overhead pull against each other, and the standard reference resolved that tension a long time ago.

The Dapper paper, published by Google in 2010, describes two choices that made ubiquitous tracing affordable. Sampling, so that overhead stays low while coverage stays broad. And instrumenting a small number of common libraries rather than every application, so that application-level transparency is achieved without asking every team to do work.

Both translate directly. Put propagation, span creation and the required attributes in one shared library or middleware that every service uses, and the coverage question becomes a dependency version question rather than a negotiation.

Sampling needs one caution. Sample errors and slow requests at a much higher rate than successful fast ones, because the interesting traces are rare by definition, and a uniform one percent sample will miss the case you needed.

An SLI specification is not its implementation

The last design decision is what number authorises a change, and the SRE Workbook draws a distinction that saves a lot of argument.

Its chapter on implementing SLOs separates the SLI specification, meaning the outcome you believe matters to users, from the SLI implementation, meaning how you actually measure it. The same specification has several implementations with different cost and coverage. Request success can be measured at the load balancer, in the application, or from a synthetic client. All three measure the same intent with different blind spots.

Two more numbers come from the same chapter. The error budget is 100 percent minus the SLO, and it needs a written policy for what happens when it is spent. Four-week rolling windows are a good general-purpose interval. And multiple grades of latency SLO are preferable to a single percentile, because users experience slow and very slow differently.

The error budget is the artifact that matters most for this book. It is a pre-agreed number that authorises a decision, which makes it the operational twin of the reversal signal Chapter 13 formalises.

A worked example, composited and labelled

The details here are composited from ordinary production shapes rather than one system, and the shape is exact.

A multi-tenant analytics product is being designed with eight cells. Before writing a service, the team lists the four questions it expects to ask during an incident: which tenants are affected, which cell, which dependency is slow, and whether the slowness is queueing or service time.

Those four questions determine the instrumentation. Tenant and cell identifiers become required span attributes, enforced in shared middleware. Queue sojourn time from Chapter 7 is emitted as a histogram separate from service time, because the fourth question cannot be answered from a single duration. Latency is bucketed with exponential boundaries from 1 millisecond to 30 seconds. Success and failure are separate series.

The SLI specification is written as a sentence: a tenant's report request returns correct data within four seconds. Two implementations are recorded, one at the edge and one from a synthetic client per cell, with the edge one as primary. The error budget policy says that spending the budget in a four-week window stops feature deploys until it recovers, which is a reversal signal with teeth.

The objection: we will add observability after launch

This is the most common plan and it is half correct, which is what makes it dangerous.

Dashboards, alerts and retention genuinely can wait. Nothing is lost by configuring them in week three, and configuring them before you have traffic usually produces thresholds that are wrong anyway.

The other half cannot wait, and the test is simple. Ask whether the decision affects the data being produced or only the way it is displayed. Propagation, required attributes and bucket boundaries affect production, so deferring them means the first month of traffic is undiagnosable, which is precisely the month with the most novel failures.

There is also a compounding cost. Retrofitting a required attribute across forty services is a coordination project, and it will be attempted during an incident by somebody who wanted to be fixing the incident. Do the small version now: one shared library, four attributes, histograms rather than percentiles, and trace context across every boundary.

Chapter summary

Observability is a design input for the three things that cannot be added later, being propagation of identifiers on every call, the attributes that decide which questions are answerable, and whether distributions are kept or discarded at the edge, while dashboards, alerts and retention remain purchases that can change on any Tuesday. Google's SRE monitoring chapter supplies the four golden signals of latency, traffic, errors and saturation, insists that latency be tracked separately for successful and failed requests, and gives the arithmetic that a 100 millisecond average at 1,000 requests per second is compatible with one percent of requests taking five seconds, recommending counts bucketed by latency with exponentially distributed boundaries. The OpenTelemetry metrics data model explains why pre-computed percentiles are the wrong shape, since legacy Summary points cannot always be merged in a meaningful way, while histogram buckets add across instances and exemplars link a metric point to the trace that produced it. W3C Trace Context, a Recommendation dated 23 November 2021, defines traceparent with version, a 32 hex digit trace-id, a 16 hex digit parent-id and trace-flags, plus tracestate with up to 32 members, which turns propagation into a contract every hop must honour, queues included. Dapper's 2010 answer to overhead was sampling plus instrumenting a few shared libraries rather than every application, and the SRE Workbook's separation of SLI specification from implementation, with an error budget of 100 percent minus the SLO over a four-week rolling window, gives the pre-agreed number that authorises a decision.

The signals are decided and the system can now be described in numbers. Those same numbers are the inputs to the question the design has been avoiding since chapter 1, which is what this will cost at the load you modelled and who will be asked to approve it. Chapter 11 is Capacity, Cost, and the Bill.