Tracing With OpenTelemetry GenAI Conventions

Tracing With OpenTelemetry GenAI Conventions is about getting the structural layer right so that everything else in this book has something to attach to. It is also about knowing precisely where a standard ends, because the evaluation data that makes a trace useful is not part of it.

Key takeaways

• The conventions moved. As of 29 July 2026, the OpenTelemetry site's GenAI semantic conventions page states they have relocated to a dedicated repository and is no longer maintained there.
• Adopt them for the structural layer: request, model call, tool call, retrieval, with consistent attribute names, so traces are portable across tools and vendors.
• They describe what happened. They do not describe whether it was correct, and no tracing standard will, because correctness is a property of your product.
• Attach your evaluation layer to the same trace: case identifier, criterion verdicts, judge version, rubric version, override state.
• Traces contain prompts, outputs and retrieved documents, which makes them a sensitive store with a retention decision attached.

Read this beside Chapter 5, whose trajectory assertions read these traces, and Chapter 4, whose system metrics are trace-derived. Chapter 12 is the incident response that depends entirely on what was recorded here.

An answer goes wrong in a way nobody can reproduce. The logs show a request, a response, a latency and a status code, all of which look entirely normal.

They do not show which documents were retrieved, which model version answered, how many tool calls happened or what the prompt contained. The investigation ends in a hypothesis, which is how the same defect ships twice.

Check where the conventions live before citing them

The first practical note is a currency check, and it is unusually load-bearing for a standards chapter.

Fetched on 29 July 2026, the page at opentelemetry.io/docs/specs/semconv/gen-ai/ states that the GenAI semantic conventions have moved to a dedicated repository and that the page is no longer maintained. The repository describes itself as holding semantic conventions for generative AI, including spans, metrics and events for GenAI clients, for the Model Context Protocol, and for provider-specific conventions, with documentation generated from model definitions.

Two implications follow. Any tutorial or internal document citing the old location is out of date by at least one move. Attribute names quoted from memory are likely wrong.

So read the current repository rather than a summary of it. Record the date and version you implemented against. This area has moved twice in the lifetime of most production LLM systems, and it will move again.

That is not an argument against adopting them. A moving standard with a shared vocabulary is still worth more than a bespoke schema, and the move itself is a sign of the area getting more serious rather than less.

What the structural layer should record

Attribute names change. The structure a useful trace needs is stable, and it is worth stating independently of the spelling.

One span per request, carrying the user or tenant identity in whatever pseudonymised form your policy allows, the product surface, and the overall outcome.

One span per model call, with the provider, the model identifier including version, the input and output token counts, the temperature and other sampling parameters, the latency, and whether the call was a retry.

One span per tool call, with the tool name, the structured arguments, the result status and the elapsed time, which is what Chapter 5's invariants are asserted against.

One span per retrieval, with the query issued, the number of candidates, the identifiers of what was selected and their ranks, which is what the context work in a companion volume needs.

And the parent-child relationships kept accurate, including for parallel calls, because a trace whose ordering is ambiguous cannot support a precedence assertion.

Where the conventions stop, and why

A tracing standard records what a system did. Whether that was correct is a judgement against criteria specific to your product. No convention will supply it.

That boundary is the useful thing to understand, because teams keep waiting for a standard to solve evaluation and it structurally cannot. Your rubric from Chapter 2 is yours. Your strata from Chapter 3 are yours. Your judge's verdicts are the output of a component you built and calibrated.

So plan two layers from the start. The structural layer follows the conventions and is portable. The evaluation layer is yours and attaches to the same trace and span identifiers.

AgentLens' contribution is instructive on the second layer: it produces readable trajectory reviews rather than a bare score, which is exactly the kind of artifact that belongs attached to a trace rather than in a separate spreadsheet.

Attach the evaluation layer to the same identifiers

The evaluation layer is small. It must share keys with the trace, or the two are useless together, which is the most common implementation mistake in this chapter.

Per request, record the dataset case identifier where the run came from the suite, the rubric version, the judge version, the verdict per criterion, the reason string for any failing judged criterion, the trajectory invariant results, and the gate outcome including any override.

Per span, record anything stage-specific worth attributing later, such as whether a retrieval stage returned the known-correct document for a labelled case.

Then make sure both layers land in the same store, or at least in stores that can be joined by trace identifier. The most common implementation failure here is evaluation results in one system and traces in another, joined by nothing, which produces two dashboards and no investigations.

The payoff arrives during an incident. A single query that returns the trace, the verdicts, the model version and the retrieved documents for a bad answer is the difference between a two-hour investigation and a two-minute one.

Instrument before you need it, not during

There is a sequencing rule that costs nothing to follow and a great deal to ignore.

Add the tracing when the feature is built. Not when the first incident happens, because during an incident the instrumentation you are adding does not describe the request that already failed, and the change itself is a deploy under pressure.

Start with the structural layer, which is a day of work with a library, and add the evaluation layer as the suite from Chapter 3 comes into existence. The two grow together naturally, because both are keyed on the same requests.

Make it a definition-of-done item for any new capability: a new tool, a new retrieval source or a new task type ships with its spans. Retrofitting instrumentation across a system that grew without it is one of those tasks that never reaches the top of a backlog.

And test the trace on a real failure before you rely on it, which is the exercise at the end of this chapter.

Sampling, cost and what to keep

Full-fidelity tracing of every request is expensive at volume. Prompts and outputs are the largest fields by far.

Sample deliberately, in three tiers. Keep everything for evaluation runs, since the volume is small and the value is high. Keep a random sample of production, sized so that the online quality series in Chapter 10 has enough data. And keep everything for requests that failed, escalated, were flagged by a user or violated an invariant, which is where the diagnostic value concentrates.

Store large payloads by reference where possible: the prompt template plus the variable parts, rather than the assembled text; the retrieved document identifiers rather than their bodies. That keeps a trace readable and shrinks it by an order of magnitude.

Set retention per tier, deliberately, and keep it shorter for the full-payload tiers. A trace store that grows without a policy becomes two problems at once. A bill, and a liability.

Traces are a sensitive store

Everything Chapter 3 says about the evaluation dataset applies here with more force, because traces are collected automatically and at volume.

A trace of a real request contains the user's words, the retrieved documents, and the model's output, which together may include personal data, credentials pasted by users, and content the requester was entitled to see but the person reading the trace is not. OWASP's 2025 entry on sensitive information disclosure is the reference point.

Four controls. Redact known secret shapes at ingestion rather than at display, because display-time redaction protects nobody with query access. Apply access control to the trace store that reflects the sensitivity of what it holds, which is usually stricter than ordinary application logs. Keep retention short for full payloads and longer for structural metadata, which is cheap and rarely sensitive. And record the pseudonymised subject identifier so a deletion request can be honoured against traces as well as against the primary store.

Say plainly in the design document that traces contain user content. It is obvious to engineers. It is frequently news to whoever signs off the data policy.

Make the trace readable by a person

The last requirement is the one that gets designed out by tooling choices.

Somebody at three in the morning needs to read one request end to end. On one page. Without asking anybody. That means a view that shows the spans in order, with the prompt, the retrieved identifiers, the tool calls and the final output, on one page, in a form a human can scan.

Two habits protect it. Keep span names stable and descriptive rather than encoding parameters into them, so the same operation is recognisable across releases. And include a short human-readable summary attribute on the request span, such as the task type and the outcome, which turns a trace list into something searchable by eye.

Test it the way Chapter 12 will need it. Take a failed request from last week, hand the trace to somebody who did not build the feature, and time how long it takes them to say what went wrong. If it takes more than a few minutes, the instrumentation is not finished.

Chapter summary

OpenTelemetry's GenAI semantic conventions supply a shared vocabulary for tracing model and tool calls, and the first practical step is a currency check, because the opentelemetry.io page states as of 29 July 2026 that the conventions have moved to a dedicated repository and is no longer maintained, which makes any older tutorial and any remembered attribute name suspect. Adopt them for the structural layer, recording one span per request, per model call with provider, versioned model identifier, token counts and sampling parameters, per tool call with structured arguments and result status, and per retrieval with the query, candidate count and selected identifiers with ranks, keeping parent-child relationships accurate enough to support precedence assertions. Understand where they stop: a trace records what happened, correctness is a judgement against your own criteria, and no convention will ever supply it, which is why the evaluation layer is a second layer sharing the same trace and span identifiers, carrying case identifier, rubric and judge versions, per-criterion verdicts with reasons, invariant results and gate outcome. Land both layers in stores that can be joined, since evaluation results and traces joined by nothing produce two dashboards and no investigations. Sample in three tiers, keeping everything for evaluation runs and for failures while sampling ordinary production, storing large payloads by reference and setting retention per tier. Treat the trace store as sensitive, with ingestion-time redaction, stricter access control than ordinary logs, short retention for payloads and a pseudonymised subject identifier for deletion requests. And keep traces readable by a person end to end, with stable span names and a human-readable summary, tested by handing a real failure to somebody who did not build the feature.

Instrumentation exists so that an incident can be explained. Chapter 12 is Incident Response for Systems That Are Fluently Wrong, which is the last chapter: how to detect an error that reads as confident, how to contain it when the output has already reached people, and how to run a review that produces a system change rather than a note about the model.