---
name: otel-tracing
description: >-
  OpenTelemetry tracing discipline: correct spans, propagation, and semantic conventions
  produce useful traces. Invoke whenever task involves any interaction with distributed
  tracing — span creation, context propagation, instrumentation, sampling configuration,
  or OpenTelemetry SDK setup.
---

# OTel Tracing

Create spans for logical operations, propagate context across every boundary, use semantic conventions for attribute
names. Every tracing decision — span granularity, attribute selection, sampling strategy — trades off cost against
visibility.

## References

| Topic                | Reference                                                  | Contents                                                               |
| -------------------- | ---------------------------------------------------------- | ---------------------------------------------------------------------- |
| Spans                | [`${CLAUDE_SKILL_DIR}/references/spans.md`]                | Span anatomy, root spans, lifetime code patterns                       |
| Span data            | [`${CLAUDE_SKILL_DIR}/references/span-data.md`]            | Events format, links format, SDK limits table                          |
| Context propagation  | [`${CLAUDE_SKILL_DIR}/references/context-propagation.md`]  | W3C header format, propagator selection, baggage details, security     |
| Instrumentation      | [`${CLAUDE_SKILL_DIR}/references/instrumentation.md`]      | Server/client/async code patterns, library rules, testing guidance     |
| Sampling             | [`${CLAUDE_SKILL_DIR}/references/sampling.md`]             | Head/tail/combined strategies, decision guide, sampler types           |
| Semantic conventions | [`${CLAUDE_SKILL_DIR}/references/semantic-conventions.md`] | HTTP/DB/messaging attribute lists, status mapping, general conventions |
| SDK components       | [`${CLAUDE_SKILL_DIR}/references/sdk-components.md`]       | Resource config, env vars, Collector deployment, exporter types        |

## Spans

A span represents a single unit of work in a trace — an HTTP request handler, a database query, a message publish. Not
every function call.

### Span Naming

Name spans for the **class** of operation, not the instance. Low-cardinality names enable aggregation; high-cardinality
names destroy it.

- Use the most general string that identifies a statistically interesting class
- Never embed high-cardinality values (IDs, emails, paths with IDs)
- HTTP span names: `{METHOD} {route}` — server: `GET /users/:id`; client: `GET`
- Database span names: `{operation} {table}` — `SELECT users`, `INSERT orders`

| Name                         | Verdict                   |
| ---------------------------- | ------------------------- |
| `get`                        | Too general               |
| `get_account/42`             | Too specific — ID in name |
| `get_account`                | Good                      |
| `GET /users/{userId}`        | Good — route template     |
| `POST /api/v2/orders/abc123` | Bad — order ID in name    |

### SpanKind

SpanKind tells backends how to assemble the trace tree. Set it correctly.

| Kind       | Direction | Use For                                |
| ---------- | --------- | -------------------------------------- |
| `SERVER`   | Incoming  | HTTP handler, gRPC server method       |
| `CLIENT`   | Outgoing  | HTTP client call, DB query, gRPC call  |
| `PRODUCER` | Outgoing  | Enqueue message, schedule job          |
| `CONSUMER` | Incoming  | Dequeue message, process job           |
| `INTERNAL` | Neither   | In-process business logic, computation |

- A single span MUST NOT serve more than one purpose
- Create a new span before injecting context for outgoing calls
- `CLIENT` -> `SERVER` for synchronous calls; `PRODUCER` -> `CONSUMER` for async
- Using `INTERNAL` for HTTP handlers or DB calls is wrong — use `SERVER` or `CLIENT`

### Span Status

| Status  | Meaning               | When to Set                                        |
| ------- | --------------------- | -------------------------------------------------- |
| `Unset` | No error              | Default — do not change on success                 |
| `Error` | Operation failed      | When an error occurs; include description          |
| `Ok`    | Explicitly successful | Only to override a previous `Error`; rarely needed |

- Leave status `Unset` on success — it already means "no error"
- `Ok` is a "final call" — once set, subsequent `Error` attempts are ignored
- Instrumentation libraries SHOULD NOT set `Ok` — leave to application code

### Span Lifetime

- **Every created span MUST be ended** — leaked spans cause memory issues and incomplete traces
- After `end()`, the span becomes non-recording; further mutations are ignored
- Use language-idiomatic patterns: `defer` (Go), try/finally (Java/JS), context manager (Python)

## Span Data: Attributes, Events, Links

### Attributes

Key-value pairs annotating a span. Value types: string, boolean, integer, float, or arrays of these.

- **Use semantic conventions** for attribute names — `http.request.method`, `db.system`, not custom names. Consistent
  naming enables cross-service analysis.
- **Add sampling-relevant attributes at span creation** — samplers can only see attributes present at creation time.
- **Keep cardinality low** — attribute values should be from a bounded set.
- **Check `IsRecording` before expensive attribute computation** — sampled-out spans discard all data.
- **Never store PII in attributes** — trace data flows to shared backends. Use opaque identifiers.
- **Respect attribute limits** — SDK enforces `AttributeCountLimit` (default 128). Excess attributes are silently
  dropped.

### Events

Timestamped annotations — structured log entries attached to a span.

- Use events when the timestamp matters; use attributes for metadata with no meaningful timestamp
- Add events for key domain occurrences ("page became interactive", "retry attempted")
- Use events for verbose data instead of additional spans

### Recording Exceptions

- **Always pair `RecordException` with `setStatus(ERROR)`** — RecordException alone does not change span status
- Never swallow application exceptions in instrumentation — catch instrumentation errors, log them, but always rethrow
  application exceptions

### Links

Associate a span with spans from other traces without parent-child hierarchy.

- **Batch processing:** link consumer span to each producer span
- **Async follow-up:** job span links back to triggering span
- Add links at span creation when possible — samplers can consider them

See `${CLAUDE_SKILL_DIR}/references/span-data.md` for events format, links format, and SDK limits table.

## Context Propagation

Context propagation correlates spans across service boundaries. Without it, each service produces isolated spans — no
distributed trace.

### Inject / Extract Pattern

1. **Sender** creates a span, makes it current, **injects** context into outgoing carrier (HTTP headers, message
   metadata)
2. **Receiver** **extracts** context from incoming carrier and uses it as parent for new spans

- Call inject AFTER creating the outgoing CLIENT or PRODUCER span
- Call extract BEFORE creating the server/consumer span
- Use W3C TraceContext as the default propagation format

### In-Process Propagation

- **Make spans active/current** — enables automatic log correlation, nested auto-instrumentation picking up correct
  parent, context propagation to child ops
- **Pass context explicitly in async code** — automatic propagation may break across thread boundaries, goroutines, or
  async callbacks
- **Never propagate request context to background work** — background tasks should start new traces and link back to the
  triggering span

### Baggage

Propagates arbitrary key-value pairs across service boundaries alongside trace context.

- Use for: tenant ID, request priority, feature flags, sampling hints
- **NEVER put PII, credentials, or API keys in baggage** — visible to all downstream services

See `${CLAUDE_SKILL_DIR}/references/context-propagation.md` for W3C header format, propagator selection table, baggage
details, and security considerations.

## Instrumentation

### Approaches

| Approach             | When                                      | Trade-off                                  |
| -------------------- | ----------------------------------------- | ------------------------------------------ |
| Automatic            | Supported libraries (HTTP, DB, messaging) | Zero code, no business attrs               |
| Manual               | Business logic, unsupported libraries     | Full control, more code                    |
| Hybrid (recommended) | Production                                | Auto for infra + manual for business logic |

### Getting a Tracer

- Name the tracer after your library, package, or module — not the application
- Include a version string matching your library version
- Libraries: accept `TracerProvider` via dependency injection or use the global one
- Applications: configure the SDK and set the global `TracerProvider`

### What to Instrument

**Good candidates:** public API methods with I/O or significant computation, request/message handlers, outbound calls
(HTTP, DB, RPC), background jobs.

**Poor candidates:** every function call (noise), thin wrapper libraries (already instrumented underneath), pure
computation with no I/O and sub-ms duration.

**Decision:** Is it a network call or significant I/O? -> Create span (CLIENT/SERVER). Is it a meaningful business
operation? -> Create span (INTERNAL). Would a span event on parent suffice? -> Add event. Otherwise -> don't instrument.

### Library Instrumentation Rules

- **Depend on OpenTelemetry API only** — never SDK. The API is a no-op without SDK, so zero overhead for users who don't
  use OTel.
- Follow semantic conventions for your domain
- Set the `schema_url` to record which semantic convention version you use
- Prefer events over spans for verbose internal details
- Support optional `TracerProvider` injection for testability

See `${CLAUDE_SKILL_DIR}/references/instrumentation.md` for server-side, client-side, and async producer/consumer
patterns, plus testing guidance.

## Sampling

Sampling controls which traces are recorded and exported — the primary mechanism for managing tracing costs.

### Head Sampling

Decision at trace creation time, before any spans complete.

| Sampler                    | Behavior                                   |
| -------------------------- | ------------------------------------------ |
| `AlwaysOn`                 | Record and sample everything               |
| `AlwaysOff`                | Drop everything                            |
| `TraceIdRatioBased(ratio)` | Sample based on trace ID hash              |
| `ParentBased(root)`        | Delegate based on parent sampling decision |

**ParentBased** is the most common production configuration — respects parent decisions, applies custom root sampling.
Default SDK sampler: `ParentBased(root=AlwaysOn)`.

### Tail Sampling

Decision after all spans in a trace complete. Requires collector infrastructure.

- All services export at 100%
- OTel Collector with tail sampling processor buffers spans by trace ID
- **All spans with the same trace ID MUST reach the same collector** — use trace-ID-aware load balancing

### Sampling Principles

- Start with no sampling — add only when cost or volume requires it
- Always sample errors — configure tail sampling to keep error traces
- Use `ParentBased` — children follow parent decisions for complete traces
- Provide attributes at span creation — samplers cannot see late-added attributes
- Filter health checks — high volume, low value; sample aggressively or filter entirely

See `${CLAUDE_SKILL_DIR}/references/sampling.md` for combined head+tail strategies, decision guide, and per-scenario
recommendations.

## Semantic Conventions

Use semantic conventions for span names and attributes. Consistent naming enables cross-service analysis without
learning custom attribute names.

- HTTP server spans: kind `SERVER`, name `{METHOD} {http.route}`, never full URI path
- HTTP client spans: kind `CLIENT`, name `{METHOD}`
- Database spans: kind `CLIENT`, name `{operation} {target}`, always sanitize queries
- Messaging producer: kind `PRODUCER`, name `{destination} publish`
- Messaging consumer: kind `CONSUMER`, name `{destination} process`

See `${CLAUDE_SKILL_DIR}/references/semantic-conventions.md` for required/recommended attributes per domain, status
mapping rules, and general conventions.

## SDK Components

### TracerProvider

- Initialize once, early in application startup
- Always call `shutdown()` on exit — flushes remaining spans
- Configure `Resource` with `service.name` — identifies your service in backends
- Use `BatchSpanProcessor` in production — `SimpleSpanProcessor` is for dev/testing only

See `${CLAUDE_SKILL_DIR}/references/sdk-components.md` for Resource attributes, BatchSpanProcessor tuning, exporter
types, environment variable configuration, and Collector deployment patterns.

## Application

When **writing** tracing code:

- Apply all conventions silently — don't narrate each rule being followed.
- Use semantic conventions for attribute names. Check the reference for your domain (HTTP, DB, messaging) before
  inventing names.
- If an existing codebase contradicts a convention, follow the codebase and flag the divergence once.

When **reviewing** tracing code:

- Cite the specific violation and show the fix inline.
- Focus on: span naming cardinality, missing context propagation, missing error recording, wrong SpanKind, missing
  `span.end()`.

## Integration

The coding skill governs workflow; this skill governs tracing implementation choices. Language-specific skills handle
SDK API differences.