otel-tracing · diff
git:20260305.a08fbb4 to git:20260313.02a8e7c
105 added, 119 removed. Audit A to A.
---
name: otel-tracing
description: >-
OpenTelemetry tracing discipline: correct spans, propagation, and semantic conventions
produce useful traces. Invoke whenever task involves any interaction with distributed
tracing — span creation, context propagation, instrumentation, sampling configuration,
or OpenTelemetry SDK setup.
---
# OTel Tracing
- Create spans for logical operations, propagate context across every boundary, use semantic
- conventions for attribute names. Every tracing decision — span granularity, attribute
- selection, sampling strategy — trades off cost against visibility.
+ Create spans for logical operations, propagate context across every boundary, use semantic conventions for attribute
+ names. Every tracing decision — span granularity, attribute selection, sampling strategy — trades off cost against
+ visibility.
## References
- | Topic | Reference | Contents |
- |-------|-----------|----------|
- | Spans | [`${CLAUDE_SKILL_DIR}/references/spans.md`] | Span anatomy, root spans, lifetime code patterns |
- | Span data | [`${CLAUDE_SKILL_DIR}/references/span-data.md`] | Events format, links format, SDK limits table |
- | Context propagation | [`${CLAUDE_SKILL_DIR}/references/context-propagation.md`] | W3C header format, propagator selection, baggage details, security |
- | Instrumentation | [`${CLAUDE_SKILL_DIR}/references/instrumentation.md`] | Server/client/async code patterns, library rules, testing guidance |
- | Sampling | [`${CLAUDE_SKILL_DIR}/references/sampling.md`] | Head/tail/combined strategies, decision guide, sampler types |
+ | Topic | Reference | Contents |
+ | -------------------- | ---------------------------------------------------------- | ---------------------------------------------------------------------- |
+ | Spans | [`${CLAUDE_SKILL_DIR}/references/spans.md`] | Span anatomy, root spans, lifetime code patterns |
+ | Span data | [`${CLAUDE_SKILL_DIR}/references/span-data.md`] | Events format, links format, SDK limits table |
+ | Context propagation | [`${CLAUDE_SKILL_DIR}/references/context-propagation.md`] | W3C header format, propagator selection, baggage details, security |
+ | Instrumentation | [`${CLAUDE_SKILL_DIR}/references/instrumentation.md`] | Server/client/async code patterns, library rules, testing guidance |
+ | Sampling | [`${CLAUDE_SKILL_DIR}/references/sampling.md`] | Head/tail/combined strategies, decision guide, sampler types |
| Semantic conventions | [`${CLAUDE_SKILL_DIR}/references/semantic-conventions.md`] | HTTP/DB/messaging attribute lists, status mapping, general conventions |
- | SDK components | [`${CLAUDE_SKILL_DIR}/references/sdk-components.md`] | Resource config, env vars, Collector deployment, exporter types |
+ | SDK components | [`${CLAUDE_SKILL_DIR}/references/sdk-components.md`] | Resource config, env vars, Collector deployment, exporter types |
## Spans
- A span represents a single unit of work in a trace — an HTTP request handler, a database
- query, a message publish. Not every function call.
+ A span represents a single unit of work in a trace — an HTTP request handler, a database query, a message publish. Not
+ every function call.
### Span Naming
- Name spans for the **class** of operation, not the instance. Low-cardinality names enable
- aggregation; high-cardinality names destroy it.
+ Name spans for the **class** of operation, not the instance. Low-cardinality names enable aggregation; high-cardinality
+ names destroy it.
- Use the most general string that identifies a statistically interesting class
- Never embed high-cardinality values (IDs, emails, paths with IDs)
- HTTP span names: `{METHOD} {route}` — server: `GET /users/:id`; client: `GET`
- Database span names: `{operation} {table}` — `SELECT users`, `INSERT orders`
- | Name | Verdict |
- |------|---------|
- | `get` | Too general |
- | `get_account/42` | Too specific — ID in name |
- | `get_account` | Good |
- | `GET /users/{userId}` | Good — route template |
- | `POST /api/v2/orders/abc123` | Bad — order ID in name |
+ | Name | Verdict |
+ | ---------------------------- | ------------------------- |
+ | `get` | Too general |
+ | `get_account/42` | Too specific — ID in name |
+ | `get_account` | Good |
+ | `GET /users/{userId}` | Good — route template |
+ | `POST /api/v2/orders/abc123` | Bad — order ID in name |
### SpanKind
SpanKind tells backends how to assemble the trace tree. Set it correctly.
- | Kind | Direction | Use For |
- |------|-----------|---------|
- | `SERVER` | Incoming | HTTP handler, gRPC server method |
- | `CLIENT` | Outgoing | HTTP client call, DB query, gRPC call |
- | `PRODUCER` | Outgoing | Enqueue message, schedule job |
- | `CONSUMER` | Incoming | Dequeue message, process job |
- | `INTERNAL` | Neither | In-process business logic, computation |
+ | Kind | Direction | Use For |
+ | ---------- | --------- | -------------------------------------- |
+ | `SERVER` | Incoming | HTTP handler, gRPC server method |
+ | `CLIENT` | Outgoing | HTTP client call, DB query, gRPC call |
+ | `PRODUCER` | Outgoing | Enqueue message, schedule job |
+ | `CONSUMER` | Incoming | Dequeue message, process job |
+ | `INTERNAL` | Neither | In-process business logic, computation |
- A single span MUST NOT serve more than one purpose
- Create a new span before injecting context for outgoing calls
- `CLIENT` -> `SERVER` for synchronous calls; `PRODUCER` -> `CONSUMER` for async
- Using `INTERNAL` for HTTP handlers or DB calls is wrong — use `SERVER` or `CLIENT`
### Span Status
- | Status | Meaning | When to Set |
- |--------|---------|-------------|
- | `Unset` | No error | Default — do not change on success |
- | `Error` | Operation failed | When an error occurs; include description |
- | `Ok` | Explicitly successful | Only to override a previous `Error`; rarely needed |
+ | Status | Meaning | When to Set |
+ | ------- | --------------------- | -------------------------------------------------- |
+ | `Unset` | No error | Default — do not change on success |
+ | `Error` | Operation failed | When an error occurs; include description |
+ | `Ok` | Explicitly successful | Only to override a previous `Error`; rarely needed |
- Leave status `Unset` on success — it already means "no error"
- `Ok` is a "final call" — once set, subsequent `Error` attempts are ignored
- Instrumentation libraries SHOULD NOT set `Ok` — leave to application code
### Span Lifetime
- - **Every created span MUST be ended** — leaked spans cause memory issues and
- incomplete traces
+ - **Every created span MUST be ended** — leaked spans cause memory issues and incomplete traces
- After `end()`, the span becomes non-recording; further mutations are ignored
- - Use language-idiomatic patterns: `defer` (Go), try/finally (Java/JS),
- context manager (Python)
+ - Use language-idiomatic patterns: `defer` (Go), try/finally (Java/JS), context manager (Python)
## Span Data: Attributes, Events, Links
### Attributes
- Key-value pairs annotating a span. Value types: string, boolean, integer, float, or
- arrays of these.
+ Key-value pairs annotating a span. Value types: string, boolean, integer, float, or arrays of these.
- - **Use semantic conventions** for attribute names — `http.request.method`, `db.system`,
- not custom names. Consistent naming enables cross-service analysis.
- - **Add sampling-relevant attributes at span creation** — samplers can only see attributes
- present at creation time.
+ - **Use semantic conventions** for attribute names — `http.request.method`, `db.system`, not custom names. Consistent
+ naming enables cross-service analysis.
+ - **Add sampling-relevant attributes at span creation** — samplers can only see attributes present at creation time.
- **Keep cardinality low** — attribute values should be from a bounded set.
- - **Check `IsRecording` before expensive attribute computation** — sampled-out spans
- discard all data.
- - **Never store PII in attributes** — trace data flows to shared backends. Use opaque
- identifiers.
- - **Respect attribute limits** — SDK enforces `AttributeCountLimit` (default 128). Excess
- attributes are silently dropped.
+ - **Check `IsRecording` before expensive attribute computation** — sampled-out spans discard all data.
+ - **Never store PII in attributes** — trace data flows to shared backends. Use opaque identifiers.
+ - **Respect attribute limits** — SDK enforces `AttributeCountLimit` (default 128). Excess attributes are silently
+ dropped.
### Events
Timestamped annotations — structured log entries attached to a span.
- - Use events when the timestamp matters; use attributes for metadata with no meaningful
- timestamp
+ - Use events when the timestamp matters; use attributes for metadata with no meaningful timestamp
- Add events for key domain occurrences ("page became interactive", "retry attempted")
- Use events for verbose data instead of additional spans
### Recording Exceptions
- - **Always pair `RecordException` with `setStatus(ERROR)`** — RecordException alone does
- not change span status
- - Never swallow application exceptions in instrumentation — catch instrumentation errors,
- log them, but always rethrow application exceptions
+ - **Always pair `RecordException` with `setStatus(ERROR)`** — RecordException alone does not change span status
+ - Never swallow application exceptions in instrumentation — catch instrumentation errors, log them, but always rethrow
+ application exceptions
### Links
Associate a span with spans from other traces without parent-child hierarchy.
- **Batch processing:** link consumer span to each producer span
- **Async follow-up:** job span links back to triggering span
- Add links at span creation when possible — samplers can consider them
- See `${CLAUDE_SKILL_DIR}/references/span-data.md` for events format, links format, and SDK
- limits table.
+ See `${CLAUDE_SKILL_DIR}/references/span-data.md` for events format, links format, and SDK limits table.
## Context Propagation
- Context propagation correlates spans across service boundaries. Without it, each service
- produces isolated spans — no distributed trace.
+ Context propagation correlates spans across service boundaries. Without it, each service produces isolated spans — no
+ distributed trace.
### Inject / Extract Pattern
- 1. **Sender** creates a span, makes it current, **injects** context into outgoing carrier
- (HTTP headers, message metadata)
- 2. **Receiver** **extracts** context from incoming carrier and uses it as parent for new
- spans
+ 1. **Sender** creates a span, makes it current, **injects** context into outgoing carrier (HTTP headers, message
+ metadata)
+ 2. **Receiver** **extracts** context from incoming carrier and uses it as parent for new spans
- Call inject AFTER creating the outgoing CLIENT or PRODUCER span
- Call extract BEFORE creating the server/consumer span
- Use W3C TraceContext as the default propagation format
### In-Process Propagation
- - **Make spans active/current** — enables automatic log correlation, nested
- auto-instrumentation picking up correct parent, context propagation to child ops
- - **Pass context explicitly in async code** — automatic propagation may break across
- thread boundaries, goroutines, or async callbacks
- - **Never propagate request context to background work** — background tasks should start
- new traces and link back to the triggering span
+ - **Make spans active/current** — enables automatic log correlation, nested auto-instrumentation picking up correct
+ parent, context propagation to child ops
+ - **Pass context explicitly in async code** — automatic propagation may break across thread boundaries, goroutines, or
+ async callbacks
+ - **Never propagate request context to background work** — background tasks should start new traces and link back to the
+ triggering span
### Baggage
Propagates arbitrary key-value pairs across service boundaries alongside trace context.
- Use for: tenant ID, request priority, feature flags, sampling hints
- - **NEVER put PII, credentials, or API keys in baggage** — visible to all downstream
- services
+ - **NEVER put PII, credentials, or API keys in baggage** — visible to all downstream services
- See `${CLAUDE_SKILL_DIR}/references/context-propagation.md` for W3C header format,
- propagator selection table, baggage details, and security considerations.
+ See `${CLAUDE_SKILL_DIR}/references/context-propagation.md` for W3C header format, propagator selection table, baggage
+ details, and security considerations.
## Instrumentation
### Approaches
- | Approach | When | Trade-off |
- |----------|------|-----------|
- | Automatic | Supported libraries (HTTP, DB, messaging) | Zero code, no business attrs |
- | Manual | Business logic, unsupported libraries | Full control, more code |
- | Hybrid (recommended) | Production | Auto for infra + manual for business logic |
+ | Approach | When | Trade-off |
+ | -------------------- | ----------------------------------------- | ------------------------------------------ |
+ | Automatic | Supported libraries (HTTP, DB, messaging) | Zero code, no business attrs |
+ | Manual | Business logic, unsupported libraries | Full control, more code |
+ | Hybrid (recommended) | Production | Auto for infra + manual for business logic |
### Getting a Tracer
- Name the tracer after your library, package, or module — not the application
- Include a version string matching your library version
- Libraries: accept `TracerProvider` via dependency injection or use the global one
- Applications: configure the SDK and set the global `TracerProvider`
### What to Instrument
- **Good candidates:** public API methods with I/O or significant computation, request/message
- handlers, outbound calls (HTTP, DB, RPC), background jobs.
+ **Good candidates:** public API methods with I/O or significant computation, request/message handlers, outbound calls
+ (HTTP, DB, RPC), background jobs.
- **Poor candidates:** every function call (noise), thin wrapper libraries (already
- instrumented underneath), pure computation with no I/O and sub-ms duration.
+ **Poor candidates:** every function call (noise), thin wrapper libraries (already instrumented underneath), pure
+ computation with no I/O and sub-ms duration.
- **Decision:** Is it a network call or significant I/O? -> Create span (CLIENT/SERVER). Is it
- a meaningful business operation? -> Create span (INTERNAL). Would a span event on parent
- suffice? -> Add event. Otherwise -> don't instrument.
+ **Decision:** Is it a network call or significant I/O? -> Create span (CLIENT/SERVER). Is it a meaningful business
+ operation? -> Create span (INTERNAL). Would a span event on parent suffice? -> Add event. Otherwise -> don't instrument.
### Library Instrumentation Rules
- - **Depend on OpenTelemetry API only** — never SDK. The API is a no-op without SDK, so
- zero overhead for users who don't use OTel.
+ - **Depend on OpenTelemetry API only** — never SDK. The API is a no-op without SDK, so zero overhead for users who don't
+ use OTel.
- Follow semantic conventions for your domain
- Set the `schema_url` to record which semantic convention version you use
- Prefer events over spans for verbose internal details
- Support optional `TracerProvider` injection for testability
- See `${CLAUDE_SKILL_DIR}/references/instrumentation.md` for server-side, client-side,
- and async producer/consumer patterns, plus testing guidance.
+ See `${CLAUDE_SKILL_DIR}/references/instrumentation.md` for server-side, client-side, and async producer/consumer
+ patterns, plus testing guidance.
## Sampling
- Sampling controls which traces are recorded and exported — the primary mechanism for
- managing tracing costs.
+ Sampling controls which traces are recorded and exported — the primary mechanism for managing tracing costs.
### Head Sampling
Decision at trace creation time, before any spans complete.
- | Sampler | Behavior |
- |---------|----------|
- | `AlwaysOn` | Record and sample everything |
- | `AlwaysOff` | Drop everything |
- | `TraceIdRatioBased(ratio)` | Sample based on trace ID hash |
- | `ParentBased(root)` | Delegate based on parent sampling decision |
+ | Sampler | Behavior |
+ | -------------------------- | ------------------------------------------ |
+ | `AlwaysOn` | Record and sample everything |
+ | `AlwaysOff` | Drop everything |
+ | `TraceIdRatioBased(ratio)` | Sample based on trace ID hash |
+ | `ParentBased(root)` | Delegate based on parent sampling decision |
- **ParentBased** is the most common production configuration — respects parent decisions,
- applies custom root sampling. Default SDK sampler: `ParentBased(root=AlwaysOn)`.
+ **ParentBased** is the most common production configuration — respects parent decisions, applies custom root sampling.
+ Default SDK sampler: `ParentBased(root=AlwaysOn)`.
### Tail Sampling
Decision after all spans in a trace complete. Requires collector infrastructure.
- All services export at 100%
- OTel Collector with tail sampling processor buffers spans by trace ID
- - **All spans with the same trace ID MUST reach the same collector** — use trace-ID-aware
- load balancing
+ - **All spans with the same trace ID MUST reach the same collector** — use trace-ID-aware load balancing
### Sampling Principles
- Start with no sampling — add only when cost or volume requires it
- Always sample errors — configure tail sampling to keep error traces
- Use `ParentBased` — children follow parent decisions for complete traces
- Provide attributes at span creation — samplers cannot see late-added attributes
- Filter health checks — high volume, low value; sample aggressively or filter entirely
- See `${CLAUDE_SKILL_DIR}/references/sampling.md` for combined head+tail strategies,
- decision guide, and per-scenario recommendations.
+ See `${CLAUDE_SKILL_DIR}/references/sampling.md` for combined head+tail strategies, decision guide, and per-scenario
+ recommendations.
## Semantic Conventions
- Use semantic conventions for span names and attributes. Consistent naming enables
- cross-service analysis without learning custom attribute names.
+ Use semantic conventions for span names and attributes. Consistent naming enables cross-service analysis without
+ learning custom attribute names.
- HTTP server spans: kind `SERVER`, name `{METHOD} {http.route}`, never full URI path
- HTTP client spans: kind `CLIENT`, name `{METHOD}`
- Database spans: kind `CLIENT`, name `{operation} {target}`, always sanitize queries
- Messaging producer: kind `PRODUCER`, name `{destination} publish`
- Messaging consumer: kind `CONSUMER`, name `{destination} process`
- See `${CLAUDE_SKILL_DIR}/references/semantic-conventions.md` for required/recommended
- attributes per domain, status mapping rules, and general conventions.
+ See `${CLAUDE_SKILL_DIR}/references/semantic-conventions.md` for required/recommended attributes per domain, status
+ mapping rules, and general conventions.
## SDK Components
### TracerProvider
- Initialize once, early in application startup
- Always call `shutdown()` on exit — flushes remaining spans
- Configure `Resource` with `service.name` — identifies your service in backends
- Use `BatchSpanProcessor` in production — `SimpleSpanProcessor` is for dev/testing only
- See `${CLAUDE_SKILL_DIR}/references/sdk-components.md` for Resource attributes,
- BatchSpanProcessor tuning, exporter types, environment variable configuration, and
- Collector deployment patterns.
+ See `${CLAUDE_SKILL_DIR}/references/sdk-components.md` for Resource attributes, BatchSpanProcessor tuning, exporter
+ types, environment variable configuration, and Collector deployment patterns.
## Application
When **writing** tracing code:
+
- Apply all conventions silently — don't narrate each rule being followed.
- - Use semantic conventions for attribute names. Check the reference for your domain
- (HTTP, DB, messaging) before inventing names.
- - If an existing codebase contradicts a convention, follow the codebase and flag the
- divergence once.
+ - Use semantic conventions for attribute names. Check the reference for your domain (HTTP, DB, messaging) before
+ inventing names.
+ - If an existing codebase contradicts a convention, follow the codebase and flag the divergence once.
When **reviewing** tracing code:
+
- Cite the specific violation and show the fix inline.
- - Focus on: span naming cardinality, missing context propagation, missing error
- recording, wrong SpanKind, missing `span.end()`.
+ - Focus on: span naming cardinality, missing context propagation, missing error recording, wrong SpanKind, missing
+ `span.end()`.
## Integration
- The coding skill governs workflow; this skill governs tracing implementation choices.
- Language-specific skills handle SDK API differences.
+ The coding skill governs workflow; this skill governs tracing implementation choices. Language-specific skills handle
+ SDK API differences.