otel-tracing · diff

git:20260220.4bda00f to git:20260305.a08fbb4

20 added, 18 removed. Audit A to A.

---

name: otel-tracing

description: >-

OpenTelemetry tracing discipline: correct spans, propagation, and semantic conventions

produce useful traces. Invoke whenever task involves any interaction with distributed

tracing — span creation, context propagation, instrumentation, sampling configuration,

or OpenTelemetry SDK setup.

---

# OTel Tracing

Create spans for logical operations, propagate context across every boundary, use semantic

conventions for attribute names. Every tracing decision — span granularity, attribute

selection, sampling strategy — trades off cost against visibility.

## References

| Topic | Reference | Contents |

|-------|-----------|----------|

- | Spans | `references/spans.md` | Span anatomy, root spans, lifetime code patterns |

- | Span data | `references/span-data.md` | Events format, links format, SDK limits table |

- | Context propagation | `references/context-propagation.md` | W3C header format, propagator selection, baggage details, security |

- | Instrumentation | `references/instrumentation.md` | Server/client/async code patterns, library rules, testing guidance |

- | Sampling | `references/sampling.md` | Head/tail/combined strategies, decision guide, sampler types |

- | Semantic conventions | `references/semantic-conventions.md` | HTTP/DB/messaging attribute lists, status mapping, general conventions |

- | SDK components | `references/sdk-components.md` | Resource config, env vars, Collector deployment, exporter types |

+ | Spans | [`${CLAUDE_SKILL_DIR}/references/spans.md`] | Span anatomy, root spans, lifetime code patterns |

+ | Span data | [`${CLAUDE_SKILL_DIR}/references/span-data.md`] | Events format, links format, SDK limits table |

+ | Context propagation | [`${CLAUDE_SKILL_DIR}/references/context-propagation.md`] | W3C header format, propagator selection, baggage details, security |

+ | Instrumentation | [`${CLAUDE_SKILL_DIR}/references/instrumentation.md`] | Server/client/async code patterns, library rules, testing guidance |

+ | Sampling | [`${CLAUDE_SKILL_DIR}/references/sampling.md`] | Head/tail/combined strategies, decision guide, sampler types |

+ | Semantic conventions | [`${CLAUDE_SKILL_DIR}/references/semantic-conventions.md`] | HTTP/DB/messaging attribute lists, status mapping, general conventions |

+ | SDK components | [`${CLAUDE_SKILL_DIR}/references/sdk-components.md`] | Resource config, env vars, Collector deployment, exporter types |

## Spans

A span represents a single unit of work in a trace — an HTTP request handler, a database

query, a message publish. Not every function call.

### Span Naming

Name spans for the **class** of operation, not the instance. Low-cardinality names enable

aggregation; high-cardinality names destroy it.

- Use the most general string that identifies a statistically interesting class

- Never embed high-cardinality values (IDs, emails, paths with IDs)

- HTTP span names: `{METHOD} {route}` — server: `GET /users/:id`; client: `GET`

- Database span names: `{operation} {table}` — `SELECT users`, `INSERT orders`

| Name | Verdict |

|------|---------|

| `get` | Too general |

| `get_account/42` | Too specific — ID in name |

| `get_account` | Good |

| `GET /users/{userId}` | Good — route template |

| `POST /api/v2/orders/abc123` | Bad — order ID in name |

### SpanKind

SpanKind tells backends how to assemble the trace tree. Set it correctly.

| Kind | Direction | Use For |

|------|-----------|---------|

| `SERVER` | Incoming | HTTP handler, gRPC server method |

| `CLIENT` | Outgoing | HTTP client call, DB query, gRPC call |

| `PRODUCER` | Outgoing | Enqueue message, schedule job |

| `CONSUMER` | Incoming | Dequeue message, process job |

| `INTERNAL` | Neither | In-process business logic, computation |

- A single span MUST NOT serve more than one purpose

- Create a new span before injecting context for outgoing calls

- `CLIENT` -> `SERVER` for synchronous calls; `PRODUCER` -> `CONSUMER` for async

- Using `INTERNAL` for HTTP handlers or DB calls is wrong — use `SERVER` or `CLIENT`

### Span Status

| Status | Meaning | When to Set |

|--------|---------|-------------|

| `Unset` | No error | Default — do not change on success |

| `Error` | Operation failed | When an error occurs; include description |

| `Ok` | Explicitly successful | Only to override a previous `Error`; rarely needed |

- Leave status `Unset` on success — it already means "no error"

- `Ok` is a "final call" — once set, subsequent `Error` attempts are ignored

- Instrumentation libraries SHOULD NOT set `Ok` — leave to application code

### Span Lifetime

- **Every created span MUST be ended** — leaked spans cause memory issues and

incomplete traces

- After `end()`, the span becomes non-recording; further mutations are ignored

- Use language-idiomatic patterns: `defer` (Go), try/finally (Java/JS),

context manager (Python)

## Span Data: Attributes, Events, Links

### Attributes

Key-value pairs annotating a span. Value types: string, boolean, integer, float, or

arrays of these.

- **Use semantic conventions** for attribute names — `http.request.method`, `db.system`,

not custom names. Consistent naming enables cross-service analysis.

- **Add sampling-relevant attributes at span creation** — samplers can only see attributes

present at creation time.

- **Keep cardinality low** — attribute values should be from a bounded set.

- **Check `IsRecording` before expensive attribute computation** — sampled-out spans

discard all data.

- **Never store PII in attributes** — trace data flows to shared backends. Use opaque

identifiers.

- **Respect attribute limits** — SDK enforces `AttributeCountLimit` (default 128). Excess

attributes are silently dropped.

### Events

Timestamped annotations — structured log entries attached to a span.

- Use events when the timestamp matters; use attributes for metadata with no meaningful

timestamp

- Add events for key domain occurrences ("page became interactive", "retry attempted")

- Use events for verbose data instead of additional spans

### Recording Exceptions

- **Always pair `RecordException` with `setStatus(ERROR)`** — RecordException alone does

not change span status

- Never swallow application exceptions in instrumentation — catch instrumentation errors,

log them, but always rethrow application exceptions

### Links

Associate a span with spans from other traces without parent-child hierarchy.

- **Batch processing:** link consumer span to each producer span

- **Async follow-up:** job span links back to triggering span

- Add links at span creation when possible — samplers can consider them

- See `references/span-data.md` for events format, links format, and SDK limits table.

+ See `${CLAUDE_SKILL_DIR}/references/span-data.md` for events format, links format, and SDK

+ limits table.

## Context Propagation

Context propagation correlates spans across service boundaries. Without it, each service

produces isolated spans — no distributed trace.

### Inject / Extract Pattern

1. **Sender** creates a span, makes it current, **injects** context into outgoing carrier

(HTTP headers, message metadata)

2. **Receiver** **extracts** context from incoming carrier and uses it as parent for new

spans

- Call inject AFTER creating the outgoing CLIENT or PRODUCER span

- Call extract BEFORE creating the server/consumer span

- Use W3C TraceContext as the default propagation format

### In-Process Propagation

- **Make spans active/current** — enables automatic log correlation, nested

auto-instrumentation picking up correct parent, context propagation to child ops

- **Pass context explicitly in async code** — automatic propagation may break across

thread boundaries, goroutines, or async callbacks

- **Never propagate request context to background work** — background tasks should start

new traces and link back to the triggering span

### Baggage

Propagates arbitrary key-value pairs across service boundaries alongside trace context.

- Use for: tenant ID, request priority, feature flags, sampling hints

- **NEVER put PII, credentials, or API keys in baggage** — visible to all downstream

services

- See `references/context-propagation.md` for W3C header format, propagator selection

- table, baggage details, and security considerations.

+ See `${CLAUDE_SKILL_DIR}/references/context-propagation.md` for W3C header format,

+ propagator selection table, baggage details, and security considerations.

## Instrumentation

### Approaches

| Approach | When | Trade-off |

|----------|------|-----------|

| Automatic | Supported libraries (HTTP, DB, messaging) | Zero code, no business attrs |

| Manual | Business logic, unsupported libraries | Full control, more code |

| Hybrid (recommended) | Production | Auto for infra + manual for business logic |

### Getting a Tracer

- Name the tracer after your library, package, or module — not the application

- Include a version string matching your library version

- Libraries: accept `TracerProvider` via dependency injection or use the global one

- Applications: configure the SDK and set the global `TracerProvider`

### What to Instrument

**Good candidates:** public API methods with I/O or significant computation, request/message

handlers, outbound calls (HTTP, DB, RPC), background jobs.

**Poor candidates:** every function call (noise), thin wrapper libraries (already

instrumented underneath), pure computation with no I/O and sub-ms duration.

**Decision:** Is it a network call or significant I/O? -> Create span (CLIENT/SERVER). Is it

a meaningful business operation? -> Create span (INTERNAL). Would a span event on parent

suffice? -> Add event. Otherwise -> don't instrument.

### Library Instrumentation Rules

- **Depend on OpenTelemetry API only** — never SDK. The API is a no-op without SDK, so

zero overhead for users who don't use OTel.

- Follow semantic conventions for your domain

- Set the `schema_url` to record which semantic convention version you use

- Prefer events over spans for verbose internal details

- Support optional `TracerProvider` injection for testability

- See `references/instrumentation.md` for server-side, client-side, and async

- producer/consumer patterns, plus testing guidance.

+ See `${CLAUDE_SKILL_DIR}/references/instrumentation.md` for server-side, client-side,

+ and async producer/consumer patterns, plus testing guidance.

## Sampling

Sampling controls which traces are recorded and exported — the primary mechanism for

managing tracing costs.

### Head Sampling

Decision at trace creation time, before any spans complete.

| Sampler | Behavior |

|---------|----------|

| `AlwaysOn` | Record and sample everything |

| `AlwaysOff` | Drop everything |

| `TraceIdRatioBased(ratio)` | Sample based on trace ID hash |

| `ParentBased(root)` | Delegate based on parent sampling decision |

**ParentBased** is the most common production configuration — respects parent decisions,

applies custom root sampling. Default SDK sampler: `ParentBased(root=AlwaysOn)`.

### Tail Sampling

Decision after all spans in a trace complete. Requires collector infrastructure.

- All services export at 100%

- OTel Collector with tail sampling processor buffers spans by trace ID

- **All spans with the same trace ID MUST reach the same collector** — use trace-ID-aware

load balancing

### Sampling Principles

- Start with no sampling — add only when cost or volume requires it

- Always sample errors — configure tail sampling to keep error traces

- Use `ParentBased` — children follow parent decisions for complete traces

- Provide attributes at span creation — samplers cannot see late-added attributes

- Filter health checks — high volume, low value; sample aggressively or filter entirely

- See `references/sampling.md` for combined head+tail strategies, decision guide, and

- per-scenario recommendations.

+ See `${CLAUDE_SKILL_DIR}/references/sampling.md` for combined head+tail strategies,

+ decision guide, and per-scenario recommendations.

## Semantic Conventions

Use semantic conventions for span names and attributes. Consistent naming enables

cross-service analysis without learning custom attribute names.

- HTTP server spans: kind `SERVER`, name `{METHOD} {http.route}`, never full URI path

- HTTP client spans: kind `CLIENT`, name `{METHOD}`

- Database spans: kind `CLIENT`, name `{operation} {target}`, always sanitize queries

- Messaging producer: kind `PRODUCER`, name `{destination} publish`

- Messaging consumer: kind `CONSUMER`, name `{destination} process`

- See `references/semantic-conventions.md` for required/recommended attributes per domain,

- status mapping rules, and general conventions.

+ See `${CLAUDE_SKILL_DIR}/references/semantic-conventions.md` for required/recommended

+ attributes per domain, status mapping rules, and general conventions.

## SDK Components

### TracerProvider

- Initialize once, early in application startup

- Always call `shutdown()` on exit — flushes remaining spans

- Configure `Resource` with `service.name` — identifies your service in backends

- Use `BatchSpanProcessor` in production — `SimpleSpanProcessor` is for dev/testing only

- See `references/sdk-components.md` for Resource attributes, BatchSpanProcessor tuning,

- exporter types, environment variable configuration, and Collector deployment patterns.

+ See `${CLAUDE_SKILL_DIR}/references/sdk-components.md` for Resource attributes,

+ BatchSpanProcessor tuning, exporter types, environment variable configuration, and

+ Collector deployment patterns.

## Application

When **writing** tracing code:

- Apply all conventions silently — don't narrate each rule being followed.

- Use semantic conventions for attribute names. Check the reference for your domain

(HTTP, DB, messaging) before inventing names.

- If an existing codebase contradicts a convention, follow the codebase and flag the

divergence once.

When **reviewing** tracing code:

- Cite the specific violation and show the fix inline.

- Focus on: span naming cardinality, missing context propagation, missing error

recording, wrong SpanKind, missing `span.end()`.

## Integration

The coding skill governs workflow; this skill governs tracing implementation choices.

Language-specific skills handle SDK API differences.