observability git:20260305.a08fbb4A

Immutable. This exact content never changes and is served at /api/v1/blob/914d572be113b4da.

---
name: observability
description: >-
  Observability discipline: structured logging, metrics instrumentation, distributed
  tracing, and signal correlation. Invoke whenever task involves any interaction with
  observability concerns — adding logging, designing metrics, instrumenting traces,
  correlating signals, reviewing instrumentation, or understanding when to use which
  pillar.
---

# Observability

**If you cannot ask arbitrary questions about your system's behavior from the outside,
your system is not observable — it is merely monitored.**

Observability is the ability to understand internal system state from its external outputs.
Three complementary signals make this possible: **logs** (discrete events), **metrics**
(aggregate measurements), and **traces** (request-scoped causal chains). Each pillar
answers different questions. Using the wrong pillar for a question wastes resources and
hides the answer.

This skill covers high-level observability discipline — purposes, interconnection, and
good practices for all three pillars. It is technology-agnostic: specific tools
(Prometheus, StatsD, OpenTelemetry) have their own dedicated skills.

---

## The Three Pillars

<pillars>

### Logs — What Happened

Logs are timestamped, discrete event records. They capture **what happened** at a specific
moment: an error thrown, a user action, a configuration loaded, a connection refused.

**Use logs when you need:**
- Rich diagnostic context for a specific event
- Debugging information with full error details and stack traces
- Audit trails of who did what and when
- Record of discrete state transitions

**Logs are poor at:**
- Showing aggregate system health (use metrics)
- Tracing request flow across services (use traces)
- High-frequency numeric trends (too expensive at volume)

### Metrics — How Is It Doing

Metrics are numeric measurements aggregated over time. They capture **how the system is
performing** as quantitative time series: request rates, error percentages, latencies,
queue depths, resource utilization.

**Use metrics when you need:**
- Real-time health signals and alerting
- Trend analysis over hours, days, weeks
- Capacity planning and saturation monitoring
- Pre-aggregated data that scales cheaply regardless of traffic

**Metrics are poor at:**
- Explaining *why* something is broken (use logs)
- Showing the path of a single request (use traces)
- Storing per-event detail (cardinality explosion)

### Traces — How Did It Flow

Traces record the causal chain of operations that make up a single request as it
propagates through distributed components. A trace is a tree of **spans**, where each
span represents one unit of work (an HTTP call, a database query, a queue publish).

**Use traces when you need:**
- End-to-end latency breakdown across services
- Dependency mapping and bottleneck identification
- Understanding the path a failing request took
- Correlating work across process and network boundaries

**Traces are poor at:**
- Aggregate health monitoring (use metrics)
- Detailed per-event diagnostics on a single node (use logs)
- Cheap, long-term trend storage (traces are expensive at 100% sampling)

</pillars>

### Choosing the Right Signal

| Question | Signal |
|----------|--------|
| "Is the system healthy right now?" | Metrics |
| "Why did this specific request fail?" | Traces + Logs |
| "What happened at 03:14 on node-7?" | Logs |
| "Where is the bottleneck in checkout flow?" | Traces |
| "Are error rates increasing over the last hour?" | Metrics |
| "What was the full stack trace of that exception?" | Logs |
| "Which downstream service is slow?" | Traces |
| "How much headroom does the database have?" | Metrics |

---

## Structured Logging

<structured-logging>

### Always Structured

Emit logs as structured records (JSON or equivalent key-value format) with a consistent
schema. Unstructured string logs are acceptable only in local development. Structured
logs are machine-parseable, indexable, and filterable at scale.

### Log Levels

Use levels consistently. Every team member must agree on what each level means.

| Level | Purpose | Alerting |
|-------|---------|----------|
| **FATAL/CRITICAL** | Process cannot continue; about to crash | Page immediately |
| **ERROR** | Operation failed; requires investigation | Alert / ticket |
| **WARN** | Unexpected condition; system compensated | Monitor trend |
| **INFO** | Significant business or lifecycle event | Dashboard |
| **DEBUG** | Diagnostic detail for developers | Never in production by default |
| **TRACE** | Extremely verbose step-by-step flow | Never in production |

Rules:
- Production defaults to INFO or above. DEBUG/TRACE are off unless explicitly enabled
  for a bounded investigation window.
- WARN is not a dumping ground. If it never leads to action, it is noise — downgrade
  to DEBUG or remove it.
- ERROR means something is broken. Expected conditions (404 for missing resources,
  validation failures from bad input) are not errors — they are INFO with a status field.
- Make log level configurable at runtime without restarts.

### Structured Fields

Every log record should include these baseline fields:

| Field | Purpose |
|-------|---------|
| `timestamp` | ISO 8601, UTC |
| `level` | Severity (ERROR, WARN, INFO, ...) |
| `message` | Human-readable summary of the event |
| `service` | Service name emitting the log |
| `version` | Service version / build / commit SHA |
| `trace_id` | Distributed trace ID (if in request context) |
| `span_id` | Current span ID (if in request context) |

Add contextual fields relevant to the event:

| Field | When |
|-------|------|
| `user_id` | User-initiated actions |
| `request_id` | Per-request correlation |
| `duration_ms` | Timed operations |
| `error.type` | Error class/name |
| `error.message` | Error description |
| `error.stack` | Stack trace (ERROR level only) |
| `http.method`, `http.path`, `http.status` | HTTP request/response |
| `db.operation`, `db.duration_ms` | Database calls |

### Sensitive Data

Never log:
- Passwords, tokens, API keys, secrets
- Full credit card numbers, SSNs, or equivalent PII
- Session tokens or authentication cookies
- Request/response bodies containing user-submitted personal data

When user identifiers are needed, log opaque IDs (user_id), not email addresses or
names. If regulations (GDPR, HIPAA) apply, verify that logged fields comply. When in
doubt, omit the field.

### Logging at Boundaries

**At application startup:**
- INFO: service name, version, loaded configuration (without secrets), listen address
- WARN: degraded mode (e.g., fallback to local cache because Redis is unreachable)
- ERROR/FATAL: unrecoverable startup failures

**Per incoming request:**
- INFO: method, path (scrubbed of PII), status code, duration, request dimensions
  (tenant, region)
- WARN/ERROR: only for unexpected exceptions; catch at the top-level handler

**Per outgoing dependency call:**
- INFO or DEBUG: target service, operation, status, duration
- ERROR: failures in dependent services (Redis, database, queue, etc.)

### Log Once, at the Right Level

Log a raised exception **once**. Do not catch-log-rethrow at every layer. Let exceptions
propagate to the top-level handler, which logs with full context. If you must log and
rethrow, do it only when adding context that would otherwise be lost.

</structured-logging>

---

## Metrics

<metrics>

### Metric Types

| Type | Behavior | Use For |
|------|----------|---------|
| **Counter** | Monotonically increasing; resets on restart | Totals: requests, errors, bytes sent |
| **Gauge** | Arbitrary value; goes up and down | Snapshots: queue depth, memory usage, connections |
| **Histogram** | Client-side aggregation into buckets | Distributions: request latency, payload size |
| **Summary** | Client-side quantile calculation | Pre-computed percentiles (less flexible than histogram) |

Rules:
- Use counters for events that accumulate. Derive rates from counters (`rate()`,
  `increase()`), never store pre-computed rates.
- Use gauges for current-state snapshots. Never `rate()` a gauge.
- Use histograms for latency and size distributions. Histograms enable percentile
  calculation across instances; summaries do not aggregate.
- Export timestamps as Unix epoch seconds, not "time since" values.
- Initialize all metrics with a zero value at startup to avoid missing-metric problems.

### What to Measure

#### The Four Golden Signals (Google SRE)

For every user-facing service, measure these four:

| Signal | What It Measures | Example |
|--------|-----------------|---------|
| **Latency** | Time to serve a request | `http_request_duration_seconds` histogram |
| **Traffic** | Demand on the system | `http_requests_total` counter by method/path |
| **Errors** | Rate of failed requests | `http_requests_total{status=~"5.."}` |
| **Saturation** | How "full" the service is | CPU usage, memory, queue depth, thread pool |

Distinguish **successful latency from error latency**. A fast 500 is not good latency.
A slow error is worse than a fast error. Track both.

#### RED Method (Request-Centric)

For every microservice:
- **R**ate — requests per second
- **E**rrors — failed requests per second
- **D**uration — distribution of request latency

RED is a focused subset of the golden signals, optimized for request-driven services.

#### USE Method (Resource-Centric)

For every resource (CPU, memory, disk, network, thread pool):
- **U**tilization — percentage of capacity in use
- **S**aturation — backlog / queue depth
- **E**rrors — resource-level error count

RED tells you *what* is degraded from the user's perspective. USE tells you *why* at the
infrastructure level. Use both together.

#### Service-Type Instrumentation

| Service Type | Key Metrics |
|--------------|-------------|
| **Online-serving** (HTTP, gRPC) | Request rate, error rate, latency (p50/p90/p99), in-flight requests |
| **Offline-processing** (workers, pipelines) | Items in/out per stage, processing duration, last-processed timestamp, queue depth |
| **Batch jobs** | Last successful completion time, job duration, records processed, exit status |
| **Caches** | Hit rate, miss rate, eviction count, latency to backend on miss |
| **Thread/connection pools** | Pool size, active count, queue length, wait time |

### Metric Naming

Metric names should be self-documenting. Follow these conventions:

- **Prefix with namespace.** `myapp_http_requests_total`, not `requests_total`.
- **Use base units.** Seconds (not milliseconds), bytes (not megabytes),
  ratio 0-1 (not percentage 0-100).
- **Suffix with unit.** `_seconds`, `_bytes`, `_total` (for unit-less counters).
- **One metric, one unit, one quantity.** Never mix request size with request duration
  in the same metric.
- **snake_case.** `http_request_duration_seconds`, not `httpRequestDurationSeconds`.

| Good | Bad |
|------|-----|
| `http_request_duration_seconds` | `request_latency` (no unit, ambiguous) |
| `http_requests_total` | `http_responses_500_total` (use labels) |
| `node_memory_usage_bytes` | `memory_mb` (not base unit) |
| `process_cpu_seconds_total` | `cpu_percent` (use ratio 0-1) |

### Labels and Cardinality

Labels add dimensions to a metric. Every unique combination of label values creates a
separate time series.

**Good labels** (bounded, low cardinality):
- `method` (GET, POST, PUT, DELETE)
- `status_code` (200, 404, 500 — or class: 2xx, 4xx, 5xx)
- `service`, `region`, `version`

**Dangerous labels** (unbounded, high cardinality):
- `user_id`, `email`, `session_id`
- `request_path` with dynamic segments (`/users/12345`)
- `error_message` (arbitrary strings)

Rules:
- Keep label cardinality below 10 values per label for most metrics.
- If a label can grow unbounded, it does not belong on a metric. Log it instead.
- Use labels instead of encoding dimensions in the metric name.
  `http_requests_total{method="GET"}`, not `http_get_requests_total`.
- Ensure `sum()` or `avg()` across all label values is meaningful. If not,
  split into separate metrics.

### Percentiles and Tail Latency

Averages hide outliers. A service with 100ms average latency may have 1% of requests
taking 5 seconds. That 1% tail can dominate user experience when users hit multiple
services per page load.

- Always track **p50, p90, p99** latency at minimum.
- Use histograms with exponentially distributed bucket boundaries
  (e.g., 5ms, 10ms, 25ms, 50ms, 100ms, 250ms, 500ms, 1s, 2.5s, 5s, 10s).
- Alert on p99, not mean. Mean latency alerts miss tail degradation.

</metrics>

---

## Distributed Tracing

<tracing>

### Core Concepts

| Concept | Definition |
|---------|------------|
| **Trace** | End-to-end record of a single request across all services |
| **Span** | One unit of work within a trace (HTTP call, DB query, function) |
| **Root span** | First span in a trace; has no parent |
| **Child span** | Span nested under a parent; represents a sub-operation |
| **Span context** | Immutable bag of `trace_id` + `span_id` + flags, propagated across boundaries |
| **Span attributes** | Key-value metadata on a span (http.method, db.statement) |
| **Span events** | Timestamped annotations within a span's lifetime |
| **Span links** | Causal references between spans in different traces |

### Span Kinds

| Kind | Direction | Example |
|------|-----------|---------|
| **Client** | Outgoing synchronous call | HTTP request to another service |
| **Server** | Incoming synchronous call | Handling an HTTP request |
| **Producer** | Creates async work | Publishing to a message queue |
| **Consumer** | Processes async work | Consuming from a message queue |
| **Internal** | No network boundary | In-process function instrumentation |

### Context Propagation

Context propagation is the mechanism that connects spans across process boundaries into a
single trace. Without it, you get disconnected spans, not traces.

Rules:
- **Propagate context on every outgoing call.** HTTP headers (W3C Trace Context or
  B3), message metadata, gRPC metadata — every cross-process boundary must carry
  trace context.
- **Extract context on every incoming call.** The receiving service must extract
  trace context and create a child span under the propagated parent.
- **Use W3C Trace Context (`traceparent`/`tracestate`)** as the default propagation
  format unless the ecosystem requires otherwise (e.g., legacy B3).
- **Never generate a new trace ID** when you should be continuing an existing trace.
  A new trace ID means a broken trace.

### What to Trace

Instrument at meaningful boundaries:

| Boundary | Instrument? |
|----------|-------------|
| Incoming HTTP/gRPC requests | Always — auto-instrument |
| Outgoing HTTP/gRPC calls | Always — auto-instrument |
| Database queries | Always — auto-instrument or manual |
| Cache operations | Yes — hit/miss as attribute |
| Queue publish/consume | Yes — link producer and consumer spans |
| Significant business operations | Yes — manual spans for key logic |
| Tight loops / trivial functions | No — noise, performance cost |

### Span Attributes

Attach attributes that enable filtering and analysis:

| Attribute | When |
|-----------|------|
| `http.method`, `http.route`, `http.status_code` | HTTP spans |
| `db.system`, `db.operation`, `db.statement` | Database spans |
| `messaging.system`, `messaging.operation` | Queue spans |
| `rpc.system`, `rpc.method` | RPC spans |
| `error` (boolean), `error.type`, `error.message` | Error conditions |
| `service.name`, `service.version` | All spans (set on resource) |

Use [semantic conventions](https://opentelemetry.io/docs/specs/semconv/) for attribute
names rather than inventing custom ones. Consistent naming enables cross-service analysis.

### Span Status

| Status | Meaning | When |
|--------|---------|------|
| `Unset` | Completed without error (default) | Most successful operations |
| `Error` | Operation failed | Server errors, exceptions |
| `Ok` | Explicitly marked successful | Only when you need to override ambiguity |

Leave status as `Unset` for normal success. Set `Error` only for actual failures. Do not
set `Error` for client errors like 404 on a server span — the server operated correctly.

### Sampling

At high traffic volumes, tracing 100% of requests is expensive. Sampling reduces cost
while preserving signal.

| Strategy | How It Works | Trade-off |
|----------|--------------|-----------|
| **Head-based** | Decide at trace start whether to sample | Simple; may miss rare errors |
| **Tail-based** | Decide after trace completes based on content | Catches errors; needs buffering infrastructure |
| **Always-on for errors** | Sample 100% of error traces, probabilistic for success | Good default balance |

Rules:
- Never drop error traces. If cost is a concern, sample successful traces at a lower
  rate but keep 100% of error and high-latency traces.
- Sample at the entry point (head) and propagate the decision. Do not let each
  service decide independently — this creates partial traces.
- Start with a low sampling rate (1-10%) and increase based on need, not the reverse.

</tracing>

---

## Connecting the Pillars

<interconnection>

The three pillars become powerful when correlated. An alert fires on a metric → you find
the offending trace → the trace points to a span → the span's logs reveal the root cause.

### Correlation Keys

| Key | Purpose | Where |
|-----|---------|-------|
| `trace_id` | Links logs and spans to the same trace | Logs, span context |
| `span_id` | Links a log to the exact span that produced it | Logs, span context |
| `request_id` | Correlates all work for one inbound request | Logs, HTTP headers |
| `service.name` + `service.version` | Groups telemetry by source | All signals |

Rules:
- **Embed trace_id and span_id in every log record** emitted within a request context.
  This is the primary bridge between logs and traces.
- **Use a correlation/request ID** that is assigned at the edge (API gateway, load
  balancer) and propagated to all downstream services.
- **Attach exemplars to metrics.** An exemplar is a trace_id attached to a specific
  metric observation, enabling drill-down from a metric spike to a representative trace.

### The Correlation Workflow

```
Metrics dashboard shows error rate spike
  → Filter by service + time window
  → Find exemplar trace_id on the error counter
  → Open trace in tracing UI
  → Identify the failing span (database timeout)
  → Search logs by trace_id for full error details
  → Root cause: connection pool exhausted
```

### Metric-to-Trace Exemplars

Exemplars attach a `trace_id` sample to a metric data point. When you see a latency
spike on a histogram, the exemplar gives you a concrete trace to investigate rather
than guessing.

- Attach exemplars to histogram observations for latency metrics.
- Attach exemplars to counter increments for error metrics.
- Not every metric point needs an exemplar — one per scrape interval is sufficient.

### Trace-to-Log Linking

When viewing a trace, each span should link to its logs. When viewing a log, the
`trace_id` should link back to the full trace. This bidirectional linking is the
backbone of incident investigation.

</interconnection>

---

## Anti-Patterns

| Anti-Pattern | Problem | Fix |
|--------------|---------|-----|
| Logging everything at DEBUG in production | Disk/cost explosion, noise buries signal | Default to INFO; enable DEBUG temporarily per-component |
| `catch (err) { log(err); throw err; }` at every layer | Same error logged N times across the call stack | Log once at the top-level handler |
| Metrics with unbounded label cardinality | Time series explosion; monitoring system degrades | Use bounded labels; move high-cardinality data to logs |
| Encoding dimensions in metric names | Cannot aggregate; proliferates metrics | Use labels: `requests_total{method="GET"}` |
| Averaging latency for alerting | Hides tail latency; misses degradation for minority of users | Alert on p99 from histograms |
| Missing trace context propagation | Broken traces; spans from different services are disconnected | Propagate context on every cross-process call |
| Sampling each service independently | Partial traces — some spans sampled, some dropped | Decide at head, propagate sampling decision |
| Logging PII / secrets | Compliance violations, security risk | Audit log fields; log opaque IDs, never raw PII |
| Alert on every metric wiggle | Alert fatigue; team ignores pages | Alert on symptoms (golden signals), not causes; require actionability |
| Treating WARN as a soft ERROR | WARN becomes noise nobody reads | WARN = system compensated but situation is unusual; ERROR = broken |
| Storing pre-computed rates instead of counters | Cannot re-aggregate over different windows | Store raw counters; derive rates at query time |
| No baseline metrics for new services | Cannot tell if behavior is normal or degraded | Instrument golden signals from day one, before first deploy |

---

## Application

<application>

### When Writing Code

- **Instrument from the start.** Add golden signal metrics, structured logging, and
  trace context propagation before the first production deploy — not after the first
  incident.
- **Follow the conventions silently.** Apply structured logging, metric naming, and
  tracing patterns without narrating each rule.
- **If the codebase has existing patterns, follow them.** Consistency within a codebase
  beats theoretical correctness. Flag divergences from this skill's guidance once, then
  move on.
- **Choose the right pillar.** Before adding instrumentation, ask: "Is this a metric,
  a log, or a span?" Use the decision table above.
- **Connect the signals.** Every log in a request context must carry `trace_id` and
  `span_id`. Every error metric should have an exemplar.

### When Reviewing Code

- **Check that new endpoints/operations have golden signal coverage.** Missing metrics
  on a new endpoint is a review blocker.
- **Verify structured logging.** Unstructured `log.Print("something happened")` in
  production code should be flagged with the fix inline.
- **Check log levels.** Expected client errors logged as ERROR, or debug noise left
  on at INFO, are common mistakes.
- **Verify trace context propagation.** Any new outgoing HTTP/gRPC/queue call must
  propagate trace context. Missing propagation breaks traces.
- **Check label cardinality.** New metric labels must be bounded. Flag unbounded labels
  (user IDs, free-text) immediately.
- **No sensitive data in logs or span attributes.** Passwords, tokens, PII in telemetry
  is a security and compliance defect.

```
Bad review comment:
  "According to observability best practices, you should consider
   adding structured logging with appropriate fields..."

Good review comment:
  "Missing trace_id in log context — requests through this handler
   won't correlate to traces. Add ctx.TraceID() to the logger fields."
```

</application>

---

## Integration

This skill provides observability discipline alongside other skills:

- **Coding skill** — Discovery, planning, verification workflow
- **Observability** (this skill) — What to log, measure, and trace
- **Tool-specific skills** (Prometheus, StatsD, OTel) — How to implement with a
  specific technology

The coding skill governs workflow. This skill governs observability design decisions.
Tool-specific skills govern implementation details for their respective technologies.

**Observability is not an afterthought. Instrument from day one. If you cannot observe
it, you cannot operate it.**