Immutable. This exact content is served forever at /api/v1/blob/b4835d3f572e09d0.
---
name: observability
description: >
· Instrument and audit observability: metrics, traces, logs, alerts, SLOs, dashboards.
Triggers: observability, metrics, tracing, prometheus, opentelemetry, grafana, slo. Not for
live diagnostics (cluster-health) or manifests (kubernetes).
license: MIT
compatibility: "Optional: prometheus, otelcol, grafana, promtool, amtool, jq"
metadata:
source: iuliandita/skills
date_added: "2026-06-14"
effort: high
argument_hint: "[service-or-stack]"
---
# Observability
Design and audit the signals a running system emits so failures are visible before users report
them. This skill builds the standing pipeline - instrumentation, metrics, traces, structured
logs, alert rules, SLOs, and dashboards-as-code - and audits a repo for the gaps that leave a
service blind. It produces config (exporters, recording/alerting rules, OTLP pipelines,
dashboard JSON), so the AI Self-Check applies.
**Target versions**: see `references/versions.md` (verified per the receipt date
in that file). Do not restate version numbers here.
## When to use
- Adding instrumentation to a service: metrics (Prometheus/OTLP), traces (OpenTelemetry), or
structured logs
- Writing or reviewing alert rules, recording rules, and Alertmanager routing
- Defining SLOs and error budgets, and the multi-burn-rate alerts that back them
- Building dashboards as code (provisioned JSON, grafonnet/Grizzly)
- Designing the signal collection layer: OTel Collector pipelines, exporters, scrape config
- Auditing a repo for observability gaps: uninstrumented services, no SLOs, alert-fatigue
patterns, cardinality risks, logs with no trace correlation
## When NOT to use
- Checking whether a live cluster is healthy right now (point-in-time, read-only diagnostics) -
use **cluster-health**
- Writing or reviewing Kubernetes manifests, Helm charts, or the Prometheus Operator CRDs as
K8s objects - use **kubernetes**
- Wiring CI/CD pipelines (the skill defines what they should emit and gate on, not the pipeline
itself) - use **ci-cd**
- Localizing an unknown-layer live failure once signals exist (consuming signals to find root
cause) - use **debug-triage**
- Application security review or secret scanning in telemetry - use **security-audit**
---
## AI Self-Check
AI tools produce the same observability mistakes. Before returning any generated instrumentation,
rule, pipeline, or dashboard, verify:
- [ ] **Metric cardinality bounded**: no unbounded label values (user IDs, request paths with IDs,
timestamps, full URLs) on metrics. High cardinality is the top cause of Prometheus OOM.
- [ ] **Rules validated**: PromQL/alert rules pass `promtool check rules`; routing passes
`amtool config routes`. AI invents plausible-but-wrong PromQL functions and label matchers.
- [ ] **Alerts are actionable**: every alert has `for:`, severity, a runbook link, and fires on
symptoms (SLO burn, user-facing error) not raw causes. No alert that a human cannot act on.
- [ ] **SLO math is real**: error budget = `1 - SLO`; burn-rate alerts use multi-window
multi-burn-rate, not a single threshold. State the window and budget explicitly.
- [ ] **Trace context propagated**: W3C `traceparent` propagation is configured end to end;
spans carry `service.name`. Logs include `trace_id`/`span_id` for correlation.
- [ ] **No secrets in telemetry**: no tokens, auth headers, PII, or full request bodies in span
attributes, log fields, or metric labels.
- [ ] **Versions and signals real**: exporter names, OTLP receiver/exporter names, PromQL
functions, and Grafana panel types verified against current docs - not assumed.
- [ ] **Sampling intentional**: trace sampling rate is stated and justified (head vs tail), not
silently defaulted; 100% sampling on a hot path is flagged.
---
## Workflow
### Step 1: Identify the signals and the questions
Pin down what the system must answer before choosing tools. For each service: what does "broken"
look like to a user, and which signal proves it? Map to the **golden signals** (latency, traffic,
errors, saturation) or **RED** (rate, errors, duration) for request-driven services and **USE**
(utilization, saturation, errors) for resources. Pick the minimum signal set that answers those
questions - do not instrument everything because you can.
### Step 2: Choose the collection path
| Need | Default |
|---|---|
| Metrics | Prometheus scrape, or OTLP metrics through the OTel Collector to a Prometheus-compatible store |
| Traces | OpenTelemetry SDK -> OTLP -> Collector -> Tempo (or vendor backend) |
| Logs | Structured JSON -> agent (Alloy/Promtail/OTel) -> Loki |
| Unified pipeline | OpenTelemetry Collector as the single ingest/route/transform layer |
Prefer OTLP and the OTel Collector as the vendor-neutral seam: instrument once, re-route backends
in config. Use direct Prometheus scrape where pull and existing exporters already fit.
### Step 3: Instrument and configure
- **Metrics**: use auto-instrumentation where it exists; add custom metrics only for
domain-specific questions. Keep labels low-cardinality. Add recording rules for expensive
queries that dashboards or alerts repeat.
- **Traces**: enable context propagation, set `service.name` and resource attributes, choose a
sampling strategy (head sampling at the SDK, or tail sampling in the Collector for
error/latency-biased retention).
- **Logs**: emit structured JSON, include `trace_id`/`span_id`, avoid logging what a metric
already counts.
- **Alerts and SLOs**: write symptom-based alert rules, define SLOs with explicit windows, back
them with multi-window multi-burn-rate alerts, route by severity in Alertmanager.
- **Dashboards**: keep them as code (provisioned JSON or grafonnet) so they are reviewable and
reproducible, not click-built.
### Step 4: Validate
- `promtool check config` / `promtool check rules` for Prometheus config and rules
- `promtool test rules` for unit tests on alerting/recording rules against sample series
- `otelcol validate --config` for Collector pipelines
- `amtool config routes test` / `amtool check-config` for Alertmanager routing
- Confirm a test signal traverses the full path (emit -> collect -> store -> query -> alert) on at
least one service before declaring coverage
- Start with `references/runnable-examples.md` when a task needs compact Collector, SLO-rule,
rule-test, or dashboard artifacts that can be passed to the validators above.
---
## Signals reference
### Metrics
- Naming: `unit`-suffixed, base units (seconds, bytes), `_total` for counters. Follow Prometheus
and OpenTelemetry semantic conventions; do not invent metric names where a convention exists.
- Cardinality is the budget. Series count = product of label-value sets. Keep label values bounded
and finite. Per-entity detail (user, request, order, full path) belongs on a trace attribute or
log field, never a metric label. Exemplars link a metric sample to a trace - enable them for
latency histograms.
- Recording rules precompute heavy expressions; alerting rules fire on conditions. Keep them in
version control and unit-test them with `promtool test rules`.
### Traces
- One trace = one request across services, stitched by propagated context. Without propagation you
get disconnected spans, not traces.
- Sampling: head sampling is cheap and simple but blind to rare errors; tail sampling (in the
Collector) keeps error/slow traces at the cost of buffering. State which and why.
- TraceQL queries Tempo; exemplars and `trace_id` in logs are the cross-signal jumps that make a
trace findable from a metric spike or a log line.
### Logs
- Structured over free text: JSON fields are queryable (LogQL), prose is grep-only. Include
`service`, `level`, `trace_id`, `span_id`, and a stable message key.
- Logs are the most expensive signal per byte of insight. If a metric can answer it, count it;
reserve logs for the context a metric cannot carry.
### Alerts and SLOs
- An SLO is a target on an SLI (e.g. 99.9% of requests < 300ms over 30 days). Error budget is the
allowed failure: `1 - SLO`. Alert on **budget burn rate**, not on every breach.
- The burn-rate alert threshold is `burn_rate * (1 - SLO)` compared against the error-*ratio* SLI -
not the raw error rate against a bare multiplier (a common off-by-budget bug). Standard tiers for
a 99.9% SLO: fast-burn 14.4x (1h + 5m windows, page), slow-burn 6x (6h + 30m, page), erosion 3x
(24h + 2h, ticket). Both windows in a tier must breach together before the alert fires - express
as an `and`: `ratio_rate1h > 14.4*(1-SLO) and ratio_rate5m > 14.4*(1-SLO)`.
- Multi-window multi-burn-rate alerting (fast-burn + slow-burn windows) catches both acute
outages and slow erosion while suppressing flapping. A single static threshold does neither.
- Alert hygiene: page only on user-impacting symptoms with a runbook; everything else is a ticket
or a dashboard. Alert fatigue is an outage you stop seeing.
### Dashboards as code
- Provision dashboards from version-controlled JSON or generate them with grafonnet/Grizzly. A
click-built dashboard is an undiffable, unreviewable, un-restorable artifact.
- Committed-in-git is necessary but not sufficient: an *exported* JSON blob (regenerated on each
export, never hand-edited) still drifts from the running dashboard and diffs unreadably. The
source of truth is authored or generated config that flows file -> Grafana, not the reverse.
---
## Audit lens (Wave 3 in deep-audit)
When auditing a repo for observability, report findings on:
- **Coverage gaps**: services that emit no metrics/traces/logs; endpoints with no latency or error
signal; background jobs with no success/failure metric.
- **No SLOs / no error budget**: alerting exists but is threshold-based with no SLO backing.
- **Alert anti-patterns**: cause-based alerts with no `for:`, no runbook, no severity; duplicate or
flapping alerts; paging on non-actionable conditions.
- **Cardinality risk**: unbounded labels (IDs, paths, emails) on metrics; high-cardinality log
fields used as metric labels.
- **Broken correlation**: logs without `trace_id`; traces without `service.name`; metrics without
exemplars on key histograms.
- **Untested rules**: alert/recording rules with no `promtool test rules` coverage.
- **Drift risk**: dashboards stored as exported blobs nobody edits, or not in version control.
Report only what the repo files show. Do not assume a running backend exists; flag "signal defined
but no evidence it is collected" as a gap, not a pass.
---
## Output Contract
See `references/output-contract.md` for the full contract.
- **Skill name:** OBSERVABILITY
- **Deliverable bucket:** `audits`
- **Mode:** conditional. When invoked to **audit a repo for observability gaps** (the Wave 3 lens), emit the full contract - boxed inline header, body summary inline plus per-finding detail in the deliverable file, boxed conclusion, conclusion table - and write the deliverable to `docs/local/audits/observability/<YYYY-MM-DD>-<slug>.md`. When invoked to **build instrumentation, rules, pipelines, or dashboards**, respond freely without the contract.
- **Severity scale:** `P0 | P1 | P2 | P3 | info` (see shared contract; only used in audit mode).
## Related Skills
- **cluster-health** - point-in-time, read-only Kubernetes diagnostics ("is it healthy now").
Observability builds the standing signal pipeline ("can we see it over time"). cluster-health
reads signals live; observability defines and audits them.
- **kubernetes** - authors manifests, Helm, and Operator CRDs as K8s objects. Observability authors
the instrumentation, rules, and SLO/alert config those objects carry, platform-agnostic.
- **debug-triage** - consumes signals to localize an unknown-layer live failure. Observability
produces the signals it consumes: producer vs consumer.
- **ci-cd** - wires the pipeline. Observability defines what the pipeline should emit and gate on,
not the pipeline itself.
- **security-audit** - reviews exploitable vulnerabilities and secret exposure. Observability flags
secrets-in-telemetry as a gap but does not replace a security review.
## Rules
1. **Cardinality is a hard budget.** Never put unbounded values (IDs, paths, emails, timestamps) in
metric labels. Bounded label sets only.
2. **Validate before returning.** Run `promtool`, `otelcol validate`, and `amtool` on generated
config and rules; do not ship unverified PromQL or routing.
3. **Alert on symptoms with runbooks.** Every alert is actionable, has `for:` and severity, and
links a runbook. No cause-only or non-actionable pages.
4. **SLO-back the alerts.** Define SLOs with explicit windows and use multi-window multi-burn-rate
alerting, not single static thresholds.
5. **Propagate context.** Configure W3C trace propagation, `service.name`, and `trace_id`/`span_id`
in logs so signals correlate.
6. **No secrets in telemetry.** No tokens, PII, auth headers, or full bodies in labels, span
attributes, or log fields.
7. **Dashboards as code.** Provision from version control; never treat a click-built dashboard as
the source of truth.
8. **Run the AI Self-Check** before returning any generated instrumentation, rule, pipeline, or
dashboard.
9. **Verify versions and signal names.** Confirm exporter/receiver names, PromQL functions, and
panel types against current docs; pin versions with dates.