monitoring-observability · diff

v2.0 to v2.1

159 added, 223 removed. Audit A to A.

---

name: monitoring-observability

description: >

- Design or review observability for services, pipelines, and live operations: instrumentation,

- health signals, dashboards, alerting, retention, and ownership handoffs. Use when the main job

- is deciding what telemetry to emit, which symptoms deserve alerts, how to review dashboard/alert

- coverage, how to make data or marketing pipelines observable, or how to add crash/session visibility

- for game or multi-service systems. Not for root-cause log triage, code-level debugging, or engine-only

- profiler diagnosis — route those to `log-analysis`, `debugging`, `performance-optimization`,

- `langsmith`, or `game-performance-profiler` as appropriate.

+ Route observability work from the current packet into one monitoring brief.

+ Use when the main job is deciding service-health signals, telemetry rollout,

+ dashboard/alert coverage, pipeline freshness/schema monitoring, or game

+ live-ops visibility; choosing between service reliability, telemetry

+ foundation, review audit, data/pipeline, and live-ops modes; and naming one

+ smallest implementation slice. Route outage-log root cause to `log-analysis`,

+ code-level failure isolation to `debugging`, bottleneck tuning to

+ `performance-optimization`, rollout execution to `deployment-automation`,

+ LLM-specific tracing to `langsmith`, and engine-profiler interpretation to

+ `game-performance-profiler`.

allowed-tools: Bash Read Write Edit Glob Grep

compatibility: >

- Best for repositories, architecture docs, dashboards, alert rules, telemetry configs, runbooks,

- incident follow-ups, and planning tasks where the team needs a repeatable observability setup or

- review workflow rather than vendor-only copy/paste snippets.

+ Best for repositories, architecture notes, alert rules, dashboard inventories,

+ telemetry configs, incident follow-ups, data-pipeline docs, and launch

+ checklists where the real decision is what to measure, what should alert a

+ human, who owns the signal, and what work belongs in a neighboring skill.

metadata:

tags: observability, monitoring, telemetry, alerts, dashboards, slos, logging, traces, metrics

- version: "2.0"

+ version: "2.1"

source: akillness/oh-my-skills

---

# Monitoring & Observability

- Use this skill to turn a vague “we need monitoring” request into a **mode-specific observability brief**.

+ Use this skill when the main question is **"what packet do we have, what should this system notice, and what should interrupt a human?"**

- The goal is **not** to dump a vendor tutorial.

- The goal is to decide:

- 1. what surface needs observability,

- 2. which signals matter,

- 3. what should page a human versus stay informational,

- 4. what ownership or route-out belongs elsewhere.

+ The job is not to dump a Prometheus / Grafana / Datadog tutorial.

+ The job is to normalize the packet, pick one primary observability mode, define the smallest useful signal plan, and route adjacent work away before the skill turns into debugging, performance tuning, rollout execution, or analytics reporting.

- Read [references/modes-and-boundaries.md](references/modes-and-boundaries.md) before handling mixed requests that blur telemetry setup, log triage, performance diagnosis, or product analytics.

- Read [references/alert-dashboard-checklist.md](references/alert-dashboard-checklist.md) when reviewing dashboards, alerts, or SLO coverage.

- Read [references/telemetry-rollout-matrix.md](references/telemetry-rollout-matrix.md) when choosing instrumentation, retention, sampling, and ownership defaults.

+ Read [references/intake-packets-and-route-outs.md](references/intake-packets-and-route-outs.md) before handling an unfamiliar packet.

+ Read [references/modes-and-boundaries.md](references/modes-and-boundaries.md) before handling mixed requests that blur telemetry setup, incident diagnosis, or product analytics.

+ Read [references/alert-dashboard-checklist.md](references/alert-dashboard-checklist.md) when reviewing dashboards, alerts, and ownership gaps.

+ Read [references/telemetry-rollout-matrix.md](references/telemetry-rollout-matrix.md) when choosing the smallest rollout slice.

## When to use this skill

- - Set up observability before launch for a web, backend, worker, or multi-service system

- - Review whether current metrics, logs, traces, health checks, and alerts are enough

- - Design symptom-first alerts, dashboard questions, SLO/SLI coverage, or metamonitoring

- - Add telemetry foundations for traces / metrics / logs without tying the workflow to one vendor

- - Make product, marketing, analytics, or BI pipelines observable via freshness / schema / volume / lineage checks

- - Add crash, alert, and live-ops visibility for game services, builds, or launch events

- - Define ownership, retention, or handoff rules after an incident showed telemetry gaps

+ - New service, worker, API, or multi-service system needs health signals, alerts, dashboards, or SLO-style coverage before launch

+ - Existing stack has dashboards / alerts / telemetry, but trust is low and a keep/fix/delete/add audit is needed

+ - Team needs to decide what to instrument, correlate, retain, or sample before choosing vendors or backend specifics

+ - Data, marketing, analytics, or pipeline work needs freshness / schema / volume / lineage monitoring rather than another manual trust check

+ - Game or live-ops work needs crash, session, build, or launch-event visibility without turning into engine-profiler interpretation

+ - Cross-functional reliability asks span backend, product/ops, marketing pipelines, and game live-ops, and the next owner is still unclear

## When not to use this skill

- - **Root-cause triage on existing logs** → use `log-analysis`

- - **Code-level reproduction / bug fixing** → use `debugging`

- - **Bottleneck diagnosis or tuning after measurements exist** → use `performance-optimization`

- - **LLM tracing / evaluation / prompt-observability workflows** → use `langsmith`

- - **Unity / Unreal frame-time and profiler-capture interpretation** → use `game-performance-profiler`

- - **Release rollout / deploy execution** → use `deployment-automation`

- ## Mode selection

- Choose one primary mode before proposing tooling.

- | Mode | Use when | Main output |

- |------|----------|-------------|

- | Service reliability | API/app/worker/service needs health signals, SLOs, dashboards, alerts | service observability brief |

- | Telemetry foundation | team needs instrumentation, event naming, traces/metrics/logs coverage | telemetry rollout plan |

- | Data / pipeline observability | analytics, marketing, BI, or data pipelines need reliability checks | data-health monitoring brief |

- | Game / live-ops visibility | crashes, player-session health, launch-event stability, backend game services | live-ops observability brief |

- | Review / gap audit | existing stack exists but trust is low or incidents escaped detection | observability review + gap list |

- If multiple modes appear, pick the primary bottleneck and list the others as secondary follow-ups.

+ - **The packet is mainly logs and the job is finding the first actionable failure** → `log-analysis`

+ - **The job is reproduce → isolate → verify for a code bug or regression** → `debugging`

+ - **Measurements already exist and the main job is naming a bottleneck or tuning it** → `performance-optimization`

+ - **The main job is release execution, promotion, rollback, or post-deploy sequencing** → `deployment-automation`

+ - **The work is LLM-specific traces, evals, or prompt-observability** → `langsmith`

+ - **The work is Unity/Unreal/Godot frame-time capture interpretation** → `game-performance-profiler`

+ - **The task is pure dashboard/report presentation on curated BigQuery data** → `looker-studio-bigquery`

## Instructions

- ### Step 1: Label the surface before choosing tools

- Capture the minimum facts first.

+ ### Step 1: Frame the packet

+ Record the smallest useful intake statement before recommending tooling.

- Record:

- - request type: new setup | review / audit | incident follow-up | migration | platform/tool switch

+ Capture:

- Do **not** start by asking “Prometheus or Datadog?” Start by labeling the workflow.

- ### Step 2: Choose the primary observability mode

- #### Mode A — Service reliability

- Use when the core job is service health, paging, and user-visible behavior.

+ Quick frame:

+ ```markdown

+ Surface: data/pipeline

+ Request type: review/audit

+ Current packet: stale dashboard complaint + job ownership notes

+ Impact: stale data and trust erosion

+ Ownership: data/ops + dashboard consumer owner

+ ```

- Focus on:

- - request rate / traffic shape

- - error rate / failure class

- - latency distribution / saturation

- - black-box probes and metamonitoring

- - SLO / SLI definitions for important journeys

+ ### Step 2: Start from the intake packet

+ Use [references/intake-packets-and-route-outs.md](references/intake-packets-and-route-outs.md).

- Return:

- - top 3–5 service questions the dashboard must answer

- - symptom-first alerts only

- - what must page now vs ticket later vs dashboard only

- - missing instrumentation or health endpoints

+ Choose the packet the user actually has now:

+ - service / reliability packet

+ - telemetry-foundation packet

+ - data / pipeline packet

+ - review / audit packet

+ - game / live-ops packet

+ - no usable packet yet

- #### Mode B — Telemetry foundation

- Use when the team needs to instrument the system, correlate signals, or standardize telemetry.

+ Output this step as:

+ ```markdown

+ ## Intake Packet

+ - Current packet:

+ - Why it is enough (or not enough):

+ - Missing context to collect next:

+ ```

- Focus on:

- - traces, metrics, logs, and correlation IDs

- - event naming / dimensions / labels

- - instrumentation ownership and rollout slices

- - retention / sampling / cardinality risk

- - vendor-neutral export path first, backend second

+ Rule: do not force a vendor comparison or telemetry-stack rewrite if the current packet already narrows the next decision.

- Return:

- - telemetry coverage map

- - required correlation fields (`request_id`, `trace_id`, `job_id`, `user_id`, etc.)

- - initial rollout order

- - unsafe telemetry patterns to avoid

+ ### Step 3: Choose one primary observability mode

+ Pick one primary mode from [references/modes-and-boundaries.md](references/modes-and-boundaries.md).

- #### Mode C — Data / pipeline observability

- Use when freshness, schema drift, volume anomalies, or broken downstream dashboards are the real issue.

+ Primary modes:

+ - `service-reliability`

+ - `telemetry-foundation`

+ - `data-pipeline-observability`

+ - `game-liveops-visibility`

+ - `review-gap-audit`

+ - `unknown-needs-better-packet`

- Focus on:

- - freshness / lateness

- - volume / duplicates / drops

- - schema drift

- - distribution / null rate / metric anomalies

- - lineage / downstream blast radius

+ Rule: one primary mode, optional secondary mode.

+ Do not blend launch telemetry, stale dashboard audits, crash visibility, and generic instrumentation into one answer.

- Return:

- - the most important pipeline-health dimensions

- - likely owners for each alert class

- - where dashboard trust can silently break

- - which checks are table-level, job-level, and consumer-level

+ ### Step 4: Name the core monitoring question

+ Before listing tools or metrics, state what the system must answer.

- #### Mode D — Game / live-ops visibility

- Use when player experience, launch stability, or cross-build crash visibility matters.

+ Good examples:

+ - “Would user-visible API pain page us before customers report it?”

+ - “Do we know when Monday’s growth dashboard is stale, why it is stale, and who owns the fix?”

+ - “Can launch-event crashes be grouped by build/platform and escalated before social reports spike?”

+ - “Do current alerts point responders to one useful dashboard/runbook instead of three noisy symptoms?”

- Focus on:

- - crash reporting and issue grouping

- - build / version / platform tags

- - player-session and backend-service health

- - launch-event alert thresholds

- - logs / breadcrumbs / device context / purchase-session context where relevant

+ Avoid vague statements like “set up better monitoring.”

- Return:

- - live-ops event checklist

- - crash/context fields that must be attached

- - which signals belong in backend telemetry vs crash tooling

- - route-out to `game-performance-profiler` if the issue becomes frame-time profiling

+ ### Step 5: Build one smallest signal plan

+ Use [references/telemetry-rollout-matrix.md](references/telemetry-rollout-matrix.md).

- #### Mode E — Review / gap audit

- Use when dashboards, alerts, or tooling exist but confidence is low.

+ For the chosen mode, define:

+ - primary questions the dashboard / alert path must answer

+ - evidence surfaces: metrics, logs, traces, black-box checks, crash tooling, freshness checks, lineage views

+ - page now vs ticket later vs dashboard-only thresholds

+ - owner for each alert or dashboard family

+ - the first 1–3 implementation slices only

- Focus on:

- - what incidents would still escape detection

- - noisy / non-actionable alerts

- - dashboards with no clear question or owner

- - missing black-box coverage / metamonitoring

- - orphan telemetry that nobody uses

+ Rules:

+ - Alert on symptoms before internal causes.

+ - Prefer one clear page over many stack-layer pages for the same incident.

+ - Keep labels bounded; push high-cardinality detail into logs/traces.

+ - Include runbook/dashboard links whenever an alert expects human action.

+ - Treat metamonitoring as first-class when alert delivery can fail silently.

- Return:

- - keep / fix / delete / add decisions

- - top false-positive and false-negative risks

- - ownership gaps

- - smallest high-value remediation order

+ ### Step 6: Make route-outs explicit

+ Hand work off when the job shifts.

- ### Step 3: Build the observability brief

- Return a concise report with this shape:

+ Common route-outs:

+ - root-cause log forensics → `log-analysis`

+ - correctness-first regression hunt → `debugging`

+ - bottleneck diagnosis or tuning → `performance-optimization`

+ - release execution / post-deploy rollback path → `deployment-automation`

+ - LLM tracing / evals / prompt observability → `langsmith`

+ - KPI interpretation / stakeholder evidence summary → `data-analysis`

+ - BigQuery-backed dashboard presentation layer → `looker-studio-bigquery`

+ - engine-profiler interpretation → `game-performance-profiler`

+ ### Step 7: Return the observability brief

```markdown

# Observability Brief

## Scope

- - System type: ...

- - Environment: ...

- - Primary mode: ...

- - Confidence: high | medium | low

- ## Current signal state

- - What exists already

- - What is missing or unreliable

+ - Surface:

+ - Request type:

+ - Intake packet:

+ - Primary mode:

+ - Confidence:

- ## Primary questions to answer

- - 3-5 questions dashboards / alerts must answer

+ ## Core Monitoring Question

+ - ...

- ## Signal plan

- - Metrics: ...

- - Logs: ...

- - Traces: ...

- - Black-box / health checks: ...

+ ## Signal Plan

+ - Metrics / checks:

+ - Logs / traces / crash context:

+ - Dashboards / views:

+ - Alert policy:

- ## Alert policy

- - Page now: ...

- - Ticket / backlog: ...

- - Dashboard only: ...

+ ## Ownership

+ - Primary owner:

+ - Secondary owner(s):

- ## Ownership and handoffs

- - Primary owner: ...

- - Route-outs: ...

+ ## First Implementation Slice

+ 1. ...

+ 2. ...

+ 3. ...

- ## First implementation slice

- - 1-3 smallest high-value changes

+ ## Route-outs

+ - ...

```

- ### Step 4: Apply symptom-first alert rules

- Use these defaults unless the evidence says otherwise.

- - Alert on **user-visible symptoms** before internal causes

- - Prefer **one clear page** over many stack-layer pages for the same failure

- - Include **runbook/dashboard links** in alert context

- - Add **slack** for brief blips; avoid paging for self-healing noise

- - Metamonitor the monitoring path itself when alert delivery is mission-critical

- Good examples:

- - sustained API error-rate spike affecting user requests

- - stale data beyond the business tolerance window

- - launch-event crash rate above threshold by platform/build

- - message backlog age causing downstream user-visible delay

- Weak examples:

- - every 5xx at every layer

- - every restart with no impact

- - every metric anomaly without action

- - alerts nobody can own

- ### Step 5: Pick the right evidence surface

- Do not recommend one surface for every problem.

- - **Metrics** for rate, latency, capacity, saturation, freshness age, backlog age

- - **Logs** for context and event details once symptoms are detected

- - **Traces** for request flow and dependency boundaries

- - **Black-box checks** for externally visible health

- - **Dashboards** for ongoing review, not as the only alert mechanism

- - **Issue/crash tools** for grouped error context, breadcrumbs, and release/build impact

- If the system only has logs, say so clearly and mark observability maturity as limited.

- ### Step 6: Make route-outs explicit

- When the job shifts, hand it off.

- - **“Find the actual failing line in these logs”** → `log-analysis`

- - **“We already know latency is bad; find the bottleneck”** → `performance-optimization`

- - **“This looks like a code bug, not a telemetry design issue”** → `debugging`

- - **“LLM trace quality / evals / prompt observability”** → `langsmith`

- - **“Unity/Unreal frame-time capture interpretation”** → `game-performance-profiler`

- - **“Deploy/release plan with post-deploy checks”** → `deployment-automation`

## Examples

### Example 1: New API before launch

- **Prompt:**

- > We’re launching a new API next week. Tell me what to instrument and what should alert us.

+ **Input:** “We’re launching a new API next week. Tell me what to instrument and what should page us.”

- Use **Mode A — Service reliability** plus a small **Mode B** foundation slice.

- Return rate/error/latency/saturation questions, minimal tracing fields, black-box checks, and symptom-first alert thresholds.

+ **Expected shape:** classify as `service-reliability`, use the current launch/readiness packet, define RED / golden-signals questions plus black-box coverage, page thresholds, and one smallest rollout slice.

- ### Example 2: Marketing dashboard keeps going stale

- **Prompt:**

- > Our Monday morning growth dashboard is stale half the time. We need observability, not another manual spreadsheet check.

+ ### Example 2: Growth dashboard keeps going stale

+ **Input:** “Our Monday morning growth dashboard is stale half the time. We need observability, not another spreadsheet check.”

- Use **Mode C — Data / pipeline observability**.

- Return freshness/schema/volume/lineage checks, ownership by pipeline/job/dashboard layer, and alert thresholds tied to business tolerance windows.

+ **Expected shape:** classify as `data-pipeline-observability`, cover freshness/schema/volume/lineage/ownership, and distinguish dashboard trust checks from KPI interpretation work.

- ### Example 3: Game launch-event visibility

- **Prompt:**

- > We need crash alerts and player-session visibility for our Unity event weekend, but this isn’t a profiler question yet.

+ ### Example 3: Review / gap audit

+ **Input:** “We have tons of alerts and dashboards, but nobody trusts them. What should we keep versus delete?”

- Use **Mode D — Game / live-ops visibility**.

- Return crash/build/platform tags, session-health metrics, launch alert levels, and route-outs to `game-performance-profiler` only if frame-time evidence becomes the bottleneck.

+ **Expected shape:** classify as `review-gap-audit`, use the alert/dashboard checklist, produce keep/fix/delete/add decisions, and call out noisy pages, ownerless panels, and missing metamonitoring.

- ### Example 4: Boundary check

- **Prompt:**

- > Here are the logs from the outage — what’s the root cause?

+ ### Example 4: Route-out to rollout execution

+ **Input:** “We just deployed and need a step-by-step rollback/promotion checklist with health checks.”

- Do **not** use this as the main workflow. Route to `log-analysis` and say observability improvements can be proposed after the first actionable failure is identified.

+ **Expected shape:** route the execution workflow to `deployment-automation`, while optionally noting the few post-deploy observability questions that matter.

+ ### Example 5: Route-out to log forensics

+ **Input:** “Here are the outage logs. Find the root cause.”

+ **Expected shape:** do not use this as the main workflow; route to `log-analysis` and only propose observability follow-up after the first actionable failure is identified.

## Best practices

- 1. Start with the workflow shape, not the vendor choice.

- 2. Prefer symptom-first alerts with low noise.

- 3. Distinguish instrumentation gaps from incident diagnosis.

- 4. Treat data/marketing pipeline observability as first-class, not as an afterthought.

- 5. For games, separate live-ops visibility from engine profiler interpretation.

- 6. Make owners and handoffs explicit, especially after incidents.

- 7. Keep dashboards question-driven; delete dead dashboards and dead alerts.

- 8. Call out missing metamonitoring when alert delivery itself can fail.

+ 1. Start with the packet and core monitoring question, not the vendor.

+ 2. Keep one primary mode and one smallest implementation slice.

+ 3. Alert on symptoms, not every possible cause.

+ 4. Make ownership, runbooks, and dashboard links explicit.

+ 5. Treat stale data / pipeline trust as first-class observability work.

+ 6. Separate game live-ops visibility from engine-profiler interpretation.

+ 7. Keep review/audit work honest: dead dashboards and noisy alerts should be deleted, not merely documented.

+ 8. Sync compact discovery surfaces whenever the front-door boundary changes.

## References

- [Google SRE — Monitoring Distributed Systems](https://sre.google/sre-book/monitoring-distributed-systems/)

- [OpenTelemetry — Observability primer](https://opentelemetry.io/docs/concepts/observability-primer/)

- - [Prometheus — Alerting best practices](https://prometheus.io/docs/practices/alerting/)

- - [Grafana — What is observability?](https://grafana.com/blog/what-is-observability-best-practices-key-metrics-methodologies-and-more/)

- - [Databricks — What is Data Observability?](https://www.databricks.com/blog/what-is-data-observability)

- - [Sentry — Game developers](https://sentry.io/solutions/game-developers/)

+ - [Prometheus — Alerting](https://prometheus.io/docs/practices/alerting/)

+ - [Grafana — Dashboard best practices](https://grafana.com/docs/grafana/latest/visualizations/dashboards/build-dashboards/best-practices/)