monitoring-observability · diff
git:20260319.0905165 to v2.0
226 added, 331 removed. Audit A to A.
---
name: monitoring-observability
- description: Set up monitoring, logging, and observability for applications and infrastructure. Use when implementing health checks, metrics collection, log aggregation, or alerting systems. Handles Prometheus, Grafana, ELK Stack, Datadog, and monitoring best practices.
+ description: >
+ Design or review observability for services, pipelines, and live operations: instrumentation,
+ health signals, dashboards, alerting, retention, and ownership handoffs. Use when the main job
+ is deciding what telemetry to emit, which symptoms deserve alerts, how to review dashboard/alert
+ coverage, how to make data or marketing pipelines observable, or how to add crash/session visibility
+ for game or multi-service systems. Not for root-cause log triage, code-level debugging, or engine-only
+ profiler diagnosis — route those to `log-analysis`, `debugging`, `performance-optimization`,
+ `langsmith`, or `game-performance-profiler` as appropriate.
+ allowed-tools: Bash Read Write Edit Glob Grep
+ compatibility: >
+ Best for repositories, architecture docs, dashboards, alert rules, telemetry configs, runbooks,
+ incident follow-ups, and planning tasks where the team needs a repeatable observability setup or
+ review workflow rather than vendor-only copy/paste snippets.
metadata:
- tags: monitoring, observability, logging, metrics, Prometheus, Grafana, alerts
- platforms: Claude, ChatGPT, Gemini
- allowed-tools: Bash Read Write
+ tags: observability, monitoring, telemetry, alerts, dashboards, slos, logging, traces, metrics
+ version: "2.0"
+ source: akillness/oh-my-skills
---
-
# Monitoring & Observability
-
- ## When to use this skill
-
- - **Before Production Deployment**: Essential monitoring system setup
- - **Performance Issues**: Identify bottlenecks
- - **Incident Response**: Quick root cause identification
- - **SLA Compliance**: Track availability/response times
+ Use this skill to turn a vague “we need monitoring” request into a **mode-specific observability brief**.
- ## Instructions
+ The goal is **not** to dump a vendor tutorial.
+ The goal is to decide:
+ 1. what surface needs observability,
+ 2. which signals matter,
+ 3. what should page a human versus stay informational,
+ 4. what ownership or route-out belongs elsewhere.
- ### Step 1: Metrics Collection (Prometheus)
+ Read [references/modes-and-boundaries.md](references/modes-and-boundaries.md) before handling mixed requests that blur telemetry setup, log triage, performance diagnosis, or product analytics.
+ Read [references/alert-dashboard-checklist.md](references/alert-dashboard-checklist.md) when reviewing dashboards, alerts, or SLO coverage.
+ Read [references/telemetry-rollout-matrix.md](references/telemetry-rollout-matrix.md) when choosing instrumentation, retention, sampling, and ownership defaults.
- **Application Instrumentation** (Node.js):
- ```typescript
- import express from 'express';
- import promClient from 'prom-client';
+ ## When to use this skill
+ - Set up observability before launch for a web, backend, worker, or multi-service system
+ - Review whether current metrics, logs, traces, health checks, and alerts are enough
+ - Design symptom-first alerts, dashboard questions, SLO/SLI coverage, or metamonitoring
+ - Add telemetry foundations for traces / metrics / logs without tying the workflow to one vendor
+ - Make product, marketing, analytics, or BI pipelines observable via freshness / schema / volume / lineage checks
+ - Add crash, alert, and live-ops visibility for game services, builds, or launch events
+ - Define ownership, retention, or handoff rules after an incident showed telemetry gaps
- const app = express();
+ ## When not to use this skill
+ - **Root-cause triage on existing logs** → use `log-analysis`
+ - **Code-level reproduction / bug fixing** → use `debugging`
+ - **Bottleneck diagnosis or tuning after measurements exist** → use `performance-optimization`
+ - **LLM tracing / evaluation / prompt-observability workflows** → use `langsmith`
+ - **Unity / Unreal frame-time and profiler-capture interpretation** → use `game-performance-profiler`
+ - **Release rollout / deploy execution** → use `deployment-automation`
- // Default metrics (CPU, Memory, etc.)
- promClient.collectDefaultMetrics();
+ ## Mode selection
+ Choose one primary mode before proposing tooling.
- // Custom metrics
- const httpRequestDuration = new promClient.Histogram({
- name: 'http_request_duration_seconds',
- help: 'Duration of HTTP requests in seconds',
- labelNames: ['method', 'route', 'status_code']
- });
+ | Mode | Use when | Main output |
+ |------|----------|-------------|
+ | Service reliability | API/app/worker/service needs health signals, SLOs, dashboards, alerts | service observability brief |
+ | Telemetry foundation | team needs instrumentation, event naming, traces/metrics/logs coverage | telemetry rollout plan |
+ | Data / pipeline observability | analytics, marketing, BI, or data pipelines need reliability checks | data-health monitoring brief |
+ | Game / live-ops visibility | crashes, player-session health, launch-event stability, backend game services | live-ops observability brief |
+ | Review / gap audit | existing stack exists but trust is low or incidents escaped detection | observability review + gap list |
- const httpRequestTotal = new promClient.Counter({
- name: 'http_requests_total',
- help: 'Total number of HTTP requests',
- labelNames: ['method', 'route', 'status_code']
- });
+ If multiple modes appear, pick the primary bottleneck and list the others as secondary follow-ups.
- // Middleware to track requests
- app.use((req, res, next) => {
- const start = Date.now();
+ ## Instructions
- res.on('finish', () => {
- const duration = (Date.now() - start) / 1000;
- const labels = {
- method: req.method,
- route: req.route?.path || req.path,
- status_code: res.statusCode
- };
+ ### Step 1: Label the surface before choosing tools
+ Capture the minimum facts first.
- httpRequestDuration.observe(labels, duration);
- httpRequestTotal.inc(labels);
- });
+ Record:
+ - system type: web app | backend/API | worker/job | data pipeline | marketing automation | game/live-ops | mixed
+ - environment: local | staging | preview | prod | launch-event | unknown
+ - request type: new setup | review / audit | incident follow-up | migration | platform/tool switch
+ - current evidence: metrics | logs | traces | dashboards | alert rules | incidents | none
+ - user impact shape: latency | errors | stale data | missing events | crashes | unknown
+ - ownership: app team | platform/SRE | data/ops | live-ops | shared | unknown
- next();
- });
+ Do **not** start by asking “Prometheus or Datadog?” Start by labeling the workflow.
- // Metrics endpoint
- app.get('/metrics', async (req, res) => {
- res.set('Content-Type', promClient.register.contentType);
- res.end(await promClient.register.metrics());
- });
+ ### Step 2: Choose the primary observability mode
- app.listen(3000);
- ```
+ #### Mode A — Service reliability
+ Use when the core job is service health, paging, and user-visible behavior.
- **prometheus.yml**:
- ```yaml
- global:
- scrape_interval: 15s
- evaluation_interval: 15s
+ Focus on:
+ - request rate / traffic shape
+ - error rate / failure class
+ - latency distribution / saturation
+ - black-box probes and metamonitoring
+ - SLO / SLI definitions for important journeys
- scrape_configs:
- - job_name: 'my-app'
- static_configs:
- - targets: ['localhost:3000']
- metrics_path: '/metrics'
+ Return:
+ - top 3–5 service questions the dashboard must answer
+ - symptom-first alerts only
+ - what must page now vs ticket later vs dashboard only
+ - missing instrumentation or health endpoints
- - job_name: 'node-exporter'
- static_configs:
- - targets: ['localhost:9100']
+ #### Mode B — Telemetry foundation
+ Use when the team needs to instrument the system, correlate signals, or standardize telemetry.
- alerting:
- alertmanagers:
- - static_configs:
- - targets: ['localhost:9093']
+ Focus on:
+ - traces, metrics, logs, and correlation IDs
+ - event naming / dimensions / labels
+ - instrumentation ownership and rollout slices
+ - retention / sampling / cardinality risk
+ - vendor-neutral export path first, backend second
- rule_files:
- - 'alert_rules.yml'
- ```
+ Return:
+ - telemetry coverage map
+ - required correlation fields (`request_id`, `trace_id`, `job_id`, `user_id`, etc.)
+ - initial rollout order
+ - unsafe telemetry patterns to avoid
- ### Step 2: Alert Rules
+ #### Mode C — Data / pipeline observability
+ Use when freshness, schema drift, volume anomalies, or broken downstream dashboards are the real issue.
- **alert_rules.yml**:
- ```yaml
- groups:
- - name: application_alerts
- interval: 30s
- rules:
- # High error rate
- - alert: HighErrorRate
- expr: |
- (
- sum(rate(http_requests_total{status_code=~"5.."}[5m]))
- /
- sum(rate(http_requests_total[5m]))
- ) > 0.05
- for: 5m
- labels:
- severity: critical
- annotations:
- summary: "High error rate detected"
- description: "Error rate is {{ $value }}% (threshold: 5%)"
+ Focus on:
+ - freshness / lateness
+ - volume / duplicates / drops
+ - schema drift
+ - distribution / null rate / metric anomalies
+ - lineage / downstream blast radius
- # Slow response time
- - alert: SlowResponseTime
- expr: |
- histogram_quantile(0.95,
- sum(rate(http_request_duration_seconds_bucket[5m])) by (le)
- ) > 1
- for: 10m
- labels:
- severity: warning
- annotations:
- summary: "Slow response time"
- description: "95th percentile is {{ $value }}s"
+ Return:
+ - the most important pipeline-health dimensions
+ - likely owners for each alert class
+ - where dashboard trust can silently break
+ - which checks are table-level, job-level, and consumer-level
- # Pod down
- - alert: PodDown
- expr: up{job="my-app"} == 0
- for: 2m
- labels:
- severity: critical
- annotations:
- summary: "Pod is down"
- description: "{{ $labels.instance }} has been down for more than 2 minutes"
+ #### Mode D — Game / live-ops visibility
+ Use when player experience, launch stability, or cross-build crash visibility matters.
- # High memory usage
- - alert: HighMemoryUsage
- expr: |
- (
- node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes
- ) / node_memory_MemTotal_bytes > 0.90
- for: 5m
- labels:
- severity: warning
- annotations:
- summary: "High memory usage"
- description: "Memory usage is {{ $value }}%"
- ```
+ Focus on:
+ - crash reporting and issue grouping
+ - build / version / platform tags
+ - player-session and backend-service health
+ - launch-event alert thresholds
+ - logs / breadcrumbs / device context / purchase-session context where relevant
- ### Step 3: Log Aggregation (Structured Logging)
+ Return:
+ - live-ops event checklist
+ - crash/context fields that must be attached
+ - which signals belong in backend telemetry vs crash tooling
+ - route-out to `game-performance-profiler` if the issue becomes frame-time profiling
- **Winston (Node.js)**:
- ```typescript
- import winston from 'winston';
+ #### Mode E — Review / gap audit
+ Use when dashboards, alerts, or tooling exist but confidence is low.
- const logger = winston.createLogger({
- level: process.env.LOG_LEVEL || 'info',
- format: winston.format.combine(
- winston.format.timestamp(),
- winston.format.errors({ stack: true }),
- winston.format.json()
- ),
- defaultMeta: {
- service: 'my-app',
- environment: process.env.NODE_ENV
- },
- transports: [
- new winston.transports.Console({
- format: winston.format.combine(
- winston.format.colorize(),
- winston.format.simple()
- )
- }),
- new winston.transports.File({
- filename: 'logs/error.log',
- level: 'error'
- }),
- new winston.transports.File({
- filename: 'logs/combined.log'
- })
- ]
- });
+ Focus on:
+ - what incidents would still escape detection
+ - noisy / non-actionable alerts
+ - dashboards with no clear question or owner
+ - missing black-box coverage / metamonitoring
+ - orphan telemetry that nobody uses
- // Usage
- logger.info('User logged in', { userId: '123', ip: '1.2.3.4' });
- logger.error('Database connection failed', { error: err.message, stack: err.stack });
+ Return:
+ - keep / fix / delete / add decisions
+ - top false-positive and false-negative risks
+ - ownership gaps
+ - smallest high-value remediation order
- // Express middleware
- app.use((req, res, next) => {
- logger.info('HTTP Request', {
- method: req.method,
- path: req.path,
- ip: req.ip,
- userAgent: req.get('user-agent')
- });
- next();
- });
- ```
+ ### Step 3: Build the observability brief
+ Return a concise report with this shape:
- ### Step 4: Grafana Dashboard
+ ```markdown
+ # Observability Brief
- **dashboard.json** (example):
- ```json
- {
- "dashboard": {
- "title": "Application Metrics",
- "panels": [
- {
- "title": "Request Rate",
- "type": "graph",
- "targets": [
- {
- "expr": "rate(http_requests_total[5m])",
- "legendFormat": "{{method}} {{route}}"
- }
- ]
- },
- {
- "title": "Error Rate",
- "type": "graph",
- "targets": [
- {
- "expr": "rate(http_requests_total{status_code=~\"5..\"}[5m])",
- "legendFormat": "Errors"
- }
- ]
- },
- {
- "title": "Response Time (p95)",
- "type": "graph",
- "targets": [
- {
- "expr": "histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))"
- }
- ]
- },
- {
- "title": "CPU Usage",
- "type": "gauge",
- "targets": [
- {
- "expr": "rate(process_cpu_seconds_total[5m]) * 100"
- }
- ]
- }
- ]
- }
- }
- ```
+ ## Scope
+ - System type: ...
+ - Environment: ...
+ - Primary mode: ...
+ - Confidence: high | medium | low
- ### Step 5: Health Checks
+ ## Current signal state
+ - What exists already
+ - What is missing or unreliable
- **Advanced Health Check**:
- ```typescript
- interface HealthStatus {
- status: 'healthy' | 'degraded' | 'unhealthy';
- timestamp: string;
- uptime: number;
- checks: {
- database: { status: string; latency?: number; error?: string };
- redis: { status: string; latency?: number };
- externalApi: { status: string; latency?: number };
- };
- }
+ ## Primary questions to answer
+ - 3-5 questions dashboards / alerts must answer
- app.get('/health', async (req, res) => {
- const startTime = Date.now();
- const health: HealthStatus = {
- status: 'healthy',
- timestamp: new Date().toISOString(),
- uptime: process.uptime(),
- checks: {
- database: { status: 'unknown' },
- redis: { status: 'unknown' },
- externalApi: { status: 'unknown' }
- }
- };
+ ## Signal plan
+ - Metrics: ...
+ - Logs: ...
+ - Traces: ...
+ - Black-box / health checks: ...
- // Database check
- try {
- const dbStart = Date.now();
- await db.raw('SELECT 1');
- health.checks.database = {
- status: 'healthy',
- latency: Date.now() - dbStart
- };
- } catch (error) {
- health.status = 'unhealthy';
- health.checks.database = {
- status: 'unhealthy',
- error: error.message
- };
- }
+ ## Alert policy
+ - Page now: ...
+ - Ticket / backlog: ...
+ - Dashboard only: ...
- // Redis check
- try {
- const redisStart = Date.now();
- await redis.ping();
- health.checks.redis = {
- status: 'healthy',
- latency: Date.now() - redisStart
- };
- } catch (error) {
- health.status = 'degraded';
- health.checks.redis = { status: 'unhealthy' };
- }
+ ## Ownership and handoffs
+ - Primary owner: ...
+ - Route-outs: ...
- const statusCode = health.status === 'healthy' ? 200 : health.status === 'degraded' ? 200 : 503;
- res.status(statusCode).json(health);
- });
+ ## First implementation slice
+ - 1-3 smallest high-value changes
```
- ## Output format
-
- ### Monitoring Dashboard Configuration
-
- ```
- Golden Signals:
- 1. Latency (Response Time)
- - P50, P95, P99 percentiles
- - Per API endpoint
+ ### Step 4: Apply symptom-first alert rules
+ Use these defaults unless the evidence says otherwise.
- 2. Traffic (Request Volume)
- - Requests per second
- - Per endpoint, per status code
+ - Alert on **user-visible symptoms** before internal causes
+ - Prefer **one clear page** over many stack-layer pages for the same failure
+ - Include **runbook/dashboard links** in alert context
+ - Add **slack** for brief blips; avoid paging for self-healing noise
+ - Metamonitor the monitoring path itself when alert delivery is mission-critical
- 3. Errors (Error Rate)
- - 5xx error rate
- - 4xx error rate
- - Per error type
+ Good examples:
+ - sustained API error-rate spike affecting user requests
+ - stale data beyond the business tolerance window
+ - launch-event crash rate above threshold by platform/build
+ - message backlog age causing downstream user-visible delay
- 4. Saturation (Resource Utilization)
- - CPU usage
- - Memory usage
- - Disk I/O
- - Network bandwidth
- ```
+ Weak examples:
+ - every 5xx at every layer
+ - every restart with no impact
+ - every metric anomaly without action
+ - alerts nobody can own
- ## Constraints
+ ### Step 5: Pick the right evidence surface
+ Do not recommend one surface for every problem.
- ### Required Rules (MUST)
+ - **Metrics** for rate, latency, capacity, saturation, freshness age, backlog age
+ - **Logs** for context and event details once symptoms are detected
+ - **Traces** for request flow and dependency boundaries
+ - **Black-box checks** for externally visible health
+ - **Dashboards** for ongoing review, not as the only alert mechanism
+ - **Issue/crash tools** for grouped error context, breadcrumbs, and release/build impact
- 1. **Structured Logging**: JSON format logs
- 2. **Metric Labels**: Maintain uniqueness (be careful of high cardinality)
- 3. **Prevent Alert Fatigue**: Only critical alerts
+ If the system only has logs, say so clearly and mark observability maturity as limited.
- ### Prohibited (MUST NOT)
+ ### Step 6: Make route-outs explicit
+ When the job shifts, hand it off.
- 1. **Do Not Log Sensitive Data**: Never log passwords, API keys
- 2. **Excessive Metrics**: Unnecessary metrics waste resources
+ - **“Find the actual failing line in these logs”** → `log-analysis`
+ - **“We already know latency is bad; find the bottleneck”** → `performance-optimization`
+ - **“This looks like a code bug, not a telemetry design issue”** → `debugging`
+ - **“LLM trace quality / evals / prompt observability”** → `langsmith`
+ - **“Unity/Unreal frame-time capture interpretation”** → `game-performance-profiler`
+ - **“Deploy/release plan with post-deploy checks”** → `deployment-automation`
- ## Best practices
+ ## Examples
- 1. **Define SLO**: Clearly define Service Level Objectives
- 2. **Write Runbooks**: Document response procedures per alert
- 3. **Dashboards**: Customize dashboards as needed per team
+ ### Example 1: New API before launch
+ **Prompt:**
+ > We’re launching a new API next week. Tell me what to instrument and what should alert us.
- ## References
+ Use **Mode A — Service reliability** plus a small **Mode B** foundation slice.
+ Return rate/error/latency/saturation questions, minimal tracing fields, black-box checks, and symptom-first alert thresholds.
- - [Prometheus](https://prometheus.io/)
- - [Grafana](https://grafana.com/)
- - [Google SRE Book](https://sre.google/books/)
+ ### Example 2: Marketing dashboard keeps going stale
+ **Prompt:**
+ > Our Monday morning growth dashboard is stale half the time. We need observability, not another manual spreadsheet check.
- ## Metadata
+ Use **Mode C — Data / pipeline observability**.
+ Return freshness/schema/volume/lineage checks, ownership by pipeline/job/dashboard layer, and alert thresholds tied to business tolerance windows.
- ### Version
- - **Current Version**: 1.0.0
- - **Last Updated**: 2025-01-01
- - **Compatible Platforms**: Claude, ChatGPT, Gemini
+ ### Example 3: Game launch-event visibility
+ **Prompt:**
+ > We need crash alerts and player-session visibility for our Unity event weekend, but this isn’t a profiler question yet.
- ### Related Skills
- - [deployment](../deployment/SKILL.md): Monitoring alongside deployment
- - [security](../security/SKILL.md): Security event monitoring
+ Use **Mode D — Game / live-ops visibility**.
+ Return crash/build/platform tags, session-health metrics, launch alert levels, and route-outs to `game-performance-profiler` only if frame-time evidence becomes the bottleneck.
- ### Tags
- `#monitoring` `#observability` `#Prometheus` `#Grafana` `#logging` `#metrics` `#infrastructure`
+ ### Example 4: Boundary check
+ **Prompt:**
+ > Here are the logs from the outage — what’s the root cause?
- ## Examples
+ Do **not** use this as the main workflow. Route to `log-analysis` and say observability improvements can be proposed after the first actionable failure is identified.
- ### Example 1: Basic usage
- <!-- Add example content here -->
+ ## Best practices
+ 1. Start with the workflow shape, not the vendor choice.
+ 2. Prefer symptom-first alerts with low noise.
+ 3. Distinguish instrumentation gaps from incident diagnosis.
+ 4. Treat data/marketing pipeline observability as first-class, not as an afterthought.
+ 5. For games, separate live-ops visibility from engine profiler interpretation.
+ 6. Make owners and handoffs explicit, especially after incidents.
+ 7. Keep dashboards question-driven; delete dead dashboards and dead alerts.
+ 8. Call out missing metamonitoring when alert delivery itself can fail.
- ### Example 2: Advanced usage
- <!-- Add advanced example content here -->
+ ## References
+ - [Google SRE — Monitoring Distributed Systems](https://sre.google/sre-book/monitoring-distributed-systems/)
+ - [OpenTelemetry — Observability primer](https://opentelemetry.io/docs/concepts/observability-primer/)
+ - [Prometheus — Alerting best practices](https://prometheus.io/docs/practices/alerting/)
+ - [Grafana — What is observability?](https://grafana.com/blog/what-is-observability-best-practices-key-metrics-methodologies-and-more/)
+ - [Databricks — What is Data Observability?](https://www.databricks.com/blog/what-is-data-observability)
+ - [Sentry — Game developers](https://sentry.io/solutions/game-developers/)