Immutable. This exact content is served forever at /api/v1/blob/6bccfc7a5d815595.
--- name: sre-runbooks description: >- Safe-by-default DevOps/SRE runbook automation for incident response, postmortems, on-call handovers, and operational troubleshooting. Implements Google SRE principles with agent-safe execution patterns including dry-run modes, human approval gates, and blast-radius limits. version: 1.0.0 platforms: [openclaw, claude, codex, cursor, gemini, copilot, opencode, windsurf] author: name: Skill Foundry (Forge) source: Google SRE Book, bregman-arie/devops-sre-skills, Pulumi DevOps Skills license: MIT risk_tier: L2 tags: [devops, sre, incident-response, runbook, postmortem, on-call, troubleshooting] requires: binaries: [] --- # SRE Runbooks Production-safe DevOps and SRE runbook automation. Execute incident response procedures, draft postmortems, generate on-call handovers, and troubleshoot production issues — all with built-in safety gates that prevent the agent from making destructive changes without human approval. ## When to Use This Skill Use this skill when: - Responding to a production incident (alert fired, service degraded) - Writing or updating a runbook for a service - Drafting a postmortem after an incident - Preparing on-call handover notes - Troubleshooting a deployment failure or pipeline issue - Performing a root cause analysis (RCA) - Any request like "investigate this alert", "write a postmortem", "create runbook for X", "prepare handover notes" ## Safety Model This skill is **risk tier L2 (elevated)**. Every automated action follows these safety rules: ### Execution Gates | Gate | Description | |------|-------------| | **Read-only first** | All investigations start read-only; writes require explicit escalation | | **Dry-run by default** | Destructive commands print what they *would* do before execution | | **Blast-radius check** | Before acting, compute and report the scope of impact | | **Human approval** | Any change to production state requires human confirmation | | **Rollback plan** | Every change proposal includes a verified rollback path | | **Audit log** | Every action is logged with timestamp, identity, and justification | ### Never-Automate List These actions require a human in the loop, no exceptions: - `kubectl delete` on running workloads - `terraform destroy` or `terraform apply -auto-approve` - Database DROP, TRUNCATE, or schema-destructive migrations - DNS record deletion or apex domain changes - IAM policy or RBAC role removal - Secrets rotation without backup verification - Firewall rule removal on production traffic paths ## Incident Response Workflow ### Phase 1: Triage (Read-Only) When an alert fires, the agent: 1. **Acknowledges the alert** in the incident management system 2. **Gathers context** — recent deployments, config changes, metrics 3. **Identifies the blast radius** — affected services, users, regions 4. **Checks for known patterns** in the incident database 5. **Declares severity** based on SLO impact ``` SEVERITY ASSESSMENT: ├── SEV0: User-visible outage, SLO breached → Page on-call ├── SEV1: Degraded but available, SLO at risk → Alert on-call ├── SEV2: Non-critical, SLO not threatened → Ticket └── SEV3: Informational, no user impact → Log only ``` ### Phase 2: Investigation The agent systematically works through: 1. **The Four Golden Signals** (Google SRE): - Latency: Is response time elevated? - Traffic: Is request rate anomalous? - Errors: Is error rate above threshold? - Saturation: Is any resource exhausted? 2. **The Five Whys** — progressive root cause drilling: - Why did the alert fire? → Error rate spiked - Why did errors spike? → Timeouts from auth service - Why auth service timing out? → Connection pool exhausted - Why pool exhausted? → New deployment changed pool size - Why was pool size changed? → Config drift in deployment template 3. **The Differential Diagnosis** — rule out common causes: - Recent deployment? Check deploy log - Config change? Check config history - Dependency issue? Check upstream health - Capacity issue? Check resource metrics - Network issue? Check connectivity between services ### Phase 3: Mitigation Execute mitigation steps with **human approval at each gate**: 1. **Contain** — Stop the bleeding (rate-limit, circuit-break, shed load) 2. **Mitigate** — Restore service (rollback, scale up, failover) 3. **Verify** — Confirm recovery (check SLOs, run health checks) 4. **Communicate** — Update status page and stakeholders ### Phase 4: Resolution After the incident is resolved: 1. **Verify full recovery** — all SLOs green for 15+ minutes 2. **Document timeline** — timestamped actions and decisions 3. **Create follow-up tickets** — prevent recurrence 4. **Archive incident artifacts** — logs, graphs, chat transcripts ## Postmortem Template Generate blameless postmortems following Google's template: ```markdown # Postmortem: [Incident Title] **Date:** YYYY-MM-DD **Severity:** SEV0/1/2 **Duration:** Xh Ym (HH:MM UTC to HH:MM UTC) **Authors:** [Names] **Status:** Draft / Review / Final ## Summary [One paragraph — what happened, impact, duration] ## Timeline (UTC) | Time | Event | |------|-------| | 14:32 | Alert fired: error rate >5% on api-gateway | | 14:33 | On-call acknowledged | | 14:38 | Identified: connection pool exhaustion | | 14:42 | Rolled back deployment v2.4.1 → v2.4.0 | | 14:47 | Error rate normalized; SLO recovered | ## Root Cause [Technical explanation — what failed and why] ## Impact - Users affected: [count or %] - Revenue impact: [$ or N/A] - SLO impact: [which SLO, how much burned] ## Detection - How was it detected? (alert, user report, partner) - Time to detect: X minutes - Could detection have been faster? ## Resolution [Steps taken to resolve — be specific] ## Action Items | # | Action | Owner | Priority | Due | |---|--------|-------|----------|-----| | 1 | Fix connection pool default | @engineer | P0 | EOW | | 2 | Add alert on pool saturation | @sre | P1 | Sprint | | 3 | Update deployment checklist | @team | P2 | Month | ## Lessons Learned - What went well? - What went poorly? - Where did we get lucky? ``` ## Runbook Template ```markdown # Runbook: [Service Name] — [Failure Mode] **Owner:** [Team] **Last Updated:** YYYY-MM-DD **Severity:** [Expected severity when this runbook is needed] ## Symptoms - [Alert name(s) that fire] - [Observable symptoms — metrics, logs, user reports] ## Prerequisites - [Access needed: VPN, jump host, cluster, dashboard URLs] - [Credentials: which tokens/roles, NOT actual values] ## Investigation Steps 1. [Check dashboard X for metric Y] 2. [Run query: SELECT ... FROM ... WHERE ...] 3. [Check recent deployments: kubectl rollout history ...] 4. [...] ## Mitigation Steps ### Quick Fix (stop the bleeding) 1. [Action] → Expected result: [what you should see] 2. [Action] → Expected result: [...] ### Full Resolution 1. [Action with rollback command] 2. [...] ## Verification - [ ] [Check 1] - [ ] [Check 2] - [ ] [SLO dashboard shows recovery] ## Escalation - If [condition]: escalate to [team/on-call rotation] - If unresolved after [N] minutes: page [escalation contact] ``` ## On-Call Handover Template ```markdown # On-Call Handover — YYYY-MM-DD **From:** [Outgoing] **To:** [Incoming] **Shift:** [Hours covered] ## Active Incidents | ID | Title | Severity | Status | Action needed | |----|-------|----------|--------|---------------| | INC-123 | API latency | SEV2 | Mitigating | Monitor for 24h | ## Known Issues (Watch List) - [Service X] has been flapping on deploys — may need rollback - [Alert Y] is noisy — ticket filed to tune threshold ## Upcoming Changes - [Service A] deployment scheduled Tue 10:00 UTC - [Infra B] maintenance window Thu 02:00-04:00 UTC ## Open Questions - [Question that needs follow-up] ``` ## Platform Notes - **All platforms:** This skill provides procedural knowledge — no binary dependencies required. The agent applies the patterns using its existing tools (shell, kubectl, terraform, monitoring APIs). - **Safety:** The risk-tier L2 designation ensures agents pause for human approval before any destructive action. Platform runtimes should enforce this through their native approval mechanisms. - **OpenClaw:** Compatible with native approval gates and elevated-tool controls.