incident-runbook · diff
git:20260713.51dc745 to git:20260730.0bf7183
5 added, 1 removed. Audit A to A.
---
name: incident-runbook
description: |
Production incident response: diagnose → mitigate → resolve, then a blameless postmortem and a repeatable
runbook. Stop the impact first, root cause second.
- Trigger phrases: "incident", "incident response", "runbook", "postmortem", "root cause", "outage", "production incident", "post-incident"
---
# Incident Response & Runbook
+
+ <!-- routing-eval reads this line; it lives in the BODY so the always-on skill LISTING stays inside
+ Claude Code's budget (1% of the context window) — an overflowing listing gets descriptions
+ truncated or dropped, which strips the very keywords a match depends on. -->
+ Trigger phrases: "incident", "incident response", "runbook", "postmortem", "root cause", "outage", "production incident", "post-incident"
Two modes: **live incident** (what to do right now) and **aftermath** (postmortem + runbook). Priority: stopping
user impact > finding the root cause. No panic, one ordered step at a time.
## Live incident — sequence
1. **Acknowledge & classify** — what is the impact (who, how much), severity (SEV1 full outage … SEV3 minor).
2. **Mitigate the impact FIRST** — rollback, turn off a feature flag, shift traffic, scale up. Without waiting on the root cause.
3. **Single coordinator** — it is clear who decides; communication goes through one channel.
4. **Diagnose** — last change? (deploy/migration/config) narrow it down with logs+metrics+traces (observability).
5. **Resolve** — the smallest safe fix; then verify (health check).
6. **Close** — confirm the impact is over; note the timeline (a postmortem input).
## Mitigation reflexes
- Last deploy suspect → **rollback** (vps-deploy revert).
- Suspect feature → turn off the **feature flag**.
- After a destructive migration → restore from backup (db-migration).
- Dependency/service down → circuit breaker / graceful degradation.
## After the incident
Blameless postmortem + producing a durable runbook: **`references/postmortem.md`**.
## Invariant rules
1. **Stop the impact, then understand** — the root cause does not hold up the resolution.
2. **Blameless culture** — the postmortem questions the system, not the person.
3. **Actions are owned + dated** — no "we'll look at it later".
4. **The runbook is executable** — real commands/steps, not wishes.
5. **Make learning permanent** — the lesson goes into an adr/runbook/monitoring, it does not get lost.