sre · diff
v2.1.0 to v2.1.0
64 added, 85 removed. Audit A to A.
---
name: sre
- description: Use when defining or auditing SLOs/SLIs/error budgets, designing production observability (metrics, alerting, dashboards), leading incident response or postmortems, reviewing production readiness, reducing operational toil, designing infrastructure as code, planning capacity, or running chaos experiments. Distinct from devex — this role owns production reliability and the outer loop (deploy → monitor → incident → postmortem), not developer productivity tools or CI/CD pipeline speed.
+ description: Use when defining or auditing SLOs/SLIs/error budgets, designing production observability (metrics, alerting, dashboards), reviewing production readiness, reducing operational toil, writing runbooks or postmortems, designing infrastructure as code, or planning chaos experiments. Distinct from devex — this skill covers production reliability and the outer loop (deploy → monitor → incident → postmortem), not developer productivity tools or CI/CD pipeline speed.
version: 2.1.0
---
- # Site Reliability Engineer
+ # Site Reliability Engineering
## Iron Law
```
- 70% of outages are caused by changes. Make every change incremental, observable, and reversible.
- Alert on symptoms, not causes — every page that doesn't require immediate human action is a bug.
- MTTR beats MTBF: optimise for fast recovery, not for preventing all failures.
+ 70% of outages are caused by changes — make every change incremental, observable, and reversible.
+ Alert on symptoms, not causes — every page that does not require immediate human action is a bug.
+ MTTR beats MTBF — optimise for fast recovery, not for preventing all failures.
```
---
## Before Taking Any Action
- 1. **Announce** what you intend to create or change and why — SLO proposal, observability design, runbook, infrastructure change, chaos experiment
- 2. **Confirm scope** — which service, which environment, what are the existing SLOs if any
+ 1. **Announce** what you intend to produce — SLO proposal, alert rules, runbook, IaC, postmortem, PRR report, chaos experiment design
+ 2. **Confirm scope** — which service, which environment, existing SLOs if any
3. **Ask for confirmation** before writing files, running commands, or proposing infrastructure changes
- 4. **Report** what was produced, what decisions were made, and what still needs human review (SLO targets need stakeholder sign-off; chaos experiments need a maintenance window)
+ 4. **Report** what was produced, what decisions were embedded, and what still needs human review (SLO targets need stakeholder sign-off; chaos experiments need a maintenance window)
---
- ## Your Workflow
+ ## Task Approach
- 1. **Establish the reliability baseline.** What are the current SLIs? What is the error budget status? Is there an on-call rotation? What was the last postmortem? You cannot improve what you haven't measured.
- 2. **Define or audit SLOs.** Are the targets grounded in user expectations, not just current performance? Are they using percentiles? Are internal targets tighter than external SLAs?
- 3. **Assess observability.** Are the Four Golden Signals instrumented? Are dashboards readable under pressure? Are alerts paging on symptoms, not causes? Is there distributed tracing for microservices?
- 4. **Review on-call health.** Is incident volume within the two-events-per-shift target? Are runbooks written and tested? Are postmortems blameless and published?
- 5. **Identify and eliminate toil.** What recurring manual tasks consume time? Which can be automated to eliminate an entire class, not just one instance?
- 6. **Run a Production Readiness Review** for new services or services undergoing significant change.
- 7. **Plan chaos experiments.** What resilience claims have never been validated? Design a game day or chaos injection to test them in a controlled environment.
+ Use this table to determine what to produce for each task type:
+ | User asks for | What to produce |
+ |---|---|
+ | SLO / SLI definition | SLI selection per service type (see framework below) + SLO target + error budget calculation + error budget policy table |
+ | Observability design | Four Golden Signals instrument checklist + Prometheus metrics + Grafana dashboard JSON or dashboard-as-code spec + alert rules |
+ | Alert rules | Burn rate alerts (fast + slow) per SLO using the two-alert model; include runbook link placeholders |
+ | Production readiness review | PRR report structured against the six-dimension checklist; flag each dimension as Pass / At Risk / Fail with remediation steps |
+ | Runbook | Step-by-step response procedure for a specific alert or failure mode; include detection, diagnosis steps, mitigation commands, escalation path, and rollback |
+ | Postmortem | Completed postmortem document using the blameless template; ask for the incident timeline and impact before writing |
+ | Toil identification | Toil audit: list recurring manual tasks, classify each against the toil criteria, propose automation approach and elimination priority |
+ | Infrastructure as code | Terraform modules or Kubernetes manifests following the IaC principles below; always include rollback plan as a comment block |
+ | Chaos experiment design | Hypothesis statement + blast radius definition + success/failure criteria + kill switch + measurement plan; do not generate commands that inject failure without explicit user confirmation |
+
---
## SLO/SLI/Error Budget Framework
### Definitions
| Term | Definition |
|---|---|
- | **SLI** (Service Level Indicator) | A quantitative measure of service behaviour (e.g. % of requests with latency < 200ms) |
- | **SLO** (Service Level Objective) | A target for the SLI over a time window (e.g. 99.9% of requests < 200ms over 30 days) |
+ | **SLI** | A quantitative measure of service behaviour (e.g. % of requests with latency < 200ms) |
+ | **SLO** | A target for the SLI over a time window (e.g. 99.9% of requests < 200ms over 30 days) |
| **Error Budget** | `1 − SLO target` — the allowed failure rate (e.g. 0.1% of requests may fail the SLI) |
- | **SLA** (Service Level Agreement) | An SLO with contractual consequences (penalties, credits) |
+ | **SLA** | An SLO with contractual consequences (penalties, credits) |
- ### SLI Design by Service Type
+ ### SLI Selection by Service Type
| Service type | Recommended SLIs |
|---|---|
| Request/response (APIs) | % of requests with latency < threshold (p99); % of successful responses (non-5xx) |
| Batch / pipeline | Throughput (records/s); end-to-end latency; % of batches completing within SLA |
| Storage | Durability (data loss rate); read latency p99; write availability |
| Scheduled jobs | % of jobs completing within time budget; error rate |
**Use percentiles (p99, p95), never averages.** An average hides the tail — users on p99 latency are real users having a bad experience.
### Error Budget Calculation
```
Error Budget = (1 − SLO%) × Time Window
Example: 99.9% SLO over 30 days
Error Budget = 0.1% × 30 × 24 × 60 = 43.2 minutes
```
### Error Budget Policy
| Budget remaining | Action |
|---|---|
- | > 50% | Full velocity; deploy freely; new feature work |
+ | > 50% | Full velocity; deploy freely |
| 25–50% | Normal operations; monitor burn rate |
- | 10–25% | Elevated caution; no risky deployments without SRE sign-off |
+ | 10–25% | Elevated caution; no risky deployments without SRE review |
| < 10% | Reliability sprint; feature freeze; postmortem required for any incident |
- | 0% (budget exhausted) | Feature freeze until budget resets; escalate to engineering leadership |
+ | 0% exhausted | Feature freeze until budget resets; escalate to engineering leadership |
---
- ## Four Golden Signals (Goldenberg, Google SRE Book)
+ ## Four Golden Signals
Build dashboards starting with these four. If all four are healthy, the service is probably healthy.
- | Signal | What it measures | Recommended alert condition |
+ | Signal | What it measures | Alert condition |
|---|---|---|
| **Latency** | Response time for successful and failed requests | p99 > SLO threshold for > 5 minutes |
- | **Traffic** | Request rate (RPS, events/s, QPS) | Anomalous spike or drop from baseline (burn rate alert) |
- | **Errors** | Rate of failed requests (4xx client errors, 5xx server errors) | Error rate > SLO-derived burn rate |
- | **Saturation** | Resource utilisation: CPU, memory, disk, connection pool | > 80% sustained for > 10 minutes |
+ | **Traffic** | Request rate (RPS, events/s, QPS) | Anomalous spike or drop from baseline |
+ | **Errors** | Rate of failed requests (4xx client, 5xx server) | Error rate > SLO-derived burn rate |
+ | **Saturation** | CPU, memory, disk, connection pool utilisation | > 80% sustained for > 10 minutes |
Alert on symptoms (p99 latency exceeds SLO threshold), not causes (CPU > 80%). Causes are investigated; symptoms are paged.
---
## Alert Design Rules
- - **Every page requires immediate human action.** A page that does not require immediate action is a bug — it trains engineers to ignore alerts.
- - **Alert on burn rate, not absolute values.** A 1-hour burn rate of 14.4× will exhaust a 30-day 99.9% budget in 5 days. Page on the burn rate.
+ - **Every page requires immediate human action.** A page that does not is a bug — it trains engineers to ignore alerts.
+ - **Alert on burn rate, not absolute values.** A 1-hour burn rate of 14.4× will exhaust a 30-day 99.9% budget in 5 days.
- **Two-alert model per SLO:**
- Fast burn: 6-hour window, 5% budget consumed → page immediately
- Slow burn: 3-day window, 10% budget consumed → ticket/warning
- - **Alert fatigue is a failure mode.** Review alert frequency monthly. Any alert firing > 5 times per week without action is a toil candidate.
+ - **Alert fatigue is a failure mode.** Any alert firing > 5 times per week without action is a toil candidate.
- **Runbook link in every alert.** Engineers under pressure must not have to remember what to do.
---
## Production Readiness Review (PRR) Checklist
- Run before a new service gets SRE on-call coverage or goes to production:
-
- 1. **System architecture** — documented service dependencies; failure modes for each dependency identified; no single points of failure without mitigation
- 2. **SLI instrumentation** — Four Golden Signals instrumented; dashboards deployed; alerts configured and tested
- 3. **Emergency response** — runbook written and reviewed; at least one engineer has practiced the incident response flow
- 4. **Capacity planning** — load tested to 120% of expected peak; N+2 provisioned (planned + unplanned loss)
- 5. **Change management** — canary or blue/green deployment configured; rollback tested and documented; deployment takes < 30 minutes
- 6. **Performance targets** — SLOs defined, agreed with the product team, and baselined against load test results
-
- A service that fails PRR is not blocked from deployment — it is blocked from SRE on-call coverage. The engineering team retains on-call responsibility until PRR passes.
-
- ---
-
- ## Incident Response
-
- ### Incident Roles
+ Run before a service goes to production or receives on-call coverage:
- | Role | Responsibility |
+ | Dimension | Pass criteria |
|---|---|
- | **Incident Commander (IC)** | Owns the incident; coordinates response; decides when to escalate; declares resolved |
- | **Communications Lead** | External and internal status updates; customer communication |
- | **Scribe** | Timeline, decisions made, actions taken |
- | **Subject Matter Experts** | Diagnose and fix; work under IC direction |
-
- ### Incident Flow
-
- 1. Page fires → IC assigned within 5 minutes
- 2. Dedicated Slack channel created (`#incident-YYYY-MM-DD-[short-description]`)
- 3. Impact assessed: number of users affected, severity level (P0/P1/P2)
- 4. Mitigation timeline: restore service first (rollback, kill switch, redirect traffic); root cause analysis second
- 5. All-clear declared by IC; postmortem scheduled within 48 hours
- 6. Postmortem published within 5 business days
-
- ### On-Call Health Targets (Google SRE Book)
+ | **System architecture** | Dependencies documented; failure modes identified; no unmitigated single points of failure |
+ | **SLI instrumentation** | Four Golden Signals instrumented; dashboards deployed; alerts configured and tested |
+ | **Emergency response** | Runbook written and reviewed; incident response flow practiced by at least one engineer |
+ | **Capacity planning** | Load tested to 120% of expected peak; N+2 provisioned |
+ | **Change management** | Canary or blue/green configured; rollback tested and documented; deployment < 30 minutes |
+ | **Performance targets** | SLOs defined, agreed with product team, and baselined against load test results |
- - Maximum 2 incident events per 8–12 hour on-call shift
- - Minimum 8 people per on-call rotation
- - On-call across 2+ geographic locations to avoid nocturnal pages for any one person
- - Post-incident rest: if an engineer is paged overnight, reduce their next day load
+ A service that fails PRR is not blocked from deployment — it is blocked from receiving on-call coverage. The engineering team retains on-call responsibility until PRR passes.
---
## Postmortem Template (Blameless)
```markdown
## Incident Summary
**Date**: [Date]
**Duration**: [Start time → End time]
**Severity**: [P0 / P1 / P2]
- **Incident Commander**: [Name]
## Impact
- Users affected: [number / segment]
- Features affected: [what stopped working]
- Revenue / SLA impact: [if applicable]
## Timeline (all times UTC)
| Time | Event |
|---|---|
- | HH:MM | [What happened] |
- | HH:MM | [Alert fired / IC assigned / mitigation attempted / resolved] |
+ | HH:MM | [Alert fired / mitigation attempted / resolved] |
## Root Cause
[Avoid "human error" as a root cause — it is always a system design issue.
Describe the systemic conditions that allowed the incident to occur.]
## Contributing Factors
- [Multiple factors — never a single root cause]
- [Factor 1]
- [Factor 2]
## What Went Well
- [Things that helped contain or resolve the incident]
## What Went Poorly
- [Things that made the incident worse or harder to resolve]
## Action Items
| Action | Owner | Due Date | Status |
|---|---|---|---|
| [Specific improvement] | [Name] | [Date] | Open |
```
- **Blameless principle:** a person made a decision that made sense given the information and tools available to them at the time. The system allowed that decision to cause harm. Fix the system.
+ **Blameless principle:** a person made a decision that made sense given the information and tools available at the time. The system allowed that decision to cause harm. Fix the system.
---
- ## Toil Identification and Reduction
-
- **Toil definition** (Google SRE Book): work that is manual, repetitive, automatable, interrupt-driven, and produces no enduring value beyond the immediate action.
+ ## Toil Identification
- **Hard rule: ≤ 50% of SRE time can be toil.** Above this, the team loses the capacity to make systemic improvements.
+ **Toil** (Google SRE Book): work that is manual, repetitive, automatable, interrupt-driven, and produces no enduring value beyond the immediate action.
| Toil candidate | Automation approach |
|---|---|
- | Manual credential rotation | Automated rotation on schedule + Vault integration |
+ | Manual credential rotation | Automated rotation on schedule + secrets manager integration |
| Restarting a service when it OOMs | HPA autoscaling + memory limits + alert on degraded pod |
- | Recurring alert → same fix | Runbook → then automate the fix → then eliminate the cause |
- | Manual scaling before a known traffic event | Scheduled scaling policy |
+ | Recurring alert → same fix every time | Runbook → automate the fix → eliminate the root cause |
+ | Manual scaling before known traffic events | Scheduled scaling policy |
| Answering "what's the status of X?" | Status page + automated incident status updates |
+ **Hard rule:** if toil exceeds 50% of available engineering time, the team loses capacity for systemic improvements. Prioritise elimination of the highest-frequency items first.
+
---
## Infrastructure as Code Principles
- All infrastructure changes are version-controlled, reviewed, and applied through CI/CD — never via console or SSH
- Terraform: one state file per environment; remote state with locking (S3 + DynamoDB or Terraform Cloud)
- - Kubernetes: resource requests and limits set on every container; `PodDisruptionBudget` for every stateful workload
+ - Kubernetes: resource requests and limits on every container; `PodDisruptionBudget` for every stateful workload
- Every infrastructure change has a rollback plan documented before it is applied to production
- Secrets are never in Terraform state — use Vault or AWS Secrets Manager with references, not values
---
## Chaos Engineering Principles
- - Start with a hypothesis: "The service will continue to serve requests if pod X is killed"
- - Define the blast radius before injecting failure — scope to a test environment or a small percentage of traffic
- - Have a kill switch ready before the experiment starts
- - Measure the system's actual response against the hypothesis
- - Game days > automated chaos for teams new to the practice — the learning comes from the exercise, not just the data
+ When designing a chaos experiment, always produce these four elements before any execution:
+
+ 1. **Hypothesis** — "The service will continue to serve requests at SLO if [failure condition]"
+ 2. **Blast radius** — scope to a test environment or a small percentage of traffic; define the kill switch
+ 3. **Measurement plan** — which SLIs to observe; what constitutes pass vs fail
+ 4. **Rollback trigger** — the condition at which the experiment is aborted immediately
+
+ Do not generate commands that inject failure into production without explicit user confirmation and a defined maintenance window.