alert-tuning git:20261005.bea4ea5A

Immutable. This exact content never changes and is served at /api/v1/blob/a97fbfd69e78cac0.

---
name: alert-tuning
description: Reduces alert noise by making alerts actionable, symptom-based, and tied to SLOs. Identifies and kills flappy, duplicate, and cause-based alerts. Use when alert fatigue is degrading on-call response quality.
metadata:
  title: "Alert Tuning"
---

# Alert Tuning

Every alert that fires without requiring human action trains engineers to ignore alerts. Alert fatigue kills real incidents. The goal: every page wakes someone up for a reason they cannot automate away.

## The Audit - Run First

Before writing new alert rules, audit what you have.

1. Pull the last 30 days of alert history. Calculate: total pages, pages outside business hours, pages that resulted in no action taken.
2. Identify flappy alerts: any alert that fires and resolves more than 3 times in a 24-hour window without a corresponding incident.
3. Identify cause-based alerts: anything that fires on a system-internal metric (queue depth, pod restart count, CPU) without a direct user-impact correlation.
4. Calculate the actionability rate: (pages with a documented remediation / total pages). Below 80% means your alert set is broken.

## Symptom vs Cause

Page on symptoms, not causes. Symptoms are what users experience. Causes are what engineers investigate.

- Wrong: alert on CPU above 80%.
- Right: alert on p99 latency above SLO threshold.
- Wrong: alert on pod restart count above 2.
- Right: alert on error rate above 1% for 5 minutes.

For every cause-based alert, ask: 'Would a user notice if this fired and nobody acted?' If no, delete it. If yes, rewrite it as the symptom the user would notice.

## SLO-Based Alerting

Tie alerts to error budget burn rate, not raw thresholds.

- Fast burn: burn rate above 14x for 5 minutes (consumes 1 hour of 30-day budget). Page immediately.
- Slow burn: burn rate above 1x for 1 hour. Ticket or warn.
- No burn: no page, regardless of cause-based signals.

This approach eliminates most threshold-tuning debates and makes alert severity self-documenting.

## Fixing Flappy Alerts

- Add a 'for' duration to the alert rule. A threshold must be sustained for at least 2-5 minutes before paging.
- Add hysteresis: alert fires at 95% threshold, resolves only when it drops below 85%.
- For bursty metrics, use a rate or average over a longer window (10-15 minutes) instead of instantaneous value.

## Routing and Severity

- Route by SLO impact, not by which team owns the metric. The team that can fix it fastest gets the page.
- SEV1/2-level alerts: immediate page.
- SEV3-level alerts: ticket, notify via Slack, no wake-up.
- Never route the same alert to more than one person simultaneously. Shared ownership is no ownership.

## Do / Don't

- Do: delete alerts you cannot improve rather than leaving them muted. A muted alert is a lie.
- Do: review alert history quarterly. Thresholds drift as systems scale.
- Don't: add a new alert without deleting or merging one. Alert sets grow monotonically unless you enforce a budget.
- Don't: alert on absence of data without accounting for scrape intervals. Missing data is not always an outage.