v1.0.0 to v1.0.0

- description: Grafana dashboard + alert standards — dashboards-as-code via Grizzly, per-service folders, one-question-per-panel design, unified alerting with actionable runbooks, data source provisioning, low-cardinality discipline. Use when designing Grafana dashboards, writing alert rules, organizing folders/permissions, or auditing existing dashboards.

- How an operator *uses* what [observability-architect](../observability-architect/SKILL.md) emits. Dashboards, alerts, data sources, exploration. **Dashboards-as-code via Grizzly** is the default; the Grafana UI is for exploration, not authoring.

- Dashboards live in the application repo (or a sibling `observability/` repo), versioned in git, applied via [Grizzly](https://grafana.github.io/grizzly/) (`grr`).

-

- ```

- observability/

- ├── grizzly.yaml # Grizzly config (Grafana URL, API key from env)

- ├── datasources/

- │ └── prometheus.yaml # data source provisioning

- ├── folders/

- │ └── orders.yaml # folder definition

- └── dashboards/

- └── orders/

- ├── orders-api-overview.json

- ├── orders-database.json

- └── orders-slo.json

- ```

- - **Sync on merge to main.** A GitHub Action runs `grr apply` against staging on every push; against production on tagged releases. Dashboards drift only via PRs that get reviewed.

- - **Never edit in the UI for production dashboards.** Grizzly will refuse to apply if the in-Grafana version is newer (drift detection); resolve by pulling the change into JSON and committing it.

- - **Exploration dashboards are different.** Build them in the UI; once they earn a permanent home, export and commit. The UI is for "is there a signal here?"; the JSON is for "this dashboard is part of how we operate."

- One folder per service. Operators on call find dashboards by what they own, not by which team made them (teams reshuffle; services don't).

-

- ```

- Orders ← service folder

- ├── API overview ← entry-point dashboard for the service

- ├── Database ← specific subsystem

- ├── Queue ← specific subsystem

- ├── SLO ← burn rate + error budget

- └── Runbook links ← links to runbook docs

- ```

- - **Folder permissions match service ownership.** Editors of the `Orders` folder = the team that owns the Orders service.

- - **General dashboards** (infrastructure overview, fleet-wide cost, etc.) live in a top-level `Platform` folder owned by SRE / platform.

- - **No personal folders for production.** Personal exploration is fine in the UI scratch space; production dashboards belong to a service.

- - **Panel title is a question or a noun phrase:** `"Request rate (req/s)"`, `"p95 latency by route"`, `"5xx errors (last 1h)"`. Not `"Stats"`, not `"Metrics"`.

- - **Y-axis unit is mandatory.** Set it explicitly — `seconds`, `bytes`, `req/s`, `percent`. Auto-formatting hides unit confusion.

- - **Time range is consistent across the dashboard.** Don't mix "last 1h" panels with "last 24h" panels unless the dashboard's purpose is the comparison.

- - **Thresholds where they exist.** SLO at 99.9% gets a red line at 99.9% on the panel. Visual context > narrative.

- - **Legends are bounded.** A legend with 50 lines is unreadable; templatize the query (`sum by (route)`) to bound it. If the natural cardinality is high, switch to a heatmap or top-N.

-

- ### Panel types — when each earns its place

- | Type | Use for |

- |---|---|

- | **Time series** | Default. Rates, latencies, gauges over time. |

- | **Stat (single value)** | One number that matters right now: current error rate, in-flight requests, last-deploy timestamp. |

- | **Gauge** | A single value against a known scale (CPU%, error-budget %). Use sparingly — most things are time series. |

- | **Heatmap** | Distribution over time (latency histogram). When line-per-bucket would be unreadable. |

- | **Bar chart / Table** | Top-N: top 10 slow endpoints, top 10 noisy errors. |

- | **Logs panel** | Live tail of structured logs filtered by query. Pairs with traces panel for incident investigation. |

- | **Traces panel** | Recent traces matching a service/operation filter. |

- | **Text / Markdown** | Section headers, runbook links, oncall context. |

- - **Standard variables on every service dashboard:**

- - `service` (the service name, often single-value on a service dashboard)

- - `environment` (`prod`, `staging`)

- - `region` if multi-region

- - **`All` is dangerous** on metrics with many series — selecting "All environments" can fan out into millions of series. Set `Include All option = false` for high-cardinality variables, or set `All` to a curated regex.

- - **Variables must NOT be the only difference between dashboards.** If you have an `Orders Prod` and `Orders Staging` dashboard, those should be one dashboard with an `environment` variable.

- Alerts live alongside dashboards in Grizzly. A panel and its alert rule are version-controlled together.

-

- ```yaml

- # observability/dashboards/orders/orders-slo.alerts.yaml — Grizzly format

- apiVersion: grizzly.grafana.com/v1alpha1

- kind: AlertRuleGroup

- metadata:

- name: orders-slo-burn

- spec:

- folder: Orders

- interval: 1m

- rules:

- - title: "Orders API — fast burn (2% in 1h)"

- condition: B

- data: [...]

- noDataState: NoData

- execErrState: Error

- for: 2m

- labels:

- severity: page

- service: orders

- annotations:

- summary: "Orders API error budget burning at 2%+/hour"

- runbook_url: "https://wiki/runbooks/orders-api-burn"

- ```

- - **Every alert is actionable.** If the response is "I'll look in the morning," it's a dashboard, not an alert. **Pages must matter.**

- - **Two-tier severity:** `severity=page` (wakes someone) and `severity=ticket` (queues for the next business day). No middle ground; clarity matters under pressure.

- - **Multi-window, multi-burn-rate for SLOs:** `fast_burn = (2% in 1h)` AND `slow_burn = (5% in 6h)` together — page only when both fire. Catches real issues without flapping on momentary spikes.

- - **`for:` is mandatory.** A spike that lasts 30s doesn't deserve a page. `for: 2m` minimum on noisy signals; `for: 5m`+ for slow burns.

- - **Annotations include a `runbook_url`.** The first thing on-call clicks is the runbook; if it doesn't exist, the alert is half-built.

- - **Group related alerts** under labels (`team`, `service`) so silencing can target a service without touching others.

- - **CPU > 80%.** Useless without context — is it serving load or saturated? Use saturation metrics (run queue, GC time) or RED (latency/errors) instead.

- - **"Disk full" without `for:`** — `for: 15m` so a temporary spike from log rotation doesn't page.

- - **Provisioned via Grizzly**, not the UI. Datasources `.yaml` files live next to dashboards.

- - **One data source per signal type per environment:** `prometheus-prod`, `prometheus-staging`, `loki-prod`, `tempo-prod`. Dashboards reference by data source name; environment switching is via Grafana data source variables or per-organization scoping.

- - **Admin sparingly.** Two or three admins per Grafana instance — they manage data sources, plugins, and global settings. Service teams don't need it.

- - **Single sign-on (SAML/OIDC) for human access**, service accounts for automation. No shared passwords.

- - **Dashboard sprawl.** Every engineer creates a "my service" dashboard; six versions exist; nobody knows which is current. **Per-service folder owned by the service team** is the structural fix; one canonical dashboard per question.

- - **Unactionable alerts.** Alerts that fire but have no clear response erode trust. Review every page-severity alert quarterly; delete or downgrade what no one actioned.

- - **Dashboards built on raw labels.** Direct PromQL on `request_id` (unbounded cardinality) is slow and wrong. Build on templatized labels per [observability-architect §2](../observability-architect/SKILL.md#2-metrics-prometheus).

- - **Mixing units silently.** A panel showing both `_seconds` and `_milliseconds` lies. Convert at query time (`* 1000`) or set the unit explicitly.

- - **Static-threshold alerts.** A threshold that worked at 100 req/s breaks at 1000. Use SLO-based alerts (burn rate of error budget) for traffic-relative signals.

- - **Edit-in-UI drift.** Production dashboards edited in the Grafana UI bypass review. Detect with `grr diff`; fix by pulling the change into JSON and committing it.

- - **Too many panels per dashboard.** A scroll-forever dashboard means nothing on it gets read. **Soft cap: ~15 panels per dashboard.** Beyond that, split.

- - **Burn rate** — instantaneous burn rate, with fast/slow window comparison (multi-window alert thresholds visible as red lines).

- This dashboard is the single source of truth during an incident. Link to it from the runbook; from the page; from the post-incident review.

- - [observability-architect](../observability-architect/SKILL.md) — produces what Grafana consumes. Naming conventions and cardinality discipline established there must hold for queries here to work.

- - [rest-api-architect](../../protocols/rest-api-architect/SKILL.md) — SLO burn rate alerts reference REST status codes (`5xx` → error budget) and latency (`http_request_duration_seconds`).

- - [improve-codebase-architecture](../../workflows/improve-codebase-architecture/SKILL.md) — friction visible in dashboards (slow endpoints, error hot spots) is input to architecture review.