---
name: promql-generator
description: Generate PromQL queries for calculating error rates, aggregating metrics across labels, creating histogram percentiles, writing recording rules, and building SLO burn-rate alerts following Prometheus best practices. Use when creating new PromQL queries, implementing monitoring and alerting rules, building observability dashboards, working with Prometheus metrics (counters, gauges, histograms, summaries), or applying RED (Rate, Errors, Duration) and USE (Utilization, Saturation, Errors) monitoring patterns.
---

# PromQL Query Generator

## When to Use This Skill

Use this skill when you need to:

- Write a PromQL query for error rates, latency percentiles, or availability.
- Author recording rules or SLO burn-rate alerting rules.
- Aggregate metrics across labels using RED or USE monitoring patterns.
- Build queries for Prometheus counters, gauges, histograms, or summaries.
- Draft Kubernetes or availability queries from the patterns in `assets/`.

## When Not to Use

- Validating or optimising an existing query: use `promql-validator` instead.
- Non-Prometheus query languages (LogQL, MetricsQL dialect specifics, SQL): use the matching tool.
- Configuring Prometheus scrape targets or storage: that is server config, not query authoring.
- Diagnosing why a metric is missing from Prometheus: check `up{job="..."}` and the scrape config instead.

## Philosophy

- **Match the function to the metric type.** rate() is for counters, histogram_quantile() for histograms; applying rate() to a gauge is meaningless.
- **Error rate is a ratio.** Express it as rated errors over rated totals across the same window, not as raw counts.
- **Control cardinality on purpose.** Choose by()/without() labels deliberately so a query cannot explode a dashboard.
- **Alert on sustained burn.** SLO alerts use multi-window burn rates and a for clause, not a single instantaneous threshold.
- **Plan before you write.** Confirm goal, metric type and parameters with the user before generating any query.

## Interactive Query Planning Workflow

**CRITICAL**: Always engage the user in collaborative planning **before** generating any query. Never skip the planning phase.

### Workflow (7 stages)

1. **Understand the goal**: Ask what the user wants to monitor (request rate, error rate, latency, resource usage, availability, SLO tracking) and the use case (dashboard, alert, recording rule, ad-hoc).
2. **Identify metrics**: Confirm metric names, types (counter/gauge/histogram/summary), and relevant labels. Suggest common naming patterns if uncertain.
3. **Determine parameters**: Confirm time range, label filters, aggregation, and thresholds. If the user already specified values (e.g., "5-minute window", "> 5% error rate"), acknowledge them as pre-filled defaults and allow quick confirmation rather than re-asking.
4. **Present the query plan**: Before writing any code, present a plain-English plan (goal, query structure, expected output, example interpretation) and ask for confirmation and offer the options: "Yes, generate this query" / "Modify [aspect]" / "Show alternatives".
5. **Generate the query**: Once confirmed, read the relevant reference file(s) before writing code, cite the applicable pattern, and apply the best practices below.
6. **Validate**: Automatically invoke `promql-validator`. Display structured results (syntax, best practices, explanation). Fix any issues and re-validate until all checks pass.
7. **Deliver**: Provide the final query, plain-English explanation, usage instructions (dashboard / alert / recording rule), customization notes, and related query suggestions.

> **Ask vs. Infer**: If the user's request already clearly specifies goal, use case, and context, acknowledge those details instead of re-asking. Only ask for missing or ambiguous information.

---

## Best Practices for Query Generation

**Always consult the relevant reference file before writing code.**

| Scenario | Reference File |
|---|---|
| Histogram queries | `references/metric_types.md` (Histogram section) |
| Error/latency patterns | `references/promql_patterns.md` (RED section) |
| Resource monitoring | `references/promql_patterns.md` (USE section) |
| Optimization / anti-patterns | `references/best_practices.md` |
| Specific functions | `references/promql_functions.md` |

### Key Rules

1. **Always add label filters**: reduces cardinality and improves performance.
2. **Match functions to metric types**: `rate()`/`increase()` on counters; `*_over_time()` or direct use for gauges; `histogram_quantile()` for histograms.
3. **Prefer `by()`/`without()`** on all aggregations.
4. **Prefer exact label matches** over regex when the value is known.
5. **Use recording rules** for queries that are expensive or reused frequently (naming: `level:metric:operations`).
6. **Format multi-line** for complex queries.

### Core Patterns

```promql
# Request rate (counter)
sum(rate(http_requests_total{job="api-server"}[5m])) by (endpoint)

# Error rate ratio
sum(rate(http_requests_total{job="api-server", status_code=~"5.."}[5m]))
/ sum(rate(http_requests_total{job="api-server"}[5m]))

# P95 latency (classic histogram)
histogram_quantile(0.95,
  sum by (le) (rate(http_request_duration_seconds_bucket{job="api-server"}[5m]))
)

# P95 latency (native histogram, Prometheus 3.x+)
histogram_quantile(0.95,
  sum by (job) (rate(http_request_duration_seconds[5m]))
)

# Availability
(count(up{job="api-server"} == 1) / count(up{job="api-server"})) * 100

# Burn rate (99.9% SLO, 1h window)
(
  sum(rate(http_requests_total{job="api", status_code=~"5.."}[1h]))
  / sum(rate(http_requests_total{job="api"}[1h]))
) / 0.001

# Multi-window burn-rate alert (page: 2% budget in 1h, burn rate 14.4)
(
  sum(rate(http_requests_total{job="api", status_code=~"5.."}[1h]))
  / sum(rate(http_requests_total{job="api"}[1h]))
) > 14.4 * 0.001
and
(
  sum(rate(http_requests_total{job="api", status_code=~"5.."}[5m]))
  / sum(rate(http_requests_total{job="api"}[5m]))
) > 14.4 * 0.001
```

For complete SLO patterns, Native Histogram functions (`histogram_count`, `histogram_sum`, `histogram_fraction`), subqueries, offset/@ modifiers, vector matching, and Kubernetes patterns: see the `assets/` files.

---

## Validation Checklist

After generating, invoke `promql-validator` and display results in this format:

```
## PromQL Validation Results

### Syntax Check
- Status: ✅ VALID / ⚠️ WARNING / ❌ ERROR
- Issues: [list any syntax errors]

### Best Practices Check
- Status: ✅ OPTIMIZED / ⚠️ CAN BE IMPROVED / ❌ HAS ISSUES
- Issues: [list problems found]
- Suggestions: [list optimizations]

### Query Explanation
- What it measures: [plain English]
- Output labels: [label list or "None (scalar)"]
- Expected result structure: [instant vector / scalar / etc.]
```

Fix all issues and re-validate until clean.

---

## Verifying Generated Rules

Check the shipped rule files, or your own, with `promtool` before loading them into Prometheus:

```bash
promtool check rules assets/alerting_rules.yaml
promtool check rules assets/recording_rules.yaml
```

Run a candidate query against a live server to confirm it returns series:

```bash
curl -s 'http://localhost:9090/api/v1/query' --data-urlencode 'query=sum by (job) (rate(http_requests_total[5m]))'
```

Query a recorded series instead of repeating its expression:

```promql
job:http_requests:rate5m{job="api-server"}
```

Use `*_over_time()` for a gauge rather than `rate()`:

```promql
avg_over_time(node_memory_MemAvailable_bytes[5m])
```

Browse the reusable query templates with a plain listing:

```bash
grep -n "^# " assets/common_queries.promql
```

---

## Alerting and Recording Rule Snippets

```yaml
# Alerting rule with for clause
alert: HighErrorRate
expr: |
  (
    sum(rate(http_requests_total{status_code=~"5.."}[5m]))
    / sum(rate(http_requests_total[5m]))
  ) > 0.05
for: 10m

# Recording rule (naming: level:metric:operations)
- record: job:http_requests:rate5m
  expr: sum by (job) (rate(http_requests_total[5m]))
```

---

## Anti-Patterns

Each entry below pairs a reason with a BAD and a GOOD query. See `references/best_practices.md` for further performance and cardinality guidance.

### NEVER use `rate()` on a gauge metric

**WHY:** `rate()` computes per-second rate of increase and assumes monotonically increasing counters. Applied to a gauge, it produces nonsensical results because gauges can decrease.

**BAD:** `rate(node_memory_MemFree_bytes[5m])`: memory is a gauge

**GOOD:** `node_memory_MemFree_bytes` (direct use) or `delta(node_memory_MemFree_bytes[5m])` for change over time

### NEVER query a histogram with `avg()` across quantile labels

**WHY:** Summary metrics expose pre-aggregated `{quantile="0.95"}` labels that cannot be re-aggregated across instances. Using `avg()` on them produces statistically meaningless results.

**BAD:** `avg(http_request_duration_seconds{quantile="0.95"})`

**GOOD:** `histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))`: use histogram type with `histogram_quantile()`

### NEVER use a high-cardinality label in `by()` without filtering first

**WHY:** Aggregating by `user_id`, `request_id`, or `pod` without label filters produces thousands of series, overwhelming dashboards and recording rules.

**BAD:** `sum by (user_id) (rate(http_requests_total[5m]))`

**GOOD:** `sum by (job, status_code) (rate(http_requests_total{job="api"}[5m]))`: filter and aggregate on stable, low-cardinality labels

### NEVER use `increase()` for alerting thresholds

**WHY:** `increase()` is extrapolated and can return non-integer values on sparse counters. For alerting, `rate()` produces stable per-second thresholds that are scrape-interval independent.

**BAD:** `increase(http_requests_total{status=~"5.."}[5m]) > 10`

**GOOD:** `rate(http_requests_total{status=~"5.."}[5m]) > 0.033` (2 errors/minute = 0.033/second)

### NEVER omit the `for` clause on alert rules

**WHY:** Alerts without `for` fire immediately on a single evaluation, causing false positives from transient spikes. The `for` clause requires the condition to be true for a sustained period.

**BAD:** `alert: HighErrorRate` with `expr: error_rate > 0.05` and no `for`

**GOOD:** Add `for: 5m` to require the condition to hold for 5 minutes before firing

### NEVER express an error rate as a raw count

**WHY:** Raw error counts rise with traffic, so a busy service looks worse than a failing quiet one. A ratio of rated errors over rated totals is comparable across load.

**BAD:** `sum(rate(http_requests_total{job="api-server", status_code=~"5.."}[5m]))` used as the alert expression.

**GOOD:** `sum(rate(http_requests_total{job="api-server", status_code=~"5.."}[5m])) / sum(rate(http_requests_total{job="api-server"}[5m]))`

### NEVER write a query before the user has confirmed the plan

**WHY:** Guessing the metric name, type or window produces a query that parses but answers the wrong question. The planning workflow exists to catch this before code is written.

**BAD:** Generate `rate(http_requests_total[5m])` straight from "show me errors".

**GOOD:** Present the plain-English plan at stage 4 and wait for confirmation, then generate.

### NEVER deliver a query without validating it

**WHY:** Stage 6 of the workflow catches syntax errors and best-practice issues before the user pastes the query into a dashboard or rule file.

**BAD:** Hand over the first draft and stop.

**GOOD:** Run the validator, show the Validation Checklist output, fix issues and re-validate until clean.

### NEVER use a regex matcher when the label value is known

**WHY:** Equality matchers are cheaper and unambiguous, while a loose regex such as `=~".*"` can match far more series than intended. Key Rule 4 prefers exact matches.

**BAD:** `http_requests_total{job=~"api.*"}` when the job is `api-server`.

**GOOD:** `http_requests_total{job="api-server"}`

### NEVER repeat an expensive expression across dashboards and alerts

**WHY:** Each copy is evaluated separately and the copies drift apart over time. A recording rule computes it once and gives every consumer the same series.

**BAD:** Pasting `sum by (job) (rate(http_requests_total[5m]))` into several panels and alert rules.

**GOOD:** Define `- record: job:http_requests:rate5m` once and query `job:http_requests:rate5m` everywhere.

### NEVER use a bare `irate()` window for alert thresholds

**WHY:** `irate()` only looks at the last two samples, so a single noisy scrape can trip or clear an alert. `rate()` over a window smooths this out.

**BAD:** `irate(http_requests_total{job="api-server", status_code=~"5.."}[1m]) > 0.05`

**GOOD:** `rate(http_requests_total{job="api-server", status_code=~"5.."}[5m]) > 0.05`

### NEVER mix metrics from different windows in one ratio

**WHY:** An error rate that divides a 5m rate by a 1h rate compares unlike quantities and gives a misleading percentage.

**BAD:** `sum(rate(errors_total[5m])) / sum(rate(requests_total[1h]))`

**GOOD:** `sum(rate(errors_total[5m])) / sum(rate(requests_total[5m]))`

## Documentation Lookup

1. **Documentation MCP server (preferred)**: resolve `prometheus`, then fetch docs with the relevant topic.
2. **Fallback WebSearch**: `"Prometheus PromQL [function/operator] documentation [version] examples"`

---

## Error Handling Quick Reference

| Symptom | Likely Cause | Fix |
|---|---|---|
| Empty results | Wrong label filters or metric not scraped | Check `up{job="..."}`, verify label values |
| Too many series | High cardinality | Add label filters, aggregate, use recording rules |
| Wrong values | Wrong function for metric type | `rate()` on counters; direct or `*_over_time()` on gauges |
| Slow queries | Large range vectors or missing filters | Narrow time range, add filters, use recording rules |

---

## References

**Internal:**

- [PromQL Functions](references/promql_functions.md): all PromQL functions with examples; read for specific function syntax questions
- [PromQL Patterns](references/promql_patterns.md): RED/USE method patterns, alerting rules, recording rules; read for standard monitoring patterns
- [Best Practices](references/best_practices.md): anti-patterns, performance optimization, cardinality management; read when optimizing queries
- [Metric Types](references/metric_types.md): counter/gauge/histogram/summary guide; read to confirm correct function choice
- [Common Queries](assets/common_queries.promql): reusable request rate, error rate, latency, and availability query templates
- [RED Method](assets/red_method.promql): complete RED method (Rate, Errors, Duration) implementation
- [USE Method](assets/use_method.promql): complete USE method (Utilization, Saturation, Errors) implementation
- [SLO Patterns](assets/slo_patterns.promql): SLO, error budget, burn rate, and multi-window alerting patterns
- [Alerting Rules](assets/alerting_rules.yaml): example alerting rules with thresholds and `for` clauses
- [Recording Rules](assets/recording_rules.yaml): example recording rules with `level:metric:operations` naming
- [Kubernetes Patterns](assets/kubernetes_patterns.promql): kube-state-metrics, cAdvisor, and vector matching examples