query-metrics · diff
git:20260709.5d15061 to git:20260811.29ee512
52 added, 13 removed. Audit A to A.
---
name: query-metrics
description: Runs metrics queries against Axiom MetricsDB via scripts. Discovers available metrics, tags, and tag values. Use when asked to query metrics, explore metric datasets, check metric values, or investigate OTel metrics data.
---
# Querying Axiom Metrics
All script paths are relative to this skill's folder; invoke as `scripts/<name>`. The target dataset must be of kind `otel:metrics:v1`.
Setup, prerequisites, and `~/.axiom.toml` configuration: see `README.md`. Edge-deployment routing is automatic — the scripts read each dataset's `edgeDeployment` and route to the right regional endpoint without configuration.
## Workflow
1. `scripts/datasets <deploy> --kind otel:metrics:v1` — list metrics datasets.
- 2. `scripts/metrics-spec <deploy> <dataset>` — **required** before composing any query. MPL evolves; the spec is the source of truth.
+ 2. `scripts/metrics-spec` — **required** before composing any query. MPL evolves; the spec is the source of truth. Also use it to answer general MPL/metrics questions.
3. `scripts/metrics-info <deploy> <dataset> metrics` — list metrics with `{type, temporality, unit}` metadata. Read this before writing the query (see [Choosing a Query Shape](#choosing-a-query-shape)).
4. `scripts/metrics-info <deploy> <dataset> tags [<tag> values]` — explore filter dimensions.
5. `scripts/metrics-query <deploy> '<MPL>' <start> <end>` — execute. Iterate.
If the user names a specific entity (service, host, …), `scripts/metrics-info <deploy> <dataset> find-metrics "<value>"` finds the metrics carrying it. `find-metrics` searches **tag values**, not metric names — don't use it for general discovery.
## Choosing a Query Shape
The `metrics-info` listing returns each metric's `{type, temporality, unit}`. Read these before composing — never assume a metric is a simple scalar.
| Field | Values | Drives |
|---|---|---|
| `type` | `Gauge`, `CounterMonotonic`, `CounterNonMonotonic`, `Histogram` | Required pre-aggregation operators. |
| `temporality` | `Cumulative`, `Delta`, `null` | Whether counter values are running totals or per-interval deltas. `null` is normal for Gauges. |
| `unit` | UCUM string (`Cel`, `kW.h`, `s`, `%`, `[ppm]`, …) or `null` | Display unit; preserve when reporting results. |
Rules per type (consult `metrics-spec` for exact operator names — they evolve):
- **Gauge** — instantaneous value. Align directly with `avg`/`min`/`max`/`sum`. Don't apply a rate; you'd be averaging meaningless deltas of an instantaneous value.
- **CounterMonotonic + Cumulative** — running total (resets aside). The raw values are rarely what you want. Convert to a per-second rate first, **then** align/aggregate.
- **CounterMonotonic + Delta** — already per-interval. Sum/align without a rate step.
- **CounterNonMonotonic** — can go up or down (queue depth, balance). Intent is ambiguous: rate, delta, or current value all make sense for different questions. **Ask the user** before picking one.
- - **Histogram** — not a scalar. `align using avg` produces nonsense. Use the bucket/quantile operators from `metrics-spec`.
+ - **Histogram** — not a scalar. `align using avg` produces nonsense. Use `bucket … using` with the histogram functions from `metrics-spec`; quantiles are float specs to those functions, and `temporality` selects the variant (`Cumulative` vs `Delta` interpolation). Consult `metrics-spec` for the exact signatures.
- **`temporality: null`** — "not applicable for this instrument type" (the norm for Gauges), not "missing data".
When surfacing numbers, attach the `unit` (treat `null` as unitless). If you combine metrics with mismatched units in arithmetic, warn rather than silently producing a meaningless number.
## Query Metrics
```bash
- scripts/metrics-query <deploy> '<MPL>' <start> <end>
+ scripts/metrics-query [-w pixels] [--pixel-per-point n] <deploy> '<MPL>' <start> <end>
```
| Parameter | Notes |
|---|---|
| `deploy` | Name from `~/.axiom.toml` (e.g. `prod`). |
| `MPL` | Pipeline string. Dataset is parsed from the MPL itself. |
| `start` / `end` | RFC3339 (`2025-01-01T00:00:00Z`) or relative (`now-1h`, `now`). |
+ | `-w` / `--chart-width <px>` | Optional. Target chart width in pixels; lets the server resolve `$__interval`. |
+ | `--pixel-per-point <n>` | Optional. Pixels per point (server default 10); with `-w` sets the bucket count. |
+ **Always single-quote the MPL string in the shell.** MPL is full of backticks; inside double quotes the shell executes them as command substitution, silently mangling the query (or running whatever the identifier names).
+
+ **Bound the output before grouping.** `group by <tag>` returns one series per tag value with no cap — on a high-cardinality tag this floods the output. Check cardinality first (`describe`, or `tags <tag> values`) and prefer plain `group using <agg>` while exploring.
+
Examples:
```bash
- scripts/metrics-query prod \
- '`my-dataset`:`http.server.duration` | align to 5m using avg' \
+ scripts/metrics-query prod -w 1200 \
+ '`my-dataset`:`http.server.duration` | align to $__interval using avg' \
now-1h now
- scripts/metrics-query prod \
+ scripts/metrics-query prod -w 1200 \
'`my-dataset`:`http.server.duration`
| where `service.name` == "frontend" and method == "GET"
- | align to 5m using avg
+ | align to $__interval using avg
| group by status_code using sum' \
now-1d now
```
+ ### Adaptive resolution (`$__interval`)
+
+ Hardcoding a step (`align to 5m`) makes charts look wrong at other zoom
+ levels — too sparse zoomed in, too dense zoomed out. Prefer the system
+ parameter `$__interval` wherever a `Duration` is expected, and pass the chart
+ width so the server picks the step:
+
+ ```bash
+ scripts/metrics-query prod -w 1200 \
+ '`my-dataset`:`http.server.duration` | align to $__interval using avg' \
+ now-7d now
+ ```
+
+ The metrics service computes `$__interval` from the query's time range and the
+ target chart width, then snaps it **up** to a nice resolution from the ladder
+ `1s, 5s, 10s, 15s, 30s, 1m, 5m, 10m, 15m, 30m, 1h, 12h, 1d, 1w, 1M, 1Y`. It
+ never drops below a metric's stored resolution.
+
+ - **No declaration needed** — the server auto-registers `$__interval`; do *not*
+ add `param $__interval: Duration;` (the edge forwards the query verbatim and
+ the metrics service injects the parameter).
+ - **Bucket count** ≈ `chart-width / pixel-per-point` (`pixel-per-point` default
+ 10). Omit `-w` and the server targets ~500 buckets.
+ - Works anywhere a `Duration` is valid, e.g. `bucket to $__interval using
+ histogram(0.5, 0.95)`.
+ - Set `-w` to your render width (e.g. the `metrics-chart` skill's plot width)
+ so one bucket ≈ one pixel column. The value is forwarded under the request
+ body's `queryOptions` (`chart-width`, `pixel-per-point`).
+
### Parameters
MPL can declare parameters (`param $svc: string;`). Pass values with repeated `-p name=value`. The script applies the API's `param__` prefix; values are forwarded verbatim as MPL literals (string literals include their quotes).
```bash
scripts/metrics-query \
-p svc='"frontend"' \
-p window='5m' \
prod \
'param $svc: string; param $window: Duration;
`otel-metrics`:`http.server.duration` | where `service.name` == $svc | align to $window using avg' \
now-1h now
```
Required parameters must be supplied; optional ones may be omitted. Resulting request body shape:
```json
{
"apl": "param $svc: string; …",
"startTime": "now-1h",
"endTime": "now",
"params": { "param__svc": "\"frontend\"", "param__window": "5m" }
}
```
Literal syntax per type lives in `metrics-spec`.
## Discovery (`metrics-info`)
- Time range defaults to the last 24h; override with `--start` / `--end`.
+ Time range defaults to the last 24h; override with `--start` / `--end`. Both accept RFC3339 (offsets allowed) or relative `now` / `now-<N><unit>` with `<unit>` in `s m h d w`, resolved to RFC3339 UTC client-side. This is **narrower** than `metrics-query`, which forwards times to the server unparsed and so also accepts forms like `now-1y`; in `metrics-info` anything outside `now` / `now-<N>[smhdw]` must already be RFC3339 or the request 400s.
| Command | Returns |
|---|---|
| `metrics-info <d> <ds> metrics` | All metrics, keyed by name, with `{type, temporality, unit}`. |
| `metrics-info <d> <ds> metrics --by-type` | Same listing grouped by `type` (client-side reshape). |
| `metrics-info <d> <ds> metrics --type Gauge --type Histogram` | Filtered listing (repeatable, OR semantics; composes with `--by-type`). |
| `metrics-info <d> <ds> metrics <metric> info` | Single metric's `{type, temporality, unit}`. Non-zero exit if absent. |
| `metrics-info <d> <ds> metrics <metric> describe` | Bundle: metadata + all tags + tag values in one call (replaces 1+1+N round trips). Flags: `--no-values` (tag names only), `--values-limit N` (cap per-tag values; default 50, 0 = unlimited). |
| `metrics-info <d> <ds> metrics <metric> tags` | Tags carried by a specific metric. |
| `metrics-info <d> <ds> metrics <metric> tags <tag> values` | Tag values for that metric. |
| `metrics-info <d> <ds> metrics <metric> tags <tag> type` | Probe whether the tag is `int`/`float`/`string`/`bool`. Returns `{type, present_types}`; `mixed` if multiple types coexist, `absent` if not present. |
| `metrics-info <d> <ds> tags` | All tags in the dataset. |
| `metrics-info <d> <ds> tags <tag> values` | All values for a tag (across metrics). |
| `metrics-info <d> <ds> find-metrics "<value>"` | Metrics that carry the given tag *value* (not metric name). |
## Error Handling
- HTTP errors return JSON with `message`, `code`, and optional `detail`:
+ HTTP errors return JSON with `code` and `message`; some include a `detail` object:
```json
- {"message": "...", "code": 400, "detail": {"errorType": 1, "message": "raw error"}}
+ {"code": 400, "message": "MPL syntax error: …"}
```
+ Syntax errors (400) include an annotated source pointer listing the valid operators at the failure position — read it, it usually names the fix.
+
| Code | Cause |
|---|---|
| 400 | Invalid query syntax or bad dataset name |
| 401 | Missing/invalid auth |
| 403 | No permission |
| 404 | Dataset not found |
- | 429 | Rate limited |
+ | 429 | Rate limited — back off and retry; don't tight-loop |
| 500 | Internal error |
+ Requests time out client-side after 120s (`AXIOM_MAX_TIME` to override; `AXIOM_CONNECT_TIMEOUT` for the 10s connect timeout).
+
On 500, re-run with `curl -v` to capture the `traceparent` / `x-axiom-trace-id` header and report it — the trace ID is what the backend team needs to debug.
## Scripts
| Script | Usage |
|---|---|
| `scripts/setup` | Check requirements and config. |
| `scripts/datasets <deploy> [--kind <kind>]` | List datasets with edge deployment. |
- | `scripts/metrics-spec <deploy> <dataset>` | Fetch the MPL query spec. |
- | `scripts/metrics-query <deploy> <mpl> <start> <end>` | Execute a query. |
+ | `scripts/metrics-spec` | Fetch the MPL query spec. |
+ | `scripts/metrics-query [-w px] [--pixel-per-point n] <deploy> <mpl> <start> <end>` | Execute a query; use `$__interval` + `-w` for adaptive resolution. |
| `scripts/metrics-info <deploy> <dataset> ...` | Discover metrics, tags, values. |
| `scripts/axiom-api <deploy> <method> <path> [body]` | Low-level API calls. |
| `scripts/resolve-url <deploy> <dataset>` | Resolve to the edge deployment URL. |
Run any script without arguments for full usage.