reliability · diff
git:20260827.7ff2e25 to git:20260910.c4b91ef
23 added, 73 removed. Audit A to A.
---
name: reliability
- description: >-
- Check whether an AI analysis answer is STABLE by running the same question several independent
- times and measuring what holds versus what drifts. Use when the user types "/reliability", or
- asks to "run this a few times", "is this answer stable / reliable", "check reliability of
- <question>", "does it give the same answer again", "run it N times and compare", or wants to
- know if a number can be trusted enough to act on. This is the cheapest eval and the only one
- that needs no answer key: it tells you about stability, not correctness.
+ description: Run one analytics task through several fresh trials to measure what repeats and what varies. Use for reliability, repeatability, variance, or repeated-run requests. This measures stability, not correctness.
---
- # Skill: Reliability check
-
- ## Purpose
- Run one analytics question several **independent** times and report whether the answer
- is stable (every run agrees) or drifting (runs disagree because the question is
- under-defined). Stability is necessary, not sufficient: a wrong query is perfectly
- stable. This check needs no ground truth.
+ # Reliability
- ## Invocation
- `/reliability "<the question>" [N]` — default N = 5.
- Example: `/reliability "What's our retention rate?"`
+ ## What this answers
- ## How to run it
+ Does this named system configuration behave consistently on this task?
- ### Step 1 — fire N independent runs
- Launch **N sub-agents in parallel** with the Task/Agent tool (N defaults to 5). They must
- be genuinely independent: each gets a fresh context and sees ONLY the question, never the
- other runs' answers. Give each sub-agent exactly this brief:
+ It does not answer whether the result is correct. A wrong analysis can repeat perfectly.
- > You are answering one analytics question against the active dataset. Load the normal
- > session context first (knowledge-bootstrap: read `.knowledge/active.yaml`, then the active
- > dataset's `schema.md`, `quirks.md`, and manifest from the local datasets dir). For the
- > metric dictionary and semantic context, first resolve the context dir with
- > `from helpers.knowledge.context_sync import resolve_context_dir` (`helpers/knowledge/context_sync.py`) and
- > read `metrics/index.yaml` and `semantic/` from the RESOLVED dir, not from the local copy.
- > Do not read `.knowledge/reliability/` history before answering. If
- > the metric you're asked about is defined in the dictionary, use that definition exactly.
- > If it is not, decide for yourself how best to define and measure it. Query the real data
- > with the repo connection (works for CSV folders too; never hand-roll DuckDB over the files) (`from helpers.data.connection_manager import ConnectionManager`).
- > Answer the question: "<QUESTION>". Then return ONLY this block:
- > `headline: <the single number you'd report>`
- > `measured: <one line: numerator, denominator, grain, window, any filter>`
- > `definition_source: <"metric dictionary" if you used a defined metric, else "my own choice">`
+ ## Run the evaluation
- Do not let the runs share state. Run them concurrently.
+ 1. Record the task, model, active data, available tools, system fingerprint, and the tolerance that matters for the decision.
+ 2. Use five fresh trials unless the user chooses another count. The trials must not share answers.
+ 3. Preserve completed, failed, blocked, error, and unparseable trials separately.
+ 4. Save the raw output from every trial.
+ 5. Use `helpers.evals.reliability.measure_reliability` for the calculation. Do not estimate agreement yourself.
+ 6. Report exact agreement and agreement within the named tolerance separately.
+ 7. State what changed if this is a comparison with an earlier run.
- ### Step 2 — record the runs (tracked + auditable)
- Write the N results to a timestamped run directory so every check leaves an audit trail:
- `.knowledge/reliability/<UTC-timestamp>-<question-slug>/runs.json`, shaped as
- `{"question": "<the question>", "runs": [{"run":1,"headline":"...","measured":"...","definition_source":"..."}, ...]}`.
+ The deterministic CLI is available at `python3 -m helpers.evals.cli run-reliability`. Use it when the task can be executed noninteractively. Use `--model claude-opus-4-6`. Add `--allow-code` only when the task genuinely requires local code or data access.
- ### Step 3 — compute the statistics (deterministic, not estimated)
- Run: `python3 helpers/stats/reliability_stats.py <that run directory>`. It computes the numbers
- deterministically (never let the model estimate them): distinct values, min/max/range,
- mean, stdev, CV, the agreement count + agreement rate, how many runs used the metric
- dictionary, and the STABLE/DRIFT verdict. It writes `stats.json` + `report.md` in the run
- dir and appends one line to `.knowledge/reliability/log.jsonl` (the audit log of every
- reliability check over time, so results can be tracked).
+ ## Report
- ### Step 4 — report (short, on-screen)
- Show the `report.md` it produced: the `Run | headline | what it measured | source` table,
- the verdict, and the computed stats (distinct values, agreement rate, CV, range). Frame it:
- - **STABLE** — "Same answer every run. Lean on it being consistent. Stable is not correct
- though, a wrong query is perfectly stable too, so this says it's settled, not that it's
- the reading you meant."
- - **DRIFT** — "N distinct readings. Each run quietly chose a different definition. The
- spread is the check telling you to go define this before you trust any single number."
+ Lead with:
- Tell the user where it was saved (the run dir + `.knowledge/reliability/log.jsonl`). Then
- the honest footnotes: N is illustrative (size it to the precision you need); this check
- needs no answer key (it measures stability, not correctness); nearly free as long as the
- runs are genuinely independent (some tools cache answers, and then it sees nothing).
+ - how many trials succeeded;
+ - every normalized result beside its raw form;
+ - exact agreement;
+ - tolerance-based agreement;
+ - what definitions or methods changed; and
+ - what the evidence does not establish.
- ## Notes
- - The active warehouse connection is whatever the session is configured for (e.g.
- Snowflake when `AAP_USE_REMOTE=1`); the sub-agents just use the repo connection manager.
- - The fix for drift is not a smarter model, it's context: define the metric once in the
- dictionary (the `/metric-spec` skill, or just tell the analyst the definition and ask it
- to save it), then run `/reliability` again and the runs converge on the meaning you set.
- - One exception: when you are running a *comparison* (with-and-without, the `/context-compare` skill),
- do NOT save the definition into the dictionary with `/metric-spec`. That writes the base
- `index.yaml` permanently and erases the no-definition baseline the comparison needs. Stage
- it as a temporary overlay through the eval tool's adapter instead
- (`AIAnalystPlusAdapter().stage(<setup_dir>)` composes base + overlay for the run,
- `.restore()` puts it back). Save into the dictionary only when you want the definition to
- be permanent, not for a comparison.
+ If the trials vary, locate the source before proposing a fix. If one definition or context change is made, freeze everything else and rerun the same task.