eval-regression ยท diff
git:20260713.d210701 to git:20260724.ba2c70f
21 added, 12 removed. Audit A to A.
---
name: eval-regression
- description: Use when the user asks to compare a plugin or skill against HEAD, run regression evals, or check behavioral regressions before a commit.
- allowed-tools: Glob, Read, Bash, Grep
+ description: Use when the user asks to check a plugin or skill with behavioral evals, compare it with HEAD, or investigate a behavioral regression.
+ argument-hint: "--agent <claude|codex> --model <id> --effort <level> [--base <commit>]"
+ allowed-tools: Glob, Read, Bash, Grep, AskUserQuestion
---
# Eval regression
- Compare the working tree with `HEAD` using the same Pydantic Evals cases, runner, agent, and repeat count. Only the plugin version may differ.
+ Use deterministic repository tests first. Stop when the target has no behavioral diff.
- Resolve the target plugin from the argument or current directory. Use its `evals/fresh-context.json`; if absent, help define executable cases first, using this plugin's file as the smallest example. Support only installed Claude and Codex CLIs. Run the requested agent, or both when none is named.
+ Resolve the plugin from the argument or cwd. Its catalog is `evals/evals.json`; the shared runner is `../ai-agent-bench/scripts/run_evals.py`. Require the user to choose agent, model, and effort. Never select a costly model or high effort silently.
- For each agent:
+ ## Normal check
- 1. Stop if the target has no behavioral diff.
- 2. Create a temporary directory and extract the base with `git archive HEAD` without touching the working tree. Accept an explicit `--base <commit>` instead.
- 3. Run `../ai-agent-bench/scripts/fresh_context_eval.py` against the archived plugin and the candidate plugin. Use `repeat=3` by default; use `1` only for an explicit smoke run.
- 4. Compare their JSON reports with the runner's `--compare BASELINE CANDIDATE` mode.
- 5. Report each `REGRESSION`, `IMPROVEMENT`, `STABLE`, or `INCONCLUSIVE` result and its observed rates. A missing run, timeout, or unequal evidence is inconclusive, never a pass.
- 6. Remove the temporary directory. Do not edit or commit the target.
+ 1. Select cases by changed paths with `--changed-from <base>`, or name them with `--case`. Do not select the full catalog implicitly.
+ 2. Run the command without `--run`. The runner prints the cases, modes, repeat count, session count, per-session timeout, and maximum duration without starting an agent.
+ 3. Present that plan and stop for explicit cost approval.
+ 4. After approval, repeat the same command with `--run`. The default is candidate-only, one run per case, 180 seconds per session, and at most four sessions.
+ 5. Report every failed assertion, timeout, non-zero exit, duration, and token count. Missing evidence is inconclusive.
- The runner sends traces to Logfire only when a token is present. Its assertions are deterministic only (`tool`, `clean_worktree`, `changed_file`, `file_contains`); a genuinely semantic contract means extending the runner with a Pydantic AI judge first, not approximating it with prose.
+ ## Escalation
+
+ - Compare base and candidate only when the user asks, or when a failed candidate check needs to distinguish a regression from an existing failure. Extract the base with `git archive`; run the same selected cases, agent, model, effort, repeat, and timeout on both; compare reports with `--compare`.
+ - Repeat only a failed or observably unstable case. Three repeats are a stability benchmark, not a default.
+ - `--all`, `--repeat > 1`, a larger `--max-sessions`, or a longer timeout needs a new run plan and explicit approval.
+ - Routing cases stop at the first Skill selection. A tool assertion is appropriate there because routing is the contract; they must not execute the selected skill.
+
+ Remove temporary base copies after the comparison. Do not edit or commit the target.
+
+ Deterministic assertions are `tool`, `tool_not`, `clean_worktree`, `changed_files_exact`, `file_contains`, `file_not_contains`, `transcript_contains`, and `tool_sequence`. Use a semantic judge only when no filesystem, command, tool, ordering, or assistant-output observation can express the contract.