git:20260617.174aad8 to git:20260713.d210701

13 added, 73 removed. Audit A to A.

---

name: eval-regression

- description: "Use when user says regression test, eval check, eval regression, test before commit, compare versions, check regressions, run evals, benchmark skill, or /eval-regression. Compares current (modified) version against last committed version to detect regressions, improvements, and behavioral changes."

- user-invocable: true

- allowed-tools: Glob, Read, Bash, Agent, Write, Grep, Edit

+ description: Use when the user asks to compare a plugin or skill against HEAD, run regression evals, or check behavioral regressions before a commit.

+ allowed-tools: Glob, Read, Bash, Grep

---

- # Eval Regression Testing

- Compare the current (working-tree) version of a plugin/skill against its last committed version to catch regressions before committing. Thin wrapper around the `skill-creator` plugin (its grader, aggregator, viewer) — locate it once:

- ```bash

- SKILL_CREATOR_PATH=$(ls -d ~/.claude/plugins/cache/claude-plugins-official/skill-creator/*/skills/skill-creator 2>/dev/null | head -1)

- ```

- ## 1. Resolve target

- Find the target from the argument: `plugins/<arg>/` (plugin) or `plugins/*/<arg>/` (skill), else cwd. Set `PLUGIN_DIR`, `PLUGIN_NAME` (from `plugin.json`), `EVALS_PATH=<PLUGIN_DIR>/evals/evals.json`.

- ## 2. Verify evals exist

- If `evals.json` is missing/empty, offer to either show the schema (`references/regression-schemas.md`) for the user to write them, or generate 5-10 starter evals from the target's behavioral contracts (routing decisions, guard conditions, required tool sequences, anti-patterns) for review.

- ## 3. Detect changes

- `git diff --name-only HEAD -- <PLUGIN_DIR>/`. No changes -> STOP. Otherwise flag which changed files are behavioral (SKILL.md, routing rules, workflow, patterns, agents) vs not (README, docs, formatting); if only non-behavioral, ask whether to proceed.

- ## 4. Setup workspace

- Workspace: `plugins/<PLUGIN_NAME>-workspace/iteration-<N>/`, sibling to the plugin dir, gitignored via `*-workspace/`. Iteration numbers are sequential and never reused — next N is `(ls -d plugins/<PLUGIN_NAME>-workspace/iteration-* | sort -t- -k2 -n | tail -1) + 1`, starting at 1.

- Snapshot the committed version without touching the working tree:

- ```bash

- SNAPSHOT_DIR="plugins/<PLUGIN_NAME>-workspace/iteration-<N>/skill-snapshot"

- mkdir -p "$SNAPSHOT_DIR"

- git archive HEAD -- <PLUGIN_DIR>/ | tar -x -C "$SNAPSHOT_DIR"

- ```

- Write `iteration-<N>/iteration_metadata.json` (schema in `references/regression-schemas.md`).

- ## 5. Run evals (parallel)

- For each eval, spawn two executor subagents (Agent tool, `run_in_background: true`, all launched in one turn = 2 x N): one reading the skill from `<PLUGIN_DIR>/`, one from `<SNAPSHOT_DIR>/<PLUGIN_DIR>/`. Each subagent must respond to `eval.prompt` as Claude Code with the skill loaded, then **STOP after its first routing decision or action choice — do NOT implement** (evals measure routing, not full execution). Save transcripts to `iteration-<N>/eval-<ID>/{new,old}_version/outputs/transcript.md` and `eval_metadata.json` per eval dir.

- ## 6. Grade results

- Grade every run in parallel using `$SKILL_CREATOR_PATH/agents/grader.md`. Per run, write `grading.json` (sibling to `outputs/`) using viewer-strict field names — `text`/`passed`/`evidence`, NOT `name`/`met`/`details`. Schema in `references/regression-schemas.md`.

- ## 7. Aggregate & analyze

- Aggregate via skill-creator, naming the two configurations `new_version`/`old_version` (not `with_skill`/`without_skill`):

- ```bash

- cd "$SKILL_CREATOR_PATH" && python -m scripts.aggregate_benchmark <workspace>/iteration-<N> --skill-name "$PLUGIN_NAME"

- ```

- If it fails, write `benchmark.json`/`benchmark.md` manually per `$SKILL_CREATOR_PATH/references/schemas.md`.

- Classify each eval by the old/new PASS-FAIL quadrant — only **REGRESSION (old PASS, new FAIL) is CRITICAL** (others: IMPROVEMENT, STABLE_PASS, STABLE_FAIL). Compare at the assertion level and write `iteration-<N>/regression-report.md` (template in `references/regression-schemas.md`), mapping any regression to the changed file likely responsible.

- ## 8. Present

- Report pass-rate delta and counts per status; name each regression with its suspect file (the user decides fix or accept). If iteration > 1, offer the comparison viewer:

+ # Eval regression

- ```bash

- python "$SKILL_CREATOR_PATH/eval-viewer/generate_review.py" <workspace>/iteration-<N> \

- --skill-name "$PLUGIN_NAME" --benchmark <workspace>/iteration-<N>/benchmark.json \

- --previous-workspace <workspace>/iteration-<N-1>

- ```

+ Compare the working tree with `HEAD` using the same Pydantic Evals cases, runner, agent, and repeat count. Only the plugin version may differ.

- ## Rules

+ Resolve the target plugin from the argument or current directory. Use its `evals/fresh-context.json`; if absent, help define executable cases first, using this plugin's file as the smallest example. Support only installed Claude and Codex CLIs. Run the requested agent, or both when none is named.

- - Read-only on the target; never commit (the user decides).

- - Same evals/assertions across both versions — only the skill version differs.

+ For each agent:

- ## Edge cases

+ 1. Stop if the target has no behavioral diff.

+ 2. Create a temporary directory and extract the base with `git archive HEAD` without touching the working tree. Accept an explicit `--base <commit>` instead.

+ 3. Run `../ai-agent-bench/scripts/fresh_context_eval.py` against the archived plugin and the candidate plugin. Use `repeat=3` by default; use `1` only for an explicit smoke run.

+ 4. Compare their JSON reports with the runner's `--compare BASELINE CANDIDATE` mode.

+ 5. Report each `REGRESSION`, `IMPROVEMENT`, `STABLE`, or `INCONCLUSIVE` result and its observed rates. A missing run, timeout, or unequal evidence is inconclusive, never a pass.

+ 6. Remove the temporary directory. Do not edit or commit the target.

- - Compare against an arbitrary commit: `--base <commit>` in place of HEAD.

- - Eval references a missing file: skip it, note in report.

- - Subagent timeout: mark INCONCLUSIVE, not a regression.

- - Committed version unchanged from a prior iteration: reuse that iteration's `old_version` results.

+ The runner sends traces to Logfire only when a token is present. Its assertions are deterministic only (`tool`, `clean_worktree`, `changed_file`, `file_contains`); a genuinely semantic contract means extending the runner with a Pydantic AI judge first, not approximating it with prose.