git:20260905.a750d17 to git:20260905.e04ae6b

Operational guide for everything under `tests/`. The authored harness

is committed, including this file; `tests/.gitignore` keeps every

regenerated subtree local. `tests/README.md` has the split and the

matching `make lint` prunes.

Per-harness design docs live in each subdirectory's `README.md` and

`RUNBOOK.md`. Use those for *what* the tests cover; use this file for

*how* to run them correctly from inside a Claude Code session and

*how* to read the results without misleading yourself.

## What's here

| Subdir | Skill | Pattern | What it covers |

| --- | --- | --- | --- |

| `wiki/` | `wiki` | Pattern B (legacy two-layer) | Layer 1 deterministic script unit tests + Layer 2 LLM skill-behavior evals via custom orchestrator. |

| `git_commit/` | `git_commit` | Pattern A (skill-creator-aligned) | `script_tests/` bundled-script unit tests + `evals/` behavioral evals run operator-driven (stage → agent runs → grade). |

| `git_checkout/` | `git_checkout` | Pattern A (skill-creator-aligned) | `script_tests/` bundled-script unit tests over staged clones with real remotes (branch resolution, the no-prune fetch, the ambiguity hold, both miss causes, both dirty-worktree branches) + `evals/` behavioral evals run operator-driven (stage → agent runs → grade). |

| `git_review/` | `git_review` | Pattern A (skill-creator-aligned) | `script_tests/` bundled-script unit tests over staged clones with real remotes (the fetch-before-diff order, the two commit walks, base-side versions of deleted files, the test merge, the head-vs-upstream relationship behind the fast-forward decision, stub-`gh` thread pagination, the heading-range helper) + `evals/` behavioral evals over 36 fixtures via `evals/run.py`. |

| `language_humanizer/` | `language_humanizer` | Pattern A (behavioral only) | 3 scenarios × fixed 5-pass denominator; deterministic `grade.py` (word counts, ledger items, prose shape) + refute-biased `judge.py` for the qualitative assertions. |

| `task/` | `task` (family hub) | Pattern A (skill-creator-aligned) | `script_tests/run.sh` unit-tests the bundled `lint.py`, `discover_tasks.sh`, and `init_tasks.sh`; `script_tests/contract_run.sh` asserts the family contract across the hub, its siblings, and the family agents; `evals/` holds a behavioral eval per family member. `run_all.sh` drives both script runners. |

| `task_create/` | `task_create` | Pattern A (behavioral only) | Three staged evals over the base **Decide or label** rule as the create path applies it; the bundled scripts it drives are covered under `task/script_tests/`. |

| `task_auto_check/` | `task_auto_check` | Pattern A (skill-creator-aligned) | `script_tests/` static contract checks + `evals/` over the autonomous readiness loop (repair-to-ready, gate/verifier/drift stops, mechanical lint cleanup). |

| `guardrail_audit/` | `guardrail_audit` | Pattern A (prose-only skill) | `script_tests/` static SKILL.md and registration contract + `evals/` over five staged fixtures with byte-identity grading. |

| `skill_doctor/` | `skill_doctor` | Pattern A (skill-creator-aligned) | `script_tests/` over `resolve_scope.py` and `discovery_safety.py` plus the static contract + `evals/` over three staged fixtures. |

| `git_refresh/` | `git_refresh` | Pattern A (script surface automated) | `script_tests/` over the bundled `refresh_repo.sh` on staged repositories. `evals/` holds four behavioral evals with fixtures but ships no runner, so a sweep does not reach them. |

| `ai_instruction_writing/` | `ai_instruction_writing` | Pattern A (script-only) | `script_tests/` asserts the static prose contract of a skill that ships no bundled scripts. |

| `charter_guardrail/` | `guardrail` | Pattern A (script-only) | `script_tests/` greps the guardrail doc set for its documented contract; no `run_all.sh`, so drive `script_tests/run.sh` directly. |

| `format_rust/` | `format_rust` | Pattern A (script-only) | `script_tests/` greps SKILL.md and the plugin README for the error-versus-invariant model, panic discipline, and clippy wiring; no `run_all.sh`. |

| `update_changelog/` | `update_changelog` | Pattern A (script-only) | `script_tests/` covers the deterministic parts of the incremental day-grouping walk. |

| `deployment/` | the deploy script (no single skill) | Pattern A (script-only) | `script_tests/run.sh` covers the OpenCode, Antigravity, and bytecode-exclusion deployment paths; `script_tests/style_run.sh` covers output-style deployment and uninstall. |

| `trigger_evals/` | wiki and `task_*` family skills | local `run.py` wrapper (auto: deployed-mode or UUID fallback) | Whether a skill *triggers* on realistic user messages (description-matching), separate from skill *behavior*. |

Pattern A is preferred for new harnesses. Pattern B stays in `wiki/`

until the next significant iteration; don't bring up new harnesses

under Pattern B.

## Universal operator lessons

These are the things that bit me in practice. They apply to every

harness here regardless of pattern.

### Model policy: pin the skill-under-test to sonnet, keep the meta level inherited

Every harness that runs a skill as a subprocess pins that worker to

model so results don't drift with whatever the host session happens to

| Harness | Worker (sonnet-pinned) | Meta level (inherited) |

| --- | --- | --- |

| `git_commit/evals/run.py` | `claude -p` per eval (`--model` default `claude-sonnet-4-6`) | `grade.sh` (deterministic) + operator prose-verdict confirmation |

| `task/evals/run.py` | `claude -p` per eval (`--model` default `claude-sonnet-4-6`) | `grade.sh` (deterministic) + operator prose-verdict confirmation |

| `task_create/evals/run.py` | `claude -p` per eval (`--model` default `claude-sonnet-4-6`) | `grade.sh` (deterministic) + operator prose-verdict confirmation |

| `task_auto_check/evals/run.py` | `claude -p` per eval (`--model` default `claude-sonnet-4-6`) | `grade.sh` (deterministic) + operator prose-verdict confirmation |

| `git_review/evals/run.py` | `claude -p` per eval (`--model` default `claude-sonnet-4-6`) | `grade.sh` (deterministic) + operator prose-verdict confirmation |

Each runner takes `--model` to override, and `--model ''` inherits the

CLI default. The behavioral eval runners (`git_commit/evals/run.py`,

`task/evals/run.py`, `task_create/evals/run.py`,

`task_auto_check/evals/run.py`, `git_review/evals/run.py`) automate the old

operator-driven Phase 2: instead of running the skill yourself in-session

(which would use the inherited model), let the runner spawn the sonnet

worker, then read `response.txt` for the prose-verdict expectations

`grade.sh` can't check.

### Verdict cache: skip re-running an eval whose inputs haven't changed

The five Pattern-A behavioral runners (`git_commit/evals/run.py`,

`task/evals/run.py`, `task_create/evals/run.py`,

`task_auto_check/evals/run.py`, `git_review/evals/run.py`) cache each eval's

graded verdict and skip re-spawning its `claude -p` worker when nothing that

determines the verdict has changed. The shared helper is

`tests/lib/eval_cache.py`; verdicts live per-harness in

`<evals>/.eval_cache/` (gitignored as run output, like `workspace/`).

the eval re-runs. It can never serve a stale pass: a hit means byte-identical

inputs to a run already graded. Over-inclusion (e.g. editing `task_select`

safe direction.

- **Default: on.** A hit prints `CACHED PASS/FAIL … skipped claude -p` and

replays the stored verdict, writing `cached.json` plus a clearly-bannered

`response.txt` into the run dir so its shape matches a fresh run.

- **`--force`:** re-run every eval and refresh the cache. Use it to resample

the stochastic worker on unchanged inputs when you want a fresh draw.

- **`--no-cache`:** neither read nor write the cache.

This is the mechanical backstop for the base `task` skill's

verification-economy rule ("re-run only when the inputs changed"), applied to

the one surface where a needless re-run is dramatically expensive. It trades

LLM-sampling variance for cost; `--force` is the escape hatch when the

variance is what you want. Unit + plumbing tests live in

`tests/lib/test_eval_cache.py` (run `python3 tests/lib/test_eval_cache.py`).

### Leave the repo's own `tasks/` and `wiki/` trees alone while a run is live

Every `task`-family eval carries the isolation fail-safe `no writes to the real

repo's tasks/ tree`, graded as `find "$REPO_ROOT/tasks" -type f -newer

"$marker"` against a per-eval marker `stage.sh` stamps at staging time. The

check cannot tell an escaped worker from the operator, so **any** write to the

backlog during a run fails every eval whose marker predates it, and the

`wiki/` harness's `real_home_wiki_absent` fail-safe has the same reach over the

wiki tree.

Measured on 2026-09-05: filing one task file mid-run failed `audit_clean` and

`finish` in a sample that was otherwise clean, and the failure reads exactly

like a sandbox escape until you compare the marker epochs against the file's

mtime.

So finish the run before filing tasks, editing the wiki, or touching anything

those fail-safes watch. When a run has already been contaminated, confirm the

cause by comparing each eval's `.eval_started_at` epoch against the mtime of

whatever landed in the watched tree, then re-run only the evals whose marker

predates that write. Keep the fail-safe as it is: it is the reason these

harnesses can run on a real filesystem, and widening it to forgive operator

writes would forgive a real escape too.

### Cheap-first: probe an LLM-eval fixture before paying for the full loop

validate the fixture with the cheapest surface that reveals the same

verdict *before* running the loop. For a readiness-repair eval that means

running the gate skill (`task_check`) alone against the staged fixture: a

issue list. A fixture-design bug (an unintended second readiness gap, an

inaccurate premise) caught at the gate costs minutes; the same bug caught

past even an 1800s timeout. `task_auto_check/RUNBOOK.md` has the specifics

and the grade-check pitfall that goes with them.

### Worker auth: nested `claude -p` reads the stored OAuth login

A `claude -p` worker spawned from inside a Claude Code session (any

harness runner) does not inherit the host session's in-memory auth. The

session env carries `CLAUDECODE` plus `CLAUDE_CODE_SDK_HAS_OAUTH_REFRESH`

/ `CLAUDE_CODE_SDK_HAS_HOST_AUTH_REFRESH` (host-managed refresh that does

not reach grandchild processes), so a child that still sees `CLAUDECODE`

treats itself as nested and expects the parent's SDK-held auth.

**The fix, derived from a sibling skill-eval runner:** strip

`CLAUDECODE` from the worker env so the child is a plain CLI invocation.

It then reads the CLI's own stored OAuth credential (macOS keychain item

`Claude Code-credentials`) directly, and no separate token is needed

forces `ANTHROPIC_API_KEY` / `apiKeyHelper` auth and 401s a subscription

`worker_env()` pops `CLAUDECODE` and the HAS_*_REFRESH flags, and its

`preflight_auth()` fails fast on a dead login with one live `claude -p`

probe (the remediation below) instead of 401-ing every eval. Every

`claude -p` runner imports the helper: `task/evals/run.py`,

`task_create/evals/run.py`, `task_auto_check/evals/run.py`,

`git_commit/evals/run.py`, and `git_review/evals/run.py` use both the

pre-flight and the worker env; `trigger_evals/run.py` and

`wiki/layer2/run.py` use `worker_env()`.

**Failure signature:** worker rc≠0 with

an expired or absent login, not a skill regression (check `timing.json`

before attributing). The CLI's keychain OAuth is not self-refreshing (a

known bug, anthropics/claude-code#31095, #50743): an expired accessToken

with no usable refreshToken 401s forever. `claude auth status` still

reports `loggedIn` past expiry, so trust the live probe, then re-auth

with `claude auth login` and re-run.

login (e.g. cron). Mint a `claude setup-token` (interactive browser

OAuth; 1-year inference-only token) and park it in the keychain;

`worker_env()` picks it up and it wins over the stored login:

```bash

claude setup-token

security add-generic-password -a "$USER" -s claude-headless-token -w '<token>' -U

```

`CLAUDE_CODE_OAUTH_TOKEN` exported in the shell wins over both, in every

runner, since they all build their worker env from the shared

`worker_env()`.

### Timed-out worker output: decode it through `tests/lib/worker_io.py`

A `claude -p` worker that overruns its deadline raises `TimeoutExpired`, and the

partial output that exception carries is **bytes** even though the runner passed

runners append a `[TIMEOUT after Ns]` note to that output, so a raw

`(exc.stderr or "") + note` raises `TypeError: can't concat str to bytes`. That

error escapes both the per-eval function and `main()`, which used to abort the

whole run: no verdict for the timed-out eval, none for anything queued behind

it, and a traceback where the graded summary belongs. It bit hardest on the

runners most likely to time out, which are the ones driving the slowest loops.

Every runner now resolves both streams through `as_text()` from the shared

`tests/lib/worker_io.py`, which returns `""` for `None`, decodes `bytes` with

`errors="replace"` (the deadline can cut the stream mid multi-byte sequence),

and passes `str` through. A timed-out eval is then recorded like any other

`tests/lib/test_worker_io.py` (run `python3 tests/lib/test_worker_io.py`).

Keep the decode in that one module rather than reintroducing a local copy.

### Running long jobs in the background: `Bash --run_in_background`, not `Monitor + tail -f`

**Right pattern** for a "wake me when this job is done" wait:

```text

Bash(

command="<long-running-command> 2>&1",

run_in_background=true,

timeout=<expected_max_ms>,

)

```

`Bash --run_in_background` fires one completion notification when the

process exits. No polling, no lingering processes.

**Wrong pattern** for a completion wait:

```text

Monitor(command="tail -f <output-file> | grep ...") # don't

```

`tail -f` never exits on its own. When the underlying job finishes,

the tail keeps the monitor armed against the static file until the

monitor's timeout. You then have to call `TaskStop` to clean it up.

`Monitor` is the right shape for *unbounded* per-occurrence streams

(log-tailing for ERROR lines indefinitely), not for completion waits.

For per-occurrence with a natural end, write a script that emits one

### Ground truth lives in files, not mid-stream events

When a harness spawns LLM subagents (Layer 2 in `wiki/`, the

behavioral evals in `git_commit/`), the host Claude Code instrumentation

may surface subprocess-level events into the parent session's

```text

[WU-2 pass-1] FAIL (524.0s)

```

printed by any script in the harness, and are **not** the graded

verdict. They are subprocess-timing artifacts from the host's wrapper

around `claude -p`. A `FAIL (524s)` usually means the subprocess ran

close to its per-pass `--timeout`; the orchestrator may have retried

it, and the final graded state is what counts.

Always cross-check mid-stream impressions against the ground-truth

files the harness writes. If those agree on "clean run," the run was

### The three ground-truth signals to check after any LLM-in-the-loop run

1. **Exit code** of the orchestrator process.

2. **The graded summary file** the harness writes (e.g.

`grading_summary.json` for `wiki/`, `grade.sh` output for

`git_commit/`, `results.json` for `trigger_evals/`).

3. **The regression compare** if the harness has one (e.g.

`benchmark.json["regressions"]` for `wiki/`).

If all three agree, the run was successful. If they disagree, trust

them over any in-flight events.

### Assert plugin-meta lockstep, never a literal version

A harness that checks the plugin metadata must assert the invariant the

standing repo rules state, which is that `.codex-plugin/plugin.json` and

both marketplace registrations carry whatever version

`.claude-plugin/plugin.json` currently holds. Pinning the literal version

a change shipped at makes the harness fail on the next routine bump, and

that is exactly what `tests/guardrail_audit/` and `tests/format_rust/`

both did until they were rewired.

Source the shared helper and call it with the plugin name:

```bash

# shellcheck source=../../lib/plugin_version.sh

. "$HERE/../../lib/plugin_version.sh"

check_plugin_version_lockstep ai_dev

```

`tests/lib/plugin_version.sh` reports through the harness's own `check`

helper, so its four results land in the existing pass and fail tallies.

It needs `REPO_ROOT` and `check` already defined, so source it after the

harness sets those up. Match a marketplace entry by plugin name rather

than grepping the file for a version string, since a registration lists

many plugins and a bare grep matches any of them; the helper already

does this.

The wiki skill ships bundled scripts (`discover_wiki.sh`,

`init_wiki.sh`, `lint.py`, `compute_sha256.py`) **and** load-bearing

prose policy. They regress independently.

### Commands

```bash

# Layer 1 only — deterministic script unit tests, ~1 sec, no LLM cost

./tests/wiki/run_all.sh

# Layer 1 + Layer 2 — full suite, ~10–15 min with 4 parallel workers

./tests/wiki/run_all.sh --layer2

# Single Layer 2 scenario while debugging

python3 ./tests/wiki/layer2/run.py --scenario L2-2

# Re-grade an existing run without re-spawning subagents (after

# editing assertions or evals.json — responses are immutable once

# captured)

RUN=tests/wiki/layer2/workspace/run-<ts>

python3 tests/wiki/layer2/normalize.py "$RUN"

python3 tests/wiki/layer2/grade.py "$RUN"

python3 tests/wiki/layer2/aggregate.py "$RUN"

python3 tests/wiki/layer2/render_report.py "$RUN"

```

### Reading the result

```text

tests/wiki/layer2/workspace/run-<ts>/

├── grading_summary.json # flat list: scenario × pass → passed: true/false

├── benchmark.json # per-assertion pass rate + "regressions": [...]

├── benchmark.md # same data, human-readable

└── report.html # interactive viewer (open in a browser)

```

One-shot verdict check after a run:

```bash

RUN=tests/wiki/layer2/workspace/run-<ts>

python3 -c "

import json, sys

s = json.load(open('$RUN/grading_summary.json'))

b = json.load(open('$RUN/benchmark.json'))

ok = sum(1 for d in s if d['passed'])

total = len(s)

regs = b.get('regressions', [])

print(f'graded: {ok}/{total} passed; regressions vs prior: {len(regs)}')

sys.exit(0 if ok == total and not regs else 1)

"

```

The aggregator compares against the most recent prior `benchmark.json`

automatically. `run.py` exits non-zero if any assertion that was 100%

in the prior run is now <100%, or any all-clean scenario now has at

least one fail.

### Audit signals if a run looks off

```bash

RUN=tests/wiki/layer2/workspace/run-<ts>

# Any subprocess that errored

find "$RUN" -name timing.json -exec grep -l '"claude_rc": [^0]' {} \;

# Any captured stderr (empty stderr files are normal/expected)

find "$RUN" -name stderr.txt -size +0

# Any graded failure

find "$RUN" -name grading.json -exec grep -l '"passed": false' {} \;

```

If all three return empty and `regressions: []`, the run was genuinely

clean.

### Per-pass timeout

`run.py --timeout` defaults to 600s. The import/wrapup scenarios

(`WI-*`, `WU-*`) regularly push past 400s and can flirt with the

default under load. If a run shows transient single-pass fails that

recover on retry, prefer `--timeout 900` over chasing the symptom.

Every Layer 2 scenario asserts:

- `no_files_outside_sandbox`

- `real_home_wiki_absent`

These are the fail-safes that justify running on the operator's real

filesystem rather than a container. They've never failed in any run,

but they're the reason the harness can be safely re-run without

isolation. Don't drop them when adding scenarios.

Two surfaces, each in its own subdir:

| Surface | Where | What it tests | Runner |

| --- | --- | --- | --- |

| `script_tests/` | `prepare_commit_context.sh`, `commit_with_message.sh` | Stdout / exit-code / git-state on real working trees | `./tests/git_commit/run_all.sh` |

| `evals/` | Skill agent behavior | Primary workflow + commit-message format + fallback discipline | Operator-driven: `stage.sh` → agent runs → `grade.sh` |

```bash

./tests/git_commit/run_all.sh

```

Stages a fresh per-scenario temp git repo under

`script_tests/scratch/<id>/`. ~1 sec. No LLM cost.

nothing under `tests/git_commit/` copies into or patches it. All

harness logic stays in this directory.

```bash

# 1. Stage one fixture; exports shell-safe name=value lines

eval "$(bash tests/git_commit/evals/stage.sh <eval_id>)"

# 2. Have an agent (in this session) load $skill_path and apply it to

# $sandbox_repo with the prompt in $prompt. The agent uses its own

# Skill/Read/Bash tools — there's no orchestrator that drives it.

# 3. Grade the post-run sandbox programmatically

bash tests/git_commit/evals/grade.sh <eval_id> "$sandbox_repo"

```

`evals/README.md` has the full recipe and the per-eval expectations.

tree to inspect (the older README claimed there was; it's been

corrected).

### What success looks like

- `run_all.sh` exits 0 with all `script_tests` scenarios PASS.

- For each behavioral eval: `grade.sh <id> "$sandbox_repo"` exits 0

and prints PASS for every expectation.

### Scope discipline

Ship the tests a change needs in the same session as the change: a

skill change lands with the tight new scenario(s) that prove its own

behavior, and you run `./tests/git_commit/run_all.sh` to confirm no

regression before committing. Keep only *unbounded* harness growth for

adding scenarios well past what the change needs. The boundary is

scope, not timing.

Tests a separate axis from the other harnesses: whether a skill's

*description* causes Claude to load the skill on a realistic user

message. Skill *behavior* is tested by `wiki/` and `git_commit/`;

skill *triggering* is tested here.

Each `<skill>.json` is an array of `{query, expected_skill}` entries.

The runner is the local wrapper `tests/trigger_evals/run.py`.

### Eval-set schema

```json

[

{"query": "add a page to my wiki about transformers", "expected_skill": "wiki"},

{"query": "ingest this URL: ...", "expected_skill": "wiki_import"},

{"query": "audit my wiki for broken links", "expected_skill": "wiki_fix"},

{"query": "wrap up this chat into my notes", "expected_skill": "wiki_wrapup"},

{"query": "format the python file at src/loader.py", "expected_skill": null}

]

```

`expected_skill` names the *specific* skill that should fire. Use

`null` to mean "no skill from this family should fire" (a request

that's outside wiki/wiki_import/wiki_fix/wiki_wrapup territory

entirely). For backward compatibility the runner still accepts the

`should_trigger: true` maps to `expected_skill = <--skill>` and

`should_trigger: false` maps to `expected_skill = null`. The legacy

form can't express "trigger a sibling instead," so prefer the

explicit `expected_skill` form when authoring or migrating an eval

set.

### Run pattern

```bash

python3 tests/trigger_evals/run.py \

--eval-set tests/trigger_evals/wiki.json \

--skill wiki \

--model claude-sonnet-4-6 \

--runs-per-query 3 \

--timeout 45 \

--workers 10

```

`run.py` auto-detects whether the skill is deployed:

- **Deployed mode** (default when `~/.claude/skills/<name>/SKILL.md`

exists): spawns `claude -p` for each query × runs-per-query and

watches the stream for the FIRST tool_use being either

`Skill(skill="<X>", …)` or `Read(/<X>/SKILL.md)`. Records *which*

skill `<X>` actually fired, not just "did SOMETHING fire." Warns

if the deployed `description:` has drifted from the source under

`plugins/…/<name>/SKILL.md` (run `make deploy` to resync).

- **UUID-proxy fallback** (when the skill is NOT deployed): delegates

to skill-creator's `run_eval.py`, which writes a temp slash command

under `~/.claude/commands/<name>-skill-<uuid>.md` carrying the

description being tested and watches for the UUID in `Skill` /

`Read` inputs. Useful for testing a description *before* deploy.

reports only the upstream single-skill pass/fail.

- **Force UUID** (`--force-uuid`): override auto-detect and use the

UUID-proxy path even when the skill is deployed. Rarely useful in

same description through the proxy, the model picks the deployed

one (because names like `wiki` beat names like `wiki-skill-<uuid>`),

and the proxy never gets called, which scores 0/N falsely.

### Family / precise grading (deployed mode)

`run.py` reports two pass rates per run:

`expected_skill`. For `expected_skill: null`, precise = "no skill

was loaded as the first tool."

For `expected_skill: null`, family = "no family member was loaded."

The family list defaults to skills sharing the same name root as

`--skill` (e.g. `wiki` → `[wiki, wiki_fix, wiki_import, wiki_wrapup]`;

`format_python` → `[format_markdown, format_python, format_rust]`).

Override explicitly with `--family wiki,wiki_import` if the

auto-derivation isn't what you want.

A query that says "audit my wiki for broken links" with

`expected_skill: wiki_fix` and an actual triggered run of

`[wiki_fix, wiki, wiki]` will score precise = 1/3 (FAIL at the 50%

wiki-territory). That's the diagnosis the family metric is designed

to surface: "description bleed between siblings" looks very different

from "the skill doesn't trigger at all."

Per-query pass uses a 50% threshold over `runs-per-query` runs.

**Why we don't call skill-creator's `run_eval.py` / `run_loop.py`

directly anymore**: the upstream runner's UUID-proxy mechanism is

incompatible with the user's environment once any skill from this

repo is `make deploy`-installed. The real skill outranks the proxy

and the upstream runner records 0 triggers across the board,

producing a misleading "10/20 passed" (only the should-NOT-trigger

queries trivially pass). The local runner sidesteps that by talking

directly to the deployed skill.

### Reading the trigger-eval result

```text

tests/trigger_evals/results/<skill>/<timestamp>/

├── results.json # {skill_name, mode, family, results: [...], summary: {...}}

└── run.log # config + progress + per-query [P|.][F|.] grid

```

Per-query row schema (deployed mode):

```json

{

"query": "audit my wiki for broken links and missing index entries",

"expected_skill": "wiki_fix",

"triggered_skill_per_run": ["wiki_fix", "wiki", "wiki"],

"precise_triggers": 1,

"family_triggers": 3,

"runs": 3,

"precise_trigger_rate": 0.333,

"family_trigger_rate": 1.0,

"precise_pass": false,

"family_pass": true

}

```

`summary` carries `total`, `precise_passed`, `family_passed`,

`precise_failed`, `family_failed`.

The run.log per-query line uses a two-letter marker `[Xy]` where

`X = P` (precise pass) or `.`, and `y = F` (family pass) or `.`:

```text

[PF] expected=wiki_fix triggered=wiki_fix/wiki_fix/wiki_fix : fix my wiki — it's been a while ...

[.F] expected=wiki_fix triggered=wiki_fix/wiki/wiki : audit my wiki for broken links ...

[..] expected=wiki_import triggered=-/wiki_import/- : the wiki at ~/work/sales-ops doesn't have a page yet ...

```

`run.py` exits 0 when *precise* passes for every query, 1 otherwise.

The graded summary is the verdict; exit code is just a CI signal.

diagnostic, not "good enough." Treat them as "fix the description

bleed," not as "done."

### When to run trigger evals

After material edits to a skill's `description:` frontmatter, after

adding or renaming a sibling skill in the same family, or when

they're slow and the description usually isn't what you just edited.

### Always diff against the prior run: `--baseline`

The absolute pass rate hides drift. The precise rate is noisy (each

query passes on a 50%-over-3-runs threshold), so two runs can report

the same headline number while individual queries move underneath it.

A routing regression once sat unnoticed for 19 days behind a flat

aggregate for exactly this reason: a `description:` change to one

sibling pulled two queries to it (2/2 precise before, 0 across five

runs after), and nothing compared runs per query.

So pass the prior run as `--baseline` on every trigger run:

```bash

python3 tests/trigger_evals/run.py \

--eval-set tests/trigger_evals/task.json \

--skill task --skill-path plugins/ai_dev/skills/task \

--model claude-sonnet-4-6 --runs-per-query 3 --timeout 45 --workers 10 \

--baseline tests/trigger_evals/results/task/<prior-timestamp>

```

`--baseline` accepts the prior run's directory or its `results.json`.

The run then diffs per query on the shared cohort and reports each

`REGRESSION` (a query that passed in the baseline and now fails) and

each `improvement` (the reverse), with the before/after trigger

counts. A regression makes the exit code non-zero, turning the

runner into a real drift signal instead of the always-non-zero

"did every query pass" check it was before. The comparison is also

written into `results.json` under `baseline_comparison`.

A lone per-query flip can still be sampling noise at the 50% threshold

before acting on it. A query that fails persistently and is understood

(name-token dominance, an inline-acting model the runner can't observe,

accepted sibling bleed) carries its disposition as a `note` field beside

its entry in the eval set, so a baseline that already shows it failing

never flags it as new. The detector's own unit tests live in

`tests/trigger_evals/script_tests/run.sh` (hermetic, no LLM cost).

### Acting on family-only passes

If a query passes family but fails precise, the wiki family is

correctly seen but a *sibling* is stealing the trigger. The fix is

usually in the *sibling's* description, not the expected one:

sharpen the sibling away from the territory it's encroaching on.

E.g. "audit my wiki for broken links" with `expected: wiki_fix` but

to NOT claim "audit / lint / fix / health-check" verbs that belong

to `wiki_fix`.

### Gotcha: stale `~/.claude/commands/<name>-skill-<uuid>.md`

If a UUID-fallback run gets interrupted, the temp slash command may

linger and show up in your skill catalog of subsequent sessions. Clean

up with `rm ~/.claude/commands/*-skill-*.md` if you spot leftovers.

Deployed-mode runs leave no temp files.

## Adding a harness for a new skill

1. Create `tests/<skill_name>/` with Pattern A's layout:

```text

tests/<skill_name>/

├── README.md

├── RUNBOOK.md

├── run_all.sh # bundled-script unit tests entrypoint

├── results/

├── script_tests/

│ ├── run.sh

│ └── scratch/<id>/ # transient per-scenario sandboxes

├── evals/

│ ├── README.md

│ ├── evals.json # canonical skill-creator schema

│ ├── stage.sh # optional: stage one fixture

│ ├── grade.sh # optional: programmatic grading

│ └── fixtures/<name>/setup.sh # per-eval sandbox stagers

└── workspace/ # iteration outputs if relevant

```

2. Implement `script_tests/run.sh` first if the skill ships bundled

scripts. Stage a fresh sandbox per scenario; never operate on the

non-zero on any failed assertion.

3. Author `evals/evals.json` and per-eval fixture `setup.sh` scripts

for any skill-prose behavior that scripts can't verify (message

format discipline, fallback behavior, user-prompting). Schema:

`{id, prompt, expected_output, files, expectations[]}` per

`skill-creator/references/schemas.md`.

4. Add per-skill `<skill>.json` to `tests/trigger_evals/` only when

skill that overlaps semantically with an existing one.

5. When designing eval assertions, lean on filesystem-state and

structured-report-field checks first. Reach for free-form

response-text checks only when no other signal captures the

surface most likely to make a real change look like a regression.

6. If the harness runs the skill against a real filesystem (no

"no files modified outside the sandbox," "no writes to

`$HOME/<destination>`." See `wiki/` Layer 2 for the load-bearing

pattern.

## What "all green" actually guarantees

- **Bundled scripts** (Layer 1 in `wiki/`, `script_tests/` in

`git_commit/`): the mechanical surface of the skill behaves as

specified for every input shape we test.

- **Behavioral evals** (Layer 2 in `wiki/`, `evals/` in

`git_commit/`): the agent followed the skill's load-bearing

composed conformant output, honored refusal/fallback rules, didn't

leak outside the sandbox.

- **Trigger evals** (`trigger_evals/`): the skill's description is

selective enough on `should_trigger: false` cases and inclusive

enough on `should_trigger: true` cases.

It does NOT guarantee the skill's prose advice leads to good

end-user output on real-world content. That's a quality question

best evaluated by humans on real artifacts.