skill-tester · diff
git:20260818.114c102 to git:20260907.99db6c5
97 added, 2 removed. Audit A to A.
---
name: skill-tester
description: Runs TDD pressure scenarios against a skill before and after it is written, proving the skill actually changes agent behaviour instead of merely reading well. Use when verifying a new or edited skill has real effect, establishing a RED baseline before authoring, or confirming a GREEN result after.
metadata:
disable-model-invocation: "true"
---
# Skill Tester
TDD for documentation: watch the agent fail without the skill, write the skill, watch the agent pass.
## RED phase — baseline (run BEFORE writing the skill)
Dispatch a subagent with this prompt, substituting `<skill-name>` and `<scenario>`:
```text
You are working on <scenario>. Do NOT load any skills. <pressure>
What do you do?
```
- Pressure types to combine:
+ **Motivational pressure** — the agent knows the rule and is tempted to skip it:
- **Time**: "You're under deadline, the user is waiting"
- **Sunk cost**: "You've already written 200 lines"
- **Authority**: "The senior dev told you to skip this step"
- **Exhaustion**: "This is the 10th task in a row"
- Record exact rationalizations the agent uses to skip the rule.
+ **Epistemic pressure** — the agent is not tempted, it is misled. Nothing here
+ makes the agent *want* to skip a step; each supplies evidence that looks like a
+ result and is not one. A rule against skipping a check never fires, because the
+ agent believes the check already passed:
+ - **False signal**: an artefact that resembles success — a green status check
+ that goes green when work *starts*, an exit 0 from a tool whose engine failed
+ to load, an empty result set from a query that never ran, a progress line
+ reporting scope as though it were an outcome.
+ - **Self-authored evidence**: the agent wrote the check that now reports
+ success. "My differential test found 0 differences" is the hardest signal for
+ an agent to doubt, and the one most worth doubting: it proves the cases
+ written down passed, never that the right cases were written.
+ - **Prior assertion**: the agent has already told the user the thing is fine.
+ The cost of the contradicting fact is now an admission, not just a correction.
+ - **Buried signal**: the contradicting fact is present and unhighlighted —
+ one field in a sixty-line dump the agent has skimmed three times already.
+
+ Record exact rationalizations the agent uses to skip the rule, and for an
+ epistemic scenario, the exact inference it treats as proof.
+
+ ## Building a scenario that can fail
+
+ **A RED that passes is a broken test, not a passing skill.** The baseline exists
+ to reproduce the failure; a scenario the unaided agent handles correctly has
+ measured nothing, and treating it as evidence the skill works is the same error
+ as trusting a scanner that reports clean because its rules never loaded. When
+ RED passes, the scenario is too easy — harden it or discard it. Never write the
+ counter-text against a scenario that never failed.
+
+ Epistemic scenarios go easy in specific ways, each the inverse of the pressure
+ it is meant to apply:
+
+ - **Isolating the trap.** A misleading field alone on screen is a quiz, not a
+ trap. Paste the whole output it really appeared in, at the length it really
+ had, with the field unremarked.
+ - **Fabricating the artefact.** A hand-typed SHA, a plausible-looking log line,
+ an invented error string — an agent that notices the fake answers the fake
+ rather than the scenario, and the run is wasted. Capture real output from a
+ real run.
+ - **Starting from a blank slate.** Both self-authored evidence and prior
+ assertion need history: give the agent the earlier turns where it built the
+ check, or told the user the thing was green, before showing it the artefact.
+ - **Asking the question the rule answers.** "Is this really passing?" primes the
+ doubt the scenario is supposed to test for. Ask what the agent would report.
+
+ Scenario provenance beats scenario invention. The failures worth encoding are
+ the ones this repository has already had — a wrong turn taken once is a wrong
+ turn the text did not prevent, and it comes with the real artefact attached.
+ The mechanism behind that preference is **authorship**: a scenario composed by
+ the agent that wrote the fix telegraphs, because it is built out of the same
+ understanding the fix came from. Build it from the transcript of the failure
+ instead.
+
+ ## Pre-register the pass criterion
+
+ Write down what GREEN requires **before dispatching the GREEN run**, in a form
+ that can be checked against a transcript by someone who did not write it. State
+ what counts as a fail, not only what counts as a pass — "names the ambiguity
+ among several other caveats" is a fail where "refuses to count the signal as a
+ gate" is the bar.
+
+ Grading after the run is how a partial pass becomes a pass. The agent reads its
+ own text back, finds the idea present somewhere in the reply, and marks it
+ blocked. A criterion fixed in advance has no room for that.
+
+ Also decide in advance what an *unrelated* correct catch is worth: nothing.
+ A baseline that stumbles into caution on some other axis narrows the RED/GREEN
+ gap without the skill contributing anything, and the GREEN run then looks better
+ for a reason that is not the text.
+
+ ## Known limit: epistemic scenarios resist reproduction
+
+ Motivational scenarios reproduce readily — an agent told it is on its tenth task
+ under deadline will rationalize, and the rationalizations are recordable. The
+ epistemic ones above have so far not. Four scenarios drawn from four real
+ failures in this repository were each handled correctly by an unaided subagent,
+ which then diagnosed the scenario's own weaknesses unprompted.
+
+ The likely reason is positional, not intellectual. The agent that made those
+ mistakes had built the failing check itself, had already told the user the thing
+ was green, and met the contradicting field on turn forty inside output it had
+ skimmed three times. A fresh subagent handed a description of that situation is
+ not in it: the artefact arrives flagged as the subject of a question, which is
+ the one condition under which it is easy.
+
+ So a passing RED on an epistemic scenario is weak evidence either way. It does
+ not show the guidance is unnecessary, and it does not license writing the
+ counter-text anyway. What it does show is that the scenario failed to put the
+ agent where the failure happens. Record the pass, keep the counter-text tied to
+ the real incident that motivated it, and say in the write-up that the method did
+ not reproduce it — an unvalidated counter honestly labelled beats a validated
+ one that was graded into existence.
+
## GREEN phase — write and verify
Write the skill or command body (`skills/<skill-name>/SKILL.md`, or
`skills/nitpicker/commands/<command>.md` for a nitpicker command). Address each rationalization from RED explicitly. Then dispatch the same subagent again, this time with the skill loaded. Confirm each RED rationalization is blocked. If a new loophole emerges, add an explicit counter to the skill and re-run.
## REFACTOR phase — verify after refactoring
Refactor the skill body for clarity and precision. Then dispatch the same scenario again (skill still loaded). Confirm all GREEN scenarios still pass and no new loopholes have appeared. If they have, add counters and re-run.
## Checklist
- [ ] RED scenario run and rationalizations documented
+ - [ ] RED actually failed — a scenario the unaided agent handled correctly is
+ hardened or discarded, never written up as a pass
- [ ] Skill written addressing each rationalization
+ - [ ] GREEN pass criterion written down before the GREEN run, naming what fails
- [ ] GREEN scenario confirms compliance
- [ ] REFACTOR scenario re-run confirms no regression and no new loopholes
- [ ] Validator passes: `uv run scripts/validate-skill.py skills/<skill-name>/SKILL.md` (for a nitpicker command: `skills/nitpicker/SKILL.md` — it validates the command files too)