git:20261017.8be9de3 to git:20261017.10def4c

---

name: write-skill-eval

description: >-

Repo-only guide to writing an eval case for one of this devkit's skills, running it with

and without the plugin, and proving each grader can fail before trusting its score. Use

when the user asks to add, write, or fix an eval, test whether a skill earns its place,

or close a gap that `scripts/check-evals.sh` reports. Not shipped with the plugin.

---

# Write a skill eval

An eval case earns its place when plain Claude fails it and the skill fixes that. The

process: start from a real failure, set up the scenario, prove plain Claude fails, write the

smallest rule that fills the gap, prove the skill fixes it and actually loaded, then keep the

case as a guard. Follow the eval rules in `CLAUDE.md` ("No build, lint, or test step")

throughout; this skill is the order to apply them in.

## 1. Start from a real failure

Begin with something Claude got wrong, not with the skill's text:

- **Knowledge skill:** a bug that shipped in a package, or a mistake Claude was seen making.

Tag a shipped bug's case `real-bug`. Writing a case from a rule you already have only

shows the rule gets loaded, not that it's needed.

- **Workflow skill** (`feature-*`, `quick-fix`, `issue-*`, ...): a rule the skill must keep,

such as stopping when its input is missing or never touching git.

One case tests one behavior. If you find two, write two cases.

## 2. Write the prompt

Create `plugins/umbraco-devkit/evals/<case-name>/prompt.md`. Name the case like its

neighbors: the skill's short name, then what it tests (`env-sync-access-rules`).

```markdown

---

description: <one line: what this case proves>

tags: [<skill-name>, should-trigger]

# covers: ["<the skill section it tests>"]

max_turns: 30

timeout_seconds: 900

allowed_tools: [Read, Glob, Grep, Skill, Write, Edit]

---

<the request, as a package developer would type it>

```

- **Ask for the change, not a review.** Given a short snippet to review, plain Claude

catches most mistakes. Review only a realistic multi-file diff, as in

`reviewer-feature-diff`.

- **Set up the scenario, never ask for the fix.** Ask for something a developer would

really ask for (a feature, a change, a job to add) in a project where the obvious answer

brings the problem in. A prompt that points at the problem tests debugging, which plain

Claude already does well, not whether the skill stops the problem happening. For a

`real-bug` case, ask for the change that first introduced the bug, not the fix.

- **State the setup facts the trap needs**, as a developer would: "sites are load

balanced", the target versions, "other packages call this". Without them the trap

isn't real, and a careful answer could fail for not guarding against something the

setup never showed.

- **Never** describe the symptom, ask for a fix, name the risk or the right API, or bait

the answer ("I think it's fine", "make sure it's secure").

- **Diagnosis cases** (symptom in, cause out) get the `diagnose` tag. Keep them rare;

they don't show a skill prevents anything.

- **Workflow cases** for bug-fixing skills (`investigate-bug`, `quick-fix`,

`issue-resolve`) are handed a bug, because the workflow's own rules are what's tested.

- **Say what the sandbox lacks.** Git, dotnet and the network don't run in an eval. Say so

in the prompt ("Git can't run here") so the answer doesn't stall trying.

- **Tags:** the skill's name, then `should-trigger`, `should-not-trigger`, or `workflow`.

Add `real-bug` when it applies.

- **Covers:** for a knowledge case, add `# covers: ["<heading>"]` after `tags:`, naming the

skill's `## ` section (in `SKILL.md` or `references/`) that holds the rule. Italic headings

(`## *Background*`) are context and take no eval. Use `[]` for a

guard whose rule was removed. Keep the `#`: the eval tool rejects frontmatter keys it

doesn't know, so the line is a YAML comment. `scripts/eval-coverage.sh --check` fails on a

heading that doesn't exist.

For a should-not-trigger case, write a nearby request the skill should stay out of.

## 3. Scaffold the project, if the task needs files

When the answer has to read or change code, add `fixture.sh` (bash that writes the starting

files) and a `case.yaml`:

```yaml

schema_version: "1.1"

name: <case-name>

context:

scaffold_script: fixture.sh

```

Keep the fixture to the few files the task touches, but real enough that the trap lives in

the code, not the prompt.

## 4. Write the graders

One file per check in `graders/`. Each checks one thing and passes every valid fix, not only

the one you had in mind.

- **Did the skill load:** a `skill-fired.md`, copied from a neighbor case. Under the default

with/without run it reports whether the skill fired and isn't scored. For a

should-not-trigger case, use `min: 0`, `max: 0`, `arm: both`.

- **Mechanical checks:** `type: regex` or `type: file_exists`. Free and exact; prefer them

when they're enough.

- **Judgment checks:** `type: llm` with a `PASS if ...` and a `FAIL if ...` paragraph. Use

`focus: trace` to grade the whole answer, or `focus: {source: file, path: ...}` for one

written file. Give the grader for the core behavior `weight: 2`.

## 5. Prove plain Claude fails

Run the case once with a single run per arm:

```bash

claude plugin eval plugins/umbraco-devkit --case <case-name> --runs 1 \

--judge-model sonnet --no-publish --scaffold --allow-tools Write Edit WebFetch

```

Drop `--scaffold` for a case with no `fixture.sh`. Read the run's warnings first: one saying a

grader "cannot pass with the granted tools" means the run measured nothing.

- **A grader that passes without the plugin measures nothing.** Make the prompt or grader

harder, or delete the grader.

- **If every grader passes without the plugin**, the case is too easy. Rework it in step 2.

- **Try a second scenario before giving up on a gap.** Plain Claude often passes the obvious

version and fails a variant: sibling code already doing it wrong, the risky part buried in

a bigger change, a related API. If it passes every variant you can think of, no rule is

needed; stop here.

- **Workflow cases** run with `--ablation none`, so there is no baseline. Read each grader

against an answer that breaks the rule; if it would still pass, tighten it.

## 6. Write the smallest rule that fills the gap

Skip this if the skill already has the rule and only needs to load (step 7).

- **General, not project-specific.** The rule describes Umbraco or stack behavior any

package hits. Leave out the names and conventions of the project the bug came from.

- **Checked against the source.** Confirm the behavior in CMS source at the target release

tags (`umbraco-cms-source`) or the docs (`umbraco-docs`), including from which version it

applies.

- **In the right skill.** Follow "Skill boundary discipline" in `CLAUDE.md`: fit the

skill's description, write only what differs from the norm, and grep for the concept

first.

## 7. Prove the skill fixes it, and loaded

Run it with the default three runs per arm, Sonnet judging:

```bash

claude plugin eval plugins/umbraco-devkit --tag <skill-name> --judge-model sonnet \

--no-publish --scaffold --allow-tools Write Edit WebFetch

```

Three runs are noisy. Don't judge a close or zero gain on one run: rerun with `--runs 6`

first.

A pass with the plugin counts only if the skill loaded: check `skill-fired` in the log. If

the plugin passes but the skill didn't load, the gain came from somewhere else. If the

skill doesn't load, fix its description's "Use when" so it matches the request, not the

rule. Then rerun.

Check the log for errored runs too (auth, spend limit). An errored run is neither a pass nor

a fail; don't count it.

## 8. Finish, and keep the case as a guard

- Run `scripts/check-evals.sh`.

- Record the run: `scripts/record-eval-scores.sh` (newest results by default), then

`scripts/eval-coverage.sh`. Commit `docs/eval-scores.json` and `docs/EVAL-COVERAGE.md`;

CI fails if the report is stale.

- Put the with and without scores in the PR. A knowledge skill that shows no gain doesn't

ship.

- `evals/` ships but never loads for a consumer, so a new case needs no version bump.

- Rerun the case whenever its skill changes. Plain Claude improves, so recheck old cases'

baselines now and then.

## Removing a rule

A rule goes only when plain Claude reliably gets it right with no prompting, across varied

scenarios where the rule applies, not just the existing case.

1. Write at least one harder variant (step 5's list).

2. Run plain Claude only, 4 runs per scenario. To skip the with-plugin arm, point the eval at

a copy of the plugin with no skills: a folder holding only `.claude-plugin/plugin.json`

(no `skills` entry) and a real copy of `evals/` (the tool refuses a symlink), run with

`--ablation none`.

3. Stop at the first failed run: one miss proves the rule is needed.

4. Remove the rule only if every graded run passed. Keep its cases as guards.