--- name: evals description: Author an Interlock eval case from an observed model-behaviour failure, and read a completed eval results run. Use when you have a reproduced failure — an archived proposal, a changelog entry, a run artifact, or a transcript — worth regression-testing against a real model, or when an eval run has finished and you need to know whether it is a regression, run-to-run variance, or no signal. Authoring needs cited evidence; the verdict comes from `interlock evals triage`, never from your own reading. license: MIT compatibility: Requires the early-access harness (claude plugin eval, enabled with CLAUDE_CODE_WALNUT_SPIRE=1) to RUN a suite. Authoring a case and triaging an existing results file need only Node.js >= 18 and the bundled interlock CLI. allowed-tools: Bash(interlock *) Bash(claude plugin eval:*) Read Grep Glob Write metadata: type: authoring outputs: - evals//case.yaml - evals//graders/*.md --- This skill covers the two jobs the eval harness does not do for you: turning an observed failure into a case, and reading a finished run. Everything with a correct answer — is this a regression? — belongs to `interlock evals triage`, not to you. Your job is evidence, grader choice, and explanation. ## 1. Authoring a case — only from an observed failure A case regression-tests a **reproduced** failure: a model read correct instructions and did the wrong thing anyway. Before you write anything, require the evidence: - an archived proposal (`openspec/changes/archive/*/proposal.md` `## Why`), - a `CHANGELOG.md` entry, - a run artifact or a transcript, - a **captured skeleton** — `interlock evals capture --run [--task ] --out ` turns a recorded ship-run trajectory into a case draft whose provenance names the trajectory and the events it was derived from, or - a `file:line` span in the prompt or skill under test that shows the surface the model talked past. **A captured skeleton is a draft, not a finished case.** It is admissible evidence because it is a run artifact and a reader can follow its citation back to the trajectory. But every value capture derived rather than read carries a `CONFIRM` marker — the grader pattern when no value could be quoted, and the prompt when the trajectory recorded none. **Resolve every `CONFIRM` marker against the cited trajectory, or ask the user, before authoring the case. Never author a case that still carries one**, and never move a skeleton into `evals/` unresolved: an unconfirmed pattern is a regression test for a failure nobody has shown happened. **If no observed failure is offered, say so and stop.** Do not author a speculative case. A suite of hypotheticals teaches the team to ignore it, which is the one thing an advisory gate cannot survive. A case that only duplicates an existing `node:test` assertion is refused for the same reason — the unit suite already covers it deterministically. Record the citation in the case (`provenance:` in `case.yaml`, plus a comment explaining what went wrong). The structural gate `test/evals.test.mjs` fails a case with no provenance. ### The case files Every case pins `schema_version` and lives in its own directory under `evals/` — never under `skills/`, `agents/`, `commands/` or `workflows/`, which would shadow a component. Follow `evals/handoff-status-enum/` as the format exemplar; it was written to establish these conventions. Where a case needs a fixture workspace, use `case.yaml` with `context.scaffold_script` and run with `--scaffold`; otherwise the same `case.yaml` shape without it. ## 2. Choosing a grader — cheapest that expresses the assertion The harness has six grader types. Four are free and deterministic — `regex`, `tool_used`, `tool_order`, `file_exists` — and two call a judge — `llm`, `baseline`. **Pick the cheapest grader that expresses the assertion.** If the assertion is set-membership, tool-invocation, file-existence, or ordering shaped, it is deterministic and a judged grader is the wrong tool — more expensive, and prone to run-to-run variance triage will (correctly) refuse to call a regression. The strongest seed case, `handoff-status-enum`, needs no judge at all: the status is in a fixed set, so a `regex` decides it. **When you do reach for a judged grader, state in the grader body which part of the assertion could not be expressed deterministically.** If you cannot name that part, the assertion was deterministic and you chose the wrong grader. A case you intend to run per-pull-request tag `smoke`, and a smoke case uses **only** deterministic graders — the gate enforces this. A grader that asserts a skill *fired* (`tool_used: Skill`) is marked `arm: with-only`: it is a plugin-fired indicator, reported against the baseline arm, not part of the score. ## 3. Reading a run — triage owns the verdict Given a finished results file, **get the classification from `interlock evals triage --results `, do not derive it yourself.** Its exit code is the verdict: `0` pass, `1` regression, `2` no signal, `3` configuration. This is deliberate — the one gate whose rule a model could re-argue on each run would be exactly the wrong gate to ship in a suite built to catch models re-arguing rules. Your contribution is **explanation, not classification**: - When triage reports a **regression**, explain which recent change to the surface under test plausibly caused it — the prompt sentence, the skill edit, the contract change. Do not reclassify it as variance because it looks flaky; triage already separated a single judged dip (variance) from a judged majority (regression). - When triage reports **variance**, report it as variance and say what would confirm it (further runs failing the same judged grader). - When triage reports **no signal** — a partial run, an interrupted run, a rejected credential — report that evals could not run. Do not present any case as passing or failing. Never restate, override, or soften a verdict triage gave you. ### Read a transcript before you explain a judged case **For every case whose score depends on a judged grader, read at least one transcript for that case before you write your explanation of it.** A score is the outcome; the trace is the only place the cause is visible. The failure this closes is a model that satisfied a judge by narrowing the task rather than by doing it — the score moves, every other step in the loop sees a number, and nobody sees why. Reading is for explanation and **never for reclassification**. If the transcript suggests a different classification than triage produced, report what you saw in the trace and leave the verdict alone — the rule above still holds without exception. A run whose graders are all deterministic needs no transcript: the tool trace the grader already read is the evidence. Your report carries a `transcripts read:` line, always present: ``` transcripts read: /, / transcripts read: none transcripts read: unavailable, because ``` Use the **none** form when you read no transcript — never omit the line, because a reader cannot otherwise tell an unexamined result from an examined one. Use the **unavailable, because …** form when the run produced no readable transcript, and in that case do not present your explanation as transcript-grounded. Degradation is spoken. ## 4. A failing case is a report, never a prompt edit When a case fails, your output is a report: the case, the grader that failed, and the sentence in the surface under test that plausibly caused it. **Do not edit the skill, shared contract, agent definition, or prompt under test to make a case pass.** Changing the surface under test is a separate, explicit decision the user makes — silently editing a prompt to turn a case green is exactly the "edit the test until it passes" failure the suite exists to prevent, one layer up. ## 5. When the harness is gated The host command claude plugin eval prints that it is currently in early access and does nothing until `CLAUDE_CODE_WALNUT_SPIRE=1` is set in the environment — never in committed `.claude/settings.json`, which would produce a suite that looks configured and does not run. When the harness refuses because early access is not enabled, **report the prerequisite and where to set it** (the environment, or the CI job's `env:` block). Do not report an empty run as a passing suite. Authoring a case and triaging an existing results file both work without the harness enabled — only running a suite needs it.