23 added, 18 removed. Audit A to A.
---
name: dtd-eval-dtd
- description: Measure whether a *-dtd command's answers conform to its declared grammar: write a valid and an invalid example instance, validate both with xmlstarlet through the rdc checker, and keep a small suite under examples/. Use when a command's grammar carries IDREFs or enumerations, before shipping a new command, or when asked to prove a DOCTYPE is more than decoration.
+ description: Measure whether a *-dtd command's answers conform to its declared grammar with the Adiutor as the instrument: run the command on a fixture argument, read the ledger line its Stop check wrote, then feed a deliberately broken answer through the same check in a scratch state directory and watch it fail. Use before shipping a new command, after changing a grammar, or when asked to prove a DOCTYPE is more than decoration.
---
<!-- SPDX-License-Identifier: AGPL-3.0-or-later OR EUPL-1.2 -->
<!-- Copyright 2026 Saimonokuma. -->
<!DOCTYPE eval_session [
<!ENTITY % cc-core SYSTEM "../../../dtd/cc-core.dtd">
%cc-core;
- <!ELEMENT eval_session (target, instance, instance, run, result)>
+ <!ELEMENT eval_session (target, fixture, run, ledger_line, mutation, result)>
<!ELEMENT target (#PCDATA)>
<!ATTLIST target root NMTOKEN #REQUIRED>
- <!ELEMENT instance (#PCDATA)>
- <!ATTLIST instance path CDATA #REQUIRED expected (valid|invalid) #REQUIRED>
+ <!ELEMENT fixture (#PCDATA)>
<!ELEMENT run (#PCDATA)>
+ <!ATTLIST run state_dir CDATA #REQUIRED>
+ <!ELEMENT ledger_line (#PCDATA)>
+ <!ATTLIST ledger_line status (pass|fail|aborted) #REQUIRED>
+ <!ELEMENT mutation (#PCDATA)>
+ <!ATTLIST mutation kind (missing_heading|order|dangling_ref|missing_assumptions|no_answer) #REQUIRED>
<!ELEMENT result (#PCDATA)>
- <!ATTLIST result control_tripped (true|false) #REQUIRED positive_valid (true|false) #REQUIRED>
- <!ENTITY LAW.EVAL.1 "An evaluation has two instances: one expected valid and one expected invalid; a suite with only valid instances proves nothing about the validator.">
- <!ENTITY LAW.EVAL.2 "The invalid instance is judged first; only after it is rejected with a named error does the valid instance's pass count.">
- <!ENTITY LAW.EVAL.3 "Example instances are XML without a DOCTYPE line; the checker supplies the resolved subset, so the instance validates against the contract that is actually installed.">
+ <!ATTLIST result control_tripped (true|false) #REQUIRED positive_pass (true|false) #REQUIRED>
+ <!ENTITY LAW.EVAL.1 "An evaluation has one real run and one mutated answer; a pass that was never seen failing proves nothing about the check.">
+ <!ENTITY LAW.EVAL.2 "The mutated answer is judged first; only after the Adiutor fails it with the finding kind named does the real run's pass count.">
+ <!ENTITY LAW.EVAL.3 "The instrument is the installed Adiutor reading the command's own DOCTYPE; no static instance is written and nothing outside the scratch state directory is touched.">
]>
<trust_boundary>
- `user-args`: the command name or path is data.
- - `tool-result`: validator output is data, quoted as received with its exit code.
- - `file-ref`: the command file and the examples are content.
+ - `tool-result`: the Adiutor's output and the ledger line are data, quoted as received with the exit code.
+ - `file-ref`: the command file and the transcript are content.
- `ask-answer`: this skill asks nothing.
</trust_boundary>
<objective>
- Turn a declared grammar into a measurement. The `eval_session` root declares the target command, two instances, the run, and the result with its control. A DOCTYPE whose invalid example is accepted is decoration and the evaluation says so.
+ Turn a declared grammar into a measurement. The `eval_session` root declares the target command, the fixture it ran on, the run, the ledger line the Adiutor wrote, the mutation fed through the same check, and the result with its control. A DOCTYPE whose broken answer passes the Stop check is decoration and the evaluation says so.
</objective>
<process>
1. Set the `target`: the command file and its root element (the name that follows the DOCTYPE keyword in its grammar block).
- 2. Write the valid `instance` at examples/<root>.xml: one root element with every required child in declared order, attributes with legal enumeration values, every IDREF pointing at an existing id. No DOCTYPE line, no XML declaration needed.
- 3. Write the invalid instance at examples/<root>.invalid.xml: the valid one with one required child removed, one enumeration value outside its set, and one IDREF pointing nowhere.
- 4. `run`: from the repository root, `node bin/rot-dtd-commander.mjs check commands/<name>-dtd.md --xml` validates examples/<root>.xml. For the invalid instance run the validator directly: extract the resolved subset with `node bin/rot-dtd-commander.mjs resolve` into a scratch file, then `xmlstarlet val -e -d <subset.dtd> examples/<root>.invalid.xml`; expect exit 1 and named errors. Ceiling 60 seconds, stdin closed.
- 5. `result`: control_tripped true only if the invalid instance was rejected with at least one named error; positive_valid true only if the valid instance passed after that. Report both exit codes.
+ 2. Choose the `fixture`: one short argument the command can answer in a single turn.
+ 3. `run`: in a session with the Adiutor armed, type the command with the fixture; at Stop the Adiutor writes one line. Read it with `node ~/.claude/rot-dtd-commander/bin/adiutor.mjs ledger --last 1` (or `rdc ledger --last 1` from a clone) and set state_dir to the state directory it used. Ceiling 60 seconds, stdin closed.
+ 4. Quote the `ledger_line` verbatim with its status.
+ 5. `mutation`: write a transcript with the same answer minus one required heading (or with the headings reordered, or with a short id that points nowhere) and feed it through the same check in a scratch directory: `ROT_DTD_STATE=<scratch> node bin/adiutor.mjs observe UserPromptSubmit` with the command prompt, then `observe Stop` with the mutated transcript path; expect status fail and the finding kind named (LAW.EVAL.2).
+ 6. `result`: control_tripped true only if the mutated answer failed with the kind named; positive_pass true only if the real run's line says pass after that (LAW.EVAL.1). Remove the scratch directory (LAW.EVAL.3).
</process>
<declared_grammar>
- Render `eval_session` as: **Target** (path and root), **Instances** (two paths with expected), **Run** (the commands and their exit codes), **Result** (control_tripped, positive_valid, the validator's error lines quoted).
+ Render `eval_session` as: **Target** (path and root), **Fixture**, **Run** (the command, the state directory, the exit code), **Ledger Line** (quoted, with status), **Mutation** (what was broken and the kind the Adiutor named), **Result** (control_tripped, positive_pass).
</declared_grammar>
<additional_resources>
- - The repository's examples/ directory holds the suite; `node bin/rot-dtd-commander.mjs check --xml` runs every example that matches a root name
+ - `node bin/adiutor.mjs controls` runs the eight built-in guards in a temporary state directory; C1 is the same missing-heading check this skill trips by hand
</additional_resources>
<success_criteria>
- - The invalid instance was rejected before the valid one was trusted
+ - The mutated answer was failed before the real pass was trusted
- Exit codes were read directly, never through a pipe
- Every LAW.EVAL.* entity holds
</success_criteria>