AGENTS.md@evals · git:20261110.381db25 · 2026-11-10 · sha256 3c1bef8f388721c1

AGENTS.md@evals git:20261110.381db25A

Immutable. This exact content never changes and is served at /api/v1/blob/3c1bef8f388721c1.

# Evals Guide

`evals/<case>/` holds end-to-end cases for `claude plugin eval`: a realistic request
with invented data (`prompt.md`), and graders for skill trigger, PDF (or HTML)
creation, and unfilled placeholders, plus fact fidelity in the HTML. Fact fidelity is
a `regex` grader wherever the facts are literal (one-pager-en, resume-cn, long-doc-cn,
equity-report-cn, letter-ko): LLM judges split on derived figures and reported false
FAILs on outputs whose facts were all present. What a judge was asked beyond the facts
is a `not_contains` regex too where it can be: invented contact details and non-Korean
copy in letter-ko, outside companies and consensus in equity-report-cn. A `focus` on a
long HTML file drew unanimous FAIL votes on outputs a standalone sonnet judge passed
with every figure rechecked, matching the harness docs' advice to keep
`llm` graders for short outputs. Invented numbers in long-doc-cn are `score.py`'s
`invented` gate: every number with a unit must be supplied, or for % and 倍 a ratio or
percent change of two supplied ones; it catches invented seconds, 万, 元 and 台 but
not an invented percentage, since ratios of a dozen inputs reach most integers.
slides-en keeps the one LLM fidelity judge, which has passed every run on its deck.
`evals/score.py` then applies Kami's own gates to each kept run workspace (page
contract, atomic facts in the PDF text, fonts, density, residue, style, resume
balance, generated chart count), which the harness cannot run itself; it is
read-only unless given `--write`, which saves `gate-scores.json`. One suite run is 10
cases, 20 document builds at `--runs 2`, at roughly $1-3 each.

- landing-page-en names Kami in its prompt because the description only claims a
  Kami landing page; an unnamed landing-page request fired the skill in 3 of
  4 runs, so the case measured trigger noise instead of the template.
- `--case` takes one glob, and a second `--case` replaces the first:
  `--case long-doc-cn --case equity-report-cn` ran only equity-report-cn.
- A file grader takes an exact path; globs such as `*.html` fail with "path does not
  exist". Every prompt pins its output file names for that reason.
- Kept workspaces live under `/private/tmp/e-*/sealed/home/cwd` with mode 000;
  `score.py` opens them itself. Results stay in the ignored `evals/results/`.
- Mathematics is not covered: the sandbox cannot install the MathJax runtime.
- LLM fidelity rubrics carry the case's full brief, because a judge that sees only a
  partial fact list fails supplied facts as invented. On artifacts over about 15k
  characters the harness judge is unreliable: before treating its FAIL as real,
  re-judge the stored `evidence` HTML with the same rubric (`claude -p --model sonnet`)
  and read the stated reason.
- resume-cn deliberately supplies thin notes (three employers, eight bullets). The
  expected behavior is two honest pages with `[DATA NEEDED]` gaps and a reported
  balance shortfall (`references/resume-writing.md` «Thin source»). `--deliver` and
  `score.py` both read that through `checks.resume_thin_source` and treat it as a
  reported gap rather than a balance failure.