CLAUDE.md@eval git:20261010.8b1b8e5A

Immutable. This exact content never changes and is served at /api/v1/blob/16fb53c8c3c1d03c.

# Working on `eval/`

Harnesses here call the model, often hundreds of times, on the maintainer's own
subscription. Read `README.md` before adding or running one.

- **Track tokens in every eval, always.** Count tokens as well as calls: each
  `claude -p` call carries ~34k tokens of Claude Code context, so call counts
  understate the cost several times over. Run single-shot calls with
  `--output-format json` and pass stdout through `usage.unwrap()` from
  `usage.mjs` (GEPA's Python runner does this in `Budget.add_usage`); read
  session logs with `usage.fromStream()`. A new harness that calls `claude`
  without recording usage is incomplete.
- **Publish the numbers.** A dated result file states input tokens (and how many
  were cached), output tokens and list-price cost, or says explicitly that they
  were not recorded.
- **Make long runs watchable.** A run over ~20 calls writes a progress file that
  `dashboard.mjs` reads (see `Budget.track` in `gepa/pilot.py`).
- **Everything is manual.** Nothing here runs in CI or from a hook. Say how many
  calls and tokens a run costs near its command, and give it a hard cap
  (`--limit`, `--max-model-calls`) so a first run is small.
- Results under `eval/results/` are historical evidence: correct them with a
  dated note, do not rewrite them.
- Use the real planner and shipped prompts rather than restating them, and
  report unfavourable results too.