AGENTS.md@evals ยท diff
git:20261002.a1cd5d0 to git:20261002.6fceb2e
21 added, 4 removed. Audit A to A.
# Promptfoo evals
Run from the repo root with Node 22.22+, authenticated Claude Code and `gh`:
```bash
npm --prefix evals ci --ignore-scripts
uv run python evals/prepare.py
npm --prefix evals run eval -- --output ../.tmp/evals/results.json
```
The command compares historical and current guidance, three attempts each.
Historical failures are expected; they make Promptfoo exit nonzero. Inspect the
JSON's per-provider results: execution errors are not behavioral failures.
To narrow a run, append `--filter-pattern Worktrunk` or `Leaf`.
Use `--filter-providers current` to check only current guidance.
Promptfoo's built-in Claude Agent SDK provider forks the prepared history for
- each attempt. It runs in an empty temporary directory with Read, Skill and Write,
+ each history attempt; fixture cases start without a history. It runs in an empty
+ temporary directory with Read, Skill and Write (and Grep for fixture evidence),
no discovered settings or MCP servers, and reads confined to that directory and
- its staged plugin. Existing Claude authentication is used without copying
- credentials. Write is permitted only for `captured.md`. A shared transform
+ its staged plugin and any staged fixture evidence. Existing Claude authentication
+ is used without copying credentials. Write is permitted only for `captured.md`.
+ A shared transform
extracts its last completed Write from the SDK trace; a missing or failed write
is an execution error. Inline `llm-rubric` assertions grade the literal body
against prose criteria, using Claude Sonnet 5 through the same SDK provider and
local login. Grading adds model calls and can vary between runs.
Preparation replaces staged plugins; never run it during an eval. These cases
- resume the drafting decision, not repository edits or live GitHub actions.
+ exercise drafting decisions, not repository edits or live GitHub actions.
A no-plugin comparison would still contain previously loaded skills in history.
Deterministic checks run with:
```bash
uv run pytest evals/test_prepare.py
npm --prefix evals test
```
## Add a case from history
Start with an issue/PR URL, a time with its time zone, and the behavior to check.
Use `/install-tend:debug-tend-run` to inspect the original Claude session.
Thread artifacts are named `claude-session-logs-n<NUMBER>`; scheduled runs use
invocation IDs, so also inspect runs around the supplied time.
Keep only `cases/<name>/source.json` and `case.yaml` in Git:
- `source.json` pins the repository, run, artifact, transcript path and SHA-256,
historical Tend ref, original model, bot, injected skills, and an exclusive
`before` timestamp. Choose a cutoff immediately before the decision, with
completed tool calls and no original bad answer. Recover the actual appended
system prompt; pin its hash if it differs from the historical shared prompt.
- `case.yaml` is a Promptfoo test: `description`, `vars.task` and `assert`.
Ask for the next artifact using retained evidence and a Write to `captured.md`;
keep expected behavior in prose assertions, out of the task, for example:
```yaml
assert:
- type: llm-rubric
value: The description references the partially addressed issue without closing it.
```
File references
resolve relative to `.tmp/evals/prepared/promptfooconfig.yaml`.
Prepared configuration pairs the test only with its own historical/current
providers.
Preparation downloads and verifies the source, cuts history, and stages both
plugins. It replaces injected Tend skill bodies in the current arm, retaining
investigation and consumer guidance. Confirm historical runs reproduce the
failure before claiming an improvement. Small samples establish reproduction,
not rates. Calibrate new graders against known bad and good artifacts.
Sources are cached by transcript hash under `~/.local/share/tend/evals/sources/`,
outside the repo and across worktrees. GitHub artifacts expire: a fresh machine
needs a surviving artifact or a copy of that cache. A URL and time cannot
recover expired logs or reconstruct edited GitHub state. This is an agent
case-authoring recipe; the downloader itself is deterministic.
+
+ ## Add a fixture reconstruction
+
+ Use `cases/<name>/source.json` with `kind: fixture`, `historical_ref`, `model`,
+ `bot` and `fixtures`: a mapping of case-relative evidence paths to SHA-256
+ hashes. Commit local evidence beside `case.yaml`. A `logs` entry for a fixture
+ path may instead name `repository` and `run`; preparation fetches that run's
+ failed log into the same hash-verified evidence cache. See
+ `cases/ci-failure-attribution/source.json`.
+
+ The test keeps the same `vars.task` and `assert` format. Its prompt receives the
+ staged evidence directory, and both arms read identical inputs through Read or
+ Grep. Do not put the expected answer in the task or evidence-selection hints.
+ Calibrate the rubric on known bad and good artifacts. A reconstruction that
+ both arms pass provides coverage, without demonstrating improvement.
## Prototype status
The Promptfoo cutover reproduced both historical failures in 3/3 attempts each.
Current guidance passed partial-close 3/3 and wrapping 2/3: one current draft
still hard-wrapped prose. These counts establish reproduction, not error rates.
The prose criteria matched all 19 known verdicts in a calibration: the 12 saved
drafts, the original bad wrapping, two valid formatting controls, and four
issue-reference edge cases. This is a single judge pass, not a reliability rate.
The file-writing step matters; returning a final answer instead failed to
reproduce wrapping in the first comparison.
Future sessions may need to refine cutoff selection, prompt recovery and skill
replacement for new session formats. Only injected guidance is replaced;
instruction source and diffs read through tools remain evidence. Inspect those
for answer leakage or conflicting historical guidance before trusting a case.
Promptfoo also has a Codex SDK provider, but Codex cannot natively resume these
Claude histories. A Codex case needs its own evidence and failing control.