AGENTS.md@evals ยท diff
git:20261009.95ca1ef to git:20261010.e7fb7ac
3 added, 3 removed. Audit A to A.
# @shadowclone/evals
- Private evaluation suites. They measure whether agents follow preferences, work through pull requests, and review code. Maintainers run them; the CLI does not.
+ Private evaluation suites. They measure whether agents follow preferences and work through pull requests. Maintainers run them; the CLI does not. The code review benchmark is the separate `code-review-bench` repository.
<!-- package-facts:start -->
(generated by `bun run guides`; do not edit by hand)
- Package name: `@shadowclone/evals`
- - Workspace dependencies: `@shadowclone/agents`, `@shadowclone/cli`, `@shadowclone/core`, `@shadowclone/environment`, `@shadowclone/learning`, `@shadowclone/redact`, `@shadowclone/review`, `@shadowclone/sessions`, `@shadowclone/skills`
+ - Workspace dependencies: `@shadowclone/agents`, `@shadowclone/cli`, `@shadowclone/core`, `@shadowclone/environment`, `@shadowclone/learning`, `@shadowclone/redact`, `@shadowclone/sessions`, `@shadowclone/skills`
- External dependencies: `commander`, `typescript`, `zod`
- Export subpaths: none
<!-- package-facts:end -->
## Owns
- `run.ts`: the `bun run eval` entry. `cli/` holds one handler for each protocol.
- `fixed/`: the frozen preference benchmark (`preference-respect-v3`), its cases, graders, and held-out manifest.
- `native/`: native agent runs, private storage checks, and receipts.
- `work/`: the `shadowclone-work` suite for `bun run eval:work`.
- - `pr-review/` and `no-comments/`: the review and comment suites. `shared/`: ledgers, locks, and isolation.
+ - `no-comments/`: the comment suite. `shared/`: ledgers, locks, and isolation.
## Rules
- A phase that calls a model needs an `approve-*` step, `--yes`, and a scope that names the host, model, and call ceiling.
- Private storage cannot be inside a repository. `requirePrivateDirectory` enforces this.
- Held-out cases stay outside the checkout. Only their hashes and family names are public.
- A report stays incomplete while a reserved call has no recorded attempt.
- Do not change a number in `evals.md` or the README results table without a new run.
## Tests
Run `bun test` in this folder, or `bun test evals` from the root. The tests use synthetic fixtures and mock runners, so they make no paid call. Some tests run `git`.
Update this file in the same commit when you change the package purpose, public exports, dependencies, or a rule above.