AGENTS.md@evals git:20261010.e7fb7acA

Immutable. This exact content never changes and is served at /api/v1/blob/583b99525c4d7eb4.

# @shadowclone/evals

Private evaluation suites. They measure whether agents follow preferences and work through pull requests. Maintainers run them; the CLI does not. The code review benchmark is the separate `code-review-bench` repository.

<!-- package-facts:start -->

(generated by `bun run guides`; do not edit by hand)

- Package name: `@shadowclone/evals`
- Workspace dependencies: `@shadowclone/agents`, `@shadowclone/cli`, `@shadowclone/core`, `@shadowclone/environment`, `@shadowclone/learning`, `@shadowclone/redact`, `@shadowclone/sessions`, `@shadowclone/skills`
- External dependencies: `commander`, `typescript`, `zod`
- Export subpaths: none

<!-- package-facts:end -->

## Owns

- `run.ts`: the `bun run eval` entry. `cli/` holds one handler for each protocol.
- `fixed/`: the frozen preference benchmark (`preference-respect-v3`), its cases, graders, and held-out manifest.
- `native/`: native agent runs, private storage checks, and receipts.
- `work/`: the `shadowclone-work` suite for `bun run eval:work`.
- `no-comments/`: the comment suite. `shared/`: ledgers, locks, and isolation.

## Rules

- A phase that calls a model needs an `approve-*` step, `--yes`, and a scope that names the host, model, and call ceiling.
- Private storage cannot be inside a repository. `requirePrivateDirectory` enforces this.
- Held-out cases stay outside the checkout. Only their hashes and family names are public.
- A report stays incomplete while a reserved call has no recorded attempt.
- Do not change a number in `evals.md` or the README results table without a new run.

## Tests

Run `bun test` in this folder, or `bun test evals` from the root. The tests use synthetic fixtures and mock runners, so they make no paid call. Some tests run `git`.

Update this file in the same commit when you change the package purpose, public exports, dependencies, or a rule above.