git:20260922.28cefc1 to git:20260928.7fa6f2a

140 added, 27 removed. Audit A to A.

---

name: arch-benchmark

description: "Run a benchmark scenario (a skill, a command, or a question) in a declared runtime and have frontier models score it against the rubric. Use to measure a change to a skill or the text."

allowed-tools: Read, Grep, Glob, Bash(uv run:*), Bash(ls:*)

disable-model-invocation: true

---

# arch-benchmark

Measure a subject against a rubric and report what the judges said.

The harness is `benchmark/run.py` in a checkout of the guideline

repository, and it is the only thing that runs a benchmark. Do not

compose a provider call by hand, and do not invent a flag: every flag

is in `benchmark/README.md`, and `uv run benchmark/run.py --help`

prints them.

## Where it runs

The current working directory must be a checkout of the guideline

repository: `architecture.md`, `.claude-plugin/plugin.json`,

`benchmark/run.py`, and `benchmark/README.md` all present. Check with

`ls` before anything else. Otherwise stop and say so: the benchmark

measures the checkout it runs from, and writes its run folders under

that checkout's `benchmark/runs/`. The copy under

`${CLAUDE_SKILL_DIR}/../..` is the installed plugin, not a checkout;

never run the harness from there.

## Input

`$ARGUMENTS` is one of:

- a scenario name, with or without flags ("explain-tenancy",

- "review-om with all four judges");

+ "review-om with all four judges"), or the path of a scenario file,

+ which `--scenario` takes as it is;

- a question about what there is ("which scenarios are there?");

- empty, which means: list the scenarios and the provider

availability, and stop.

## Procedure

1. Confirm the working directory is a checkout, as "Where it runs"

says. Every command below runs from it.

2. Run `uv run benchmark/run.py list`. It prints the scenarios with

- their kind and default judges, and each provider with its flag and

- whether its key is present. Report an absent key as absent; it is a

- provider that will be skipped, not a failure. It also shows whether

- the subject's own key, `SUBJECT_ANTHROPIC_API_KEY`, is present; a

- skill scenario needs it, and without it a `--strict` run refuses.

+ their kind, their default judges, the runtimes each runs on, and

+ what each requires of its runtime when it requires anything. It

+ prints each provider with its flag and whether its key is present.

+ Report an absent key as absent; it is a provider that will be

+ skipped, not a failure. It also shows whether the subject's own key,

+ `SUBJECT_ANTHROPIC_API_KEY`, is present; a skill scenario needs it,

+ and without it a `--strict` run refuses. `list` shows only the

+ scenarios under `benchmark/scenarios/`. For a scenario given by its

+ path, read its `kind`, `runtimes`, `requires`, and `judges` from the

+ file.

3. Map what the prompt asks to the flags that exist. The judges are a

bit flag: `3` is Anthropic and OpenAI, `7` adds Gemini, `15` adds

xAI; names joined by commas work too. Effort is `low`, `medium`, or

`high`. `--repeat N` runs the subject N times, 3 by default.

- `--subject-model` pins the subject's model. `--runtime` is

- `host`, `container`, or `vm`. When the prompt names something with

+ `--subject-model` pins the subject's model. `--max-spend-usd N` is

+ the run's spend cap in US dollars, over the subject and the judges

+ together. It is not a hard ceiling: the harness checks it before each

+ repeat and each phase and starts nothing more once it is reached, but

+ what is running finishes, so a run can end above N. Each session of a

+ skill subject also has its own cap, which Claude Code holds: the

+ subject's `max_usd`, or for a scenario in phases each phase's. Say

+ both when the prompt asks for a cap. When it asks what a run could

+ spend, say that it can pass N by about the largest session cap and

+ one repeat's judges. A dry run does not price one-shot judges. An

+ agentic judge's budget `max_usd` is not a hard cap: a judge can pass

+ it by what its last call adds, its output, at most

+ `max_output_tokens` at the model's output price, and the input the

+ turn before it read. So one repeat's agentic judges spend about

+ `max_usd` times the number of judges, and each judge can end above

+ its `max_usd` by that one call. When the prompt names something with

no flag behind it, say so and run without it.

- 4. Run the scenario, for example

- `uv run benchmark/run.py --scenario explain-tenancy --providers 7 --effort medium --repeat 1`.

+ 4. Choose the runtime. `--runtime` is `host`, `container`, or `vm`,

+ and one the scenario lists: its first when the flag is not given.

+ The harness refuses a runtime the scenario does not list with exit

+ 7 and runs nothing. So when the prompt names a runtime the scenario

+ does not list, run nothing, not even a dry run, neither on that

+ runtime nor on another, and say so. The runtimes a scenario lists are

+ the ones `list` shows for it, or, for a scenario given by its path,

+ its file's `runtimes`. When the prompt names no runtime, pass no

+ `--runtime`.

+ A `qa` subject runs no command on any runtime: the harness asks the

+ model itself and builds no image for it, so it needs no engine, no

+ machine, and no `--build`. On `vm` it still takes the

+ `--runtime-config` below, which every vm run needs, a dry run

+ included. For a skill or command subject, the runtime the run takes,

+ named or the scenario's first, needs more:

+ - `container` runs the subject with `docker run` on this machine,

+ so it needs a Docker engine here. Pass `--build`. The harness

+ builds the image only when `--build` is passed, and nothing this

+ skill runs can see whether the image is there. A build whose

+ layers are cached takes seconds, and a first build takes minutes;

+ say so before a run that builds. With no engine here the build

+ fails, and the harness exits 4 before the subject runs or any

+ provider is called. Report that, and do not retry on another

+ runtime.

+ - `vm` runs the subject on another machine, which the runtime

+ config reaches. Pass

+ `--runtime-config benchmark/runtime/lima/runtime-config.yaml`,

+ which drives the Lima machine `swe-benchmark`, unless the prompt

+ names another config. This skill does not start the machine. A

+ machine that does not answer fails every repeat with a note, no

+ judge scores a repeat whose subject did not run, and the harness

+ exits 6. Report that.

+ - `host` runs the subject on this machine, so a skill subject needs

+ Claude Code installed here, and a command subject its command.

+ A scenario's requirements need no flag. A scenario that lists a

+ runtime unable to provide what it requires does not load, so every

+ runtime a scenario lists provides what it requires. A scenario that

+ requires `docker` lists `vm` only. A dry run (step 6)

+ builds nothing and asks no runtime anything, so it needs neither the

+ engine nor the machine. Pass the flags a run would take anyway, so

+ the command is the run's.

+ 5. Run the scenario, for example

+ `uv run benchmark/run.py --scenario explain-tenancy --providers 7 --effort medium --repeat 1 --build`.

A run takes minutes and costs money at every provider selected. When

the prompt has not said which judges or how many repeats, use the

- scenario's default judges and the default of 3 repeats, and say

- which they were.

- 5. When the prompt asks what a run would do rather than for a

+ scenario's default judges and the default of 3 repeats, by leaving

+ out `--providers`, `--effort`, and `--repeat`, and say which they

+ were.

+ 6. When the prompt asks what a run would do rather than for a

measurement, add `--dry-run`: it resolves everything, writes

- `run.json`, and calls nothing. A dry run leaves `run.json` and

+ `run.json`, and calls nothing. For agentic judges, it also copies

+ each reference and fetches each repository reference at its tag,

+ which spends nothing. A dry run leaves `run.json` and

nothing else, so read that file and report the resolved plan: the

subject command, the runtime, the judges with the model and the

- fallbacks each would use, the effort, and the repeats. There is no

- score to report, and inventing one is the worst thing this skill

+ fallbacks each would use (`models` lists each provider's model

+ first, then its fallbacks in order), the effort, the repeats, the run's spend

+ cap when one was given, and the subject's `max_usd`. For a scenario

+ in phases, `run.json` lists each phase under `phases`, and the

+ phases' bounds are the ones in effect: the subject's `max_usd` is

+ null, and its `max_turns` and `timeout_s` are defaults nothing uses.

+ `subject_argv` is only the first phase's command. Report each phase

+ with the fields it holds: its `session` (`fresh`, or `resume` of the

+ phase before), where it starts (`cwd`: the workspace or the output

+ folder), whether it keeps the handoff note (`hint`), its bounds

+ (`max_turns`, `max_usd`, `max_gate_reruns`, and `timeout_s`, the

+ backstop), what follows a bound (`on_cap`), and what its `argv` runs:

+ the prompt as the `argv` carries it, with the target's path in it when

+ the phase's prompt names `{target}` and the handoff sentence when it

+ is hinted, and whether it gets `--add-dir` or `--resume`.

+ Report the subject's `output` folder and its `gates`, which the

+ harness runs on the final tree. `benchmark/README.md`, "A subject in

+ phases", says what each field means and what follows when a phase

+ hits a bound or fails; answer from it when the prompt asks. When

+ `scenario.judges.mode` is `agentic`, report every key of the judges'

+ `budget` as `run.json` resolves it, defaults included, and

+ each reference with its weight, and what the run resolved for it

+ under `versions.references`: the paths of the checkout, or the

+ repository's URL, tag, commit, and the guideline release it pins

+ (`pins`). Report every line under `notes`. `benchmark/README.md`,

+ "Agentic judges", says what each field means. There is

+ no score to report, and inventing one is the worst thing this skill

could do.

- 6. `--runtime container` runs the subject with `docker run`, from an

- image it builds only when `--build` is passed. The first run on a

- machine needs `--build`; a later run adds it when the image's inputs

- changed. A build takes minutes, so say that before starting one.

- 7. List the run folder the command printed with `ls`, then read

+ 7. After a measurement, list the run folder the command printed with `ls`, then read

`report.md` in it. Read `results.json` when a number in the report

- needs its source.

+ needs its source. A run that exits 4 leaves no `report.md`; the

+ Output section says what to report then.

8. Never edit a scenario, a rubric, or the model matrix to get a

better score. A score that needs the rubric changed is the finding.

## Output

After a dry run: the resolved plan from `run.json`, in prose, and the

- sentence that nothing was executed and no provider was called.

+ sentence that nothing was executed and no provider was called; for

+ agentic judges, add that each repository reference was fetched at its

+ tag.

+ When the prompt names a runtime the scenario does not list: that

+ runtime, the runtimes the scenario lists, and the sentence that nothing

+ was executed and no provider was called. Nothing ran, so there is no

+ plan to report.

+ After exit 4, the image build that failed: the reason

+ `streams/build.jsonl` in the run folder gives, and the sentence that no

+ subject ran and no provider was called. There is no score to report.

+ After exit 6, a subject that failed in some repeats: the measurement

+ below, with the repeats that failed and the reason the stream

+ `streams/cli.jsonl` gives for each. `report.md` scores a failed repeat 0

+ in every mean; say those zeros are failures no judge scored. When every

+ repeat failed, as on a vm machine that does not answer, report the

+ reason and no score.

After a measurement, short, in prose:

- the scenario, the runtime, the repeats, and the judges that answered,

each with its model id;

- - the score per provider and the overall mean;

- - the findings that matter, most severe first, in one line each;

- - any provider that did not answer, with the reason it gave;

- - the run folder path, and the fact that it is not checked in.

+ - the score per provider and the overall mean; for agentic judges, say

+ that each score is the weighted score the harness computed, and give

+ each reference's weight and mean;

+ - for a scenario in phases, how each phase ended and the cap that

+ stopped any, and which gates passed on the final tree;

+ - the findings that matter, most severe first, in one line each; for

+ agentic judges, the gaps, per reference;

+ - any provider that did not answer, with the reason it gave, and for

+ an agentic judge that ended `missed`, the budget it names;

+ - the run folder path, and that it is checked in only once

+ `run.py redact` has scanned it for keys.

Say what was measured, not what it means for the roadmap.