arch-benchmark git:20260928.7fa6f2aA

Immutable. This exact content never changes and is served at /api/v1/blob/60f447eae9cb39cf.

---
name: arch-benchmark
description: "Run a benchmark scenario (a skill, a command, or a question) in a declared runtime and have frontier models score it against the rubric. Use to measure a change to a skill or the text."
allowed-tools: Read, Grep, Glob, Bash(uv run:*), Bash(ls:*)
disable-model-invocation: true
---

# arch-benchmark

Measure a subject against a rubric and report what the judges said.
The harness is `benchmark/run.py` in a checkout of the guideline
repository, and it is the only thing that runs a benchmark. Do not
compose a provider call by hand, and do not invent a flag: every flag
is in `benchmark/README.md`, and `uv run benchmark/run.py --help`
prints them.

## Where it runs

The current working directory must be a checkout of the guideline
repository: `architecture.md`, `.claude-plugin/plugin.json`,
`benchmark/run.py`, and `benchmark/README.md` all present. Check with
`ls` before anything else. Otherwise stop and say so: the benchmark
measures the checkout it runs from, and writes its run folders under
that checkout's `benchmark/runs/`. The copy under
`${CLAUDE_SKILL_DIR}/../..` is the installed plugin, not a checkout;
never run the harness from there.

## Input

`$ARGUMENTS` is one of:

- a scenario name, with or without flags ("explain-tenancy",
  "review-om with all four judges"), or the path of a scenario file,
  which `--scenario` takes as it is;
- a question about what there is ("which scenarios are there?");
- empty, which means: list the scenarios and the provider
  availability, and stop.

## Procedure

1. Confirm the working directory is a checkout, as "Where it runs"
   says. Every command below runs from it.
2. Run `uv run benchmark/run.py list`. It prints the scenarios with
   their kind, their default judges, the runtimes each runs on, and
   what each requires of its runtime when it requires anything. It
   prints each provider with its flag and whether its key is present.
   Report an absent key as absent; it is a provider that will be
   skipped, not a failure. It also shows whether the subject's own key,
   `SUBJECT_ANTHROPIC_API_KEY`, is present; a skill scenario needs it,
   and without it a `--strict` run refuses. `list` shows only the
   scenarios under `benchmark/scenarios/`. For a scenario given by its
   path, read its `kind`, `runtimes`, `requires`, and `judges` from the
   file.
3. Map what the prompt asks to the flags that exist. The judges are a
   bit flag: `3` is Anthropic and OpenAI, `7` adds Gemini, `15` adds
   xAI; names joined by commas work too. Effort is `low`, `medium`, or
   `high`. `--repeat N` runs the subject N times, 3 by default.
   `--subject-model` pins the subject's model. `--max-spend-usd N` is
   the run's spend cap in US dollars, over the subject and the judges
   together. It is not a hard ceiling: the harness checks it before each
   repeat and each phase and starts nothing more once it is reached, but
   what is running finishes, so a run can end above N. Each session of a
   skill subject also has its own cap, which Claude Code holds: the
   subject's `max_usd`, or for a scenario in phases each phase's. Say
   both when the prompt asks for a cap. When it asks what a run could
   spend, say that it can pass N by about the largest session cap and
   one repeat's judges. A dry run does not price one-shot judges. An
   agentic judge's budget `max_usd` is not a hard cap: a judge can pass
   it by what its last call adds, its output, at most
   `max_output_tokens` at the model's output price, and the input the
   turn before it read. So one repeat's agentic judges spend about
   `max_usd` times the number of judges, and each judge can end above
   its `max_usd` by that one call. When the prompt names something with
   no flag behind it, say so and run without it.
4. Choose the runtime. `--runtime` is `host`, `container`, or `vm`,
   and one the scenario lists: its first when the flag is not given.
   The harness refuses a runtime the scenario does not list with exit
   7 and runs nothing. So when the prompt names a runtime the scenario
   does not list, run nothing, not even a dry run, neither on that
   runtime nor on another, and say so. The runtimes a scenario lists are
   the ones `list` shows for it, or, for a scenario given by its path,
   its file's `runtimes`. When the prompt names no runtime, pass no
   `--runtime`.

   A `qa` subject runs no command on any runtime: the harness asks the
   model itself and builds no image for it, so it needs no engine, no
   machine, and no `--build`. On `vm` it still takes the
   `--runtime-config` below, which every vm run needs, a dry run
   included. For a skill or command subject, the runtime the run takes,
   named or the scenario's first, needs more:
   - `container` runs the subject with `docker run` on this machine,
     so it needs a Docker engine here. Pass `--build`. The harness
     builds the image only when `--build` is passed, and nothing this
     skill runs can see whether the image is there. A build whose
     layers are cached takes seconds, and a first build takes minutes;
     say so before a run that builds. With no engine here the build
     fails, and the harness exits 4 before the subject runs or any
     provider is called. Report that, and do not retry on another
     runtime.
   - `vm` runs the subject on another machine, which the runtime
     config reaches. Pass
     `--runtime-config benchmark/runtime/lima/runtime-config.yaml`,
     which drives the Lima machine `swe-benchmark`, unless the prompt
     names another config. This skill does not start the machine. A
     machine that does not answer fails every repeat with a note, no
     judge scores a repeat whose subject did not run, and the harness
     exits 6. Report that.
   - `host` runs the subject on this machine, so a skill subject needs
     Claude Code installed here, and a command subject its command.

   A scenario's requirements need no flag. A scenario that lists a
   runtime unable to provide what it requires does not load, so every
   runtime a scenario lists provides what it requires. A scenario that
   requires `docker` lists `vm` only. A dry run (step 6)
   builds nothing and asks no runtime anything, so it needs neither the
   engine nor the machine. Pass the flags a run would take anyway, so
   the command is the run's.
5. Run the scenario, for example
   `uv run benchmark/run.py --scenario explain-tenancy --providers 7 --effort medium --repeat 1 --build`.
   A run takes minutes and costs money at every provider selected. When
   the prompt has not said which judges or how many repeats, use the
   scenario's default judges and the default of 3 repeats, by leaving
   out `--providers`, `--effort`, and `--repeat`, and say which they
   were.
6. When the prompt asks what a run would do rather than for a
   measurement, add `--dry-run`: it resolves everything, writes
   `run.json`, and calls nothing. For agentic judges, it also copies
   each reference and fetches each repository reference at its tag,
   which spends nothing. A dry run leaves `run.json` and
   nothing else, so read that file and report the resolved plan: the
   subject command, the runtime, the judges with the model and the
   fallbacks each would use (`models` lists each provider's model
   first, then its fallbacks in order), the effort, the repeats, the run's spend
   cap when one was given, and the subject's `max_usd`. For a scenario
   in phases, `run.json` lists each phase under `phases`, and the
   phases' bounds are the ones in effect: the subject's `max_usd` is
   null, and its `max_turns` and `timeout_s` are defaults nothing uses.
   `subject_argv` is only the first phase's command. Report each phase
   with the fields it holds: its `session` (`fresh`, or `resume` of the
   phase before), where it starts (`cwd`: the workspace or the output
   folder), whether it keeps the handoff note (`hint`), its bounds
   (`max_turns`, `max_usd`, `max_gate_reruns`, and `timeout_s`, the
   backstop), what follows a bound (`on_cap`), and what its `argv` runs:
   the prompt as the `argv` carries it, with the target's path in it when
   the phase's prompt names `{target}` and the handoff sentence when it
   is hinted, and whether it gets `--add-dir` or `--resume`.
   Report the subject's `output` folder and its `gates`, which the
   harness runs on the final tree. `benchmark/README.md`, "A subject in
   phases", says what each field means and what follows when a phase
   hits a bound or fails; answer from it when the prompt asks. When
   `scenario.judges.mode` is `agentic`, report every key of the judges'
   `budget` as `run.json` resolves it, defaults included, and
   each reference with its weight, and what the run resolved for it
   under `versions.references`: the paths of the checkout, or the
   repository's URL, tag, commit, and the guideline release it pins
   (`pins`). Report every line under `notes`. `benchmark/README.md`,
   "Agentic judges", says what each field means. There is
   no score to report, and inventing one is the worst thing this skill
   could do.
7. After a measurement, list the run folder the command printed with `ls`, then read
   `report.md` in it. Read `results.json` when a number in the report
   needs its source. A run that exits 4 leaves no `report.md`; the
   Output section says what to report then.
8. Never edit a scenario, a rubric, or the model matrix to get a
   better score. A score that needs the rubric changed is the finding.

## Output

After a dry run: the resolved plan from `run.json`, in prose, and the
sentence that nothing was executed and no provider was called; for
agentic judges, add that each repository reference was fetched at its
tag.

When the prompt names a runtime the scenario does not list: that
runtime, the runtimes the scenario lists, and the sentence that nothing
was executed and no provider was called. Nothing ran, so there is no
plan to report.

After exit 4, the image build that failed: the reason
`streams/build.jsonl` in the run folder gives, and the sentence that no
subject ran and no provider was called. There is no score to report.

After exit 6, a subject that failed in some repeats: the measurement
below, with the repeats that failed and the reason the stream
`streams/cli.jsonl` gives for each. `report.md` scores a failed repeat 0
in every mean; say those zeros are failures no judge scored. When every
repeat failed, as on a vm machine that does not answer, report the
reason and no score.

After a measurement, short, in prose:

- the scenario, the runtime, the repeats, and the judges that answered,
  each with its model id;
- the score per provider and the overall mean; for agentic judges, say
  that each score is the weighted score the harness computed, and give
  each reference's weight and mean;
- for a scenario in phases, how each phase ended and the cap that
  stopped any, and which gates passed on the final tree;
- the findings that matter, most severe first, in one line each; for
  agentic judges, the gaps, per reference;
- any provider that did not answer, with the reason it gave, and for
  an agentic judge that ended `missed`, the budget it names;
- the run folder path, and that it is checked in only once
  `run.py redact` has scanned it for keys.

Say what was measured, not what it means for the roadmap.