benchmarking-the-agent · diff
git:20260921.93cc818 to git:20260928.3afaf2e
103 added, 127 removed. Audit A to A.
---
name: benchmarking-the-agent
- description: Measure the flagship GAIA agent against Claude Code and across models — quality, truthfulness, steps, tokens, time and real cost — using the task battery, the adversarial suite and the LLM judge. The recipes need the local `~/gaia-sweep/bench` harness, which is not in this repository; without it only the traps and how to read a result apply. Use when asked to benchmark the agent, compare models, check whether a change cost quality, work out what a run costs, or reproduce the model comparison table.
+ description: Measure the flagship GAIA agent against Claude Code and across models — quality, truthfulness, steps, tokens, time and real cost — with `gaia eval tasks`. Use when asked to benchmark the agent, compare models or harnesses, check whether a change cost quality, work out what a run costs, or reproduce the harness × model table.
---
# Benchmarking the flagship agent
- The harness runs the same tasks through GAIA and through Claude Code, scores them
- mechanically, then has an LLM judge grade the transcripts. It answers three questions:
- **does the work get done**, **is the answer honest about it**, and **what did it cost**.
+ `gaia eval tasks` runs the same tasks through GAIA and through Claude Code,
+ scores them mechanically, then has an LLM judge grade the transcripts. It
+ answers three questions: **does the work get done**, **is the answer honest
+ about it**, and **what did it cost**.
- Everything here was learned by running it. The traps section is the part that saves a day.
+ Everything here was learned by running it. The traps section is the part that
+ saves a day. The full reference is
+ [`docs/reference/eval.mdx`](../../../docs/reference/eval.mdx).
- ## Where things live
+ ## The suites
- | What | Where |
+ | Suite | What it is |
|---|---|
- | Harness | `~/gaia-sweep/bench` (not in the repo — it is local tooling) |
- | Integration checkout under test | `~/gaia-sweep/bench-fixes` (a worktree of amd/gaia) |
- | Task workdirs while running | `/Users/Shared/gaia-bench-work/runs*` — outside the bench tree on purpose |
- | Per-run copies kept for judging | `~/gaia-sweep/bench/runs.<tag>/` |
- | Mechanical results | `results.<batch>.<tag>.jsonl` |
- | Judge output | `judge2.<tag>.jsonl` |
-
- `env.sh` sets the live Lemonade port and key, `GAIA_HOME`, the memory DB and
- `GAIA_AGENT_MAX_STEPS`. **The batch runners source it themselves; anything you run
- standalone does not** — so `. ./env.sh` first before `judge2.py`, `cc_trace.py`, or
- anything touching `$LEMONADE_BASE_URL`. It also puts the project toolchain on the agent's PATH —
- without it `pytest` is missing, every verification is refused for the wrong reason, and
- the benchmark scores a harness failure as the agent's. `BENCH_ENV=broken` withholds it
- deliberately, as the adaptability condition.
+ | `everyday` | The 14 tasks a developer actually asks for. The comparison table's suite. |
+ | `therock` | Four real merged fixes in TheRock, from the commit before the fix. Two forbid the internet in the prompt. |
+ | `core` / `full` / `adversarial` | What CI gates on: the traps and the committed expectations. |
## The one rule: serial
- Every runner wipes and reuses a shared `BENCH_WORK_ROOT`. **Two batteries at once
- destroy each other's workdirs.** Before starting anything:
+ Every run wants the local Lemonade model slot, and two at once race-evict each
+ other's models — the same reason CLAUDE.md gives for `gaia eval agent`. Before
+ starting anything:
```bash
- # Bracketed so the pattern does not match the shell running the check itself.
- pgrep -fl "[r]un_task.py|[r]un_full_branch.sh|[c]c_trace.py|[j]udge2.py" | wc -l # must be 0
+ pgrep -fl "[g]aia eval tasks" | wc -l # must be 0
```
- If something is in flight, **queue rather than wait at the keyboard**: copy the wait loop
- at the top of `run_slate.sh` (`while pgrep -f …; do sleep 60; done`) into your own script
- and launch it with `nohup`. Budget 4–9 minutes per model for the 14-task battery
- depending on the model's speed, plus about 2 minutes to judge it.
-
- The same applies to `gaia eval agent`, for a different reason (CLAUDE.md: concurrent
- evals race-evict each other's models on the local Lemonade slot). Never `pkill` a run
- mid-task, and never stop Lemonade or the daemon while one is in flight.
+ Budget 4–9 minutes per model for the 14-task suite depending on the model's
+ speed, plus about 2 minutes to judge it. Never `pkill` a run mid-task, and
+ never stop Lemonade or the daemon while one is in flight.
## Recipes
- **A branch against Claude Code, 14 tasks, judged.** The main loop of any harness change:
+ **One model, three runs, real cost.** The main loop of any harness change:
```bash
- cd ~/gaia-sweep/bench && GAIA_SRC=~/gaia-sweep/bench-fixes TAG=mytag ./run_full_branch.sh
+ gaia eval tasks run --suite everyday --model fireworks.glm-5p3-flash \
+ --repeats 3 --meter fireworks --out runs/gaia-glm
```
- Runs `batch1_basics`, `batch3_github`, `batch6_usecases` against the checkout in
- `GAIA_SRC`, copies the workdirs to `runs.mytag/`, and judges. `MODEL=` picks the model
- (default Kimi K2.7 Code).
-
- **Several models, one at a time.** `run_slate.sh` waits for anything in flight, then runs
- each `model:tag` pair through the battery and judges each:
+ **The same model through Claude Code**, so the difference is the harness:
```bash
- GAIA_SRC=~/gaia-sweep/bench-fixes ./run_slate.sh \
- fireworks.glm-5p3-flash:glm53flash fireworks.deepseek-v4p1-flash:v4p1flash
+ gaia eval tasks run --suite everyday --harness claude-code \
+ --model fireworks.glm-5p3-flash --repeats 3 --out runs/cc-glm
```
- It pins `RUN_TIMEOUT=900`. Do not lower that — see the traps.
-
- **The adversarial suite** (12 tasks: rate limits, a missing `gh`, prompt injection in an
- issue, a lying README, a hanging test, an issue that does not exist):
+ **Claude Code on Anthropic's own model**, as the reference row. Run Opus 5, not
+ Sonnet: it is the strongest leg available, so a cost share reads as a fraction
+ of the best agent and it sets an honest quality ceiling.
```bash
- TAG=adv7 LEG=gaia MODEL=fireworks.glm-5p3-flash ./run_adversarial.sh # or LEG=cc
- python3.12 judge2.py /Users/Shared/gaia-bench-work/runs-adv adv7 \
- --agent-dir fireworks_glm-5p3-flash --baseline /Users/Shared/gaia-bench-work/baseline --batch 6
- GAIA_DIR=fireworks_glm-5p3-flash python3.12 adversarial_report.py adv7 <cc-tag>
+ gaia eval tasks run --suite everyday --harness claude-code --model claude-opus-5 \
+ --out runs/cc-opus
```
- These must be serial even among themselves: the `gh` stand-in keeps one state file.
-
- **Claude Code at any model**, as a reference leg:
+ **The TheRock tier**, a real 144K-line codebase:
```bash
- export BENCH_WORK_ROOT=/Users/Shared/gaia-bench-work/runs-ccopus
- RESULTS_TAG=cc-opus python3.12 cc_trace.py claude-opus-5 # or claude-sonnet-5
- cp -R "$BENCH_WORK_ROOT" runs.cc-opus
- python3.12 judge2.py runs.cc-opus cc-opus --agent-dir cc-opus --baseline /Users/Shared/gaia-bench-work/baseline
+ gaia eval tasks run --suite therock --model fireworks.glm-5p3-flash --out runs/gaia-therock
```
- Claude Code reports its own spend, so its cost column is real, not rate-carded.
+ **The table.** The first directory is the 100% row:
- **Report.** `slate_report.py` ranks any set of tags by judged quality, with cost as a
- percentage of the Claude Code reference:
+ ```bash
+ gaia eval tasks report runs/cc-opus runs/gaia-glm runs/cc-glm --out runs/report
+ ```
+ **The judge's own controls**, before you trust any quality number:
+
```bash
- python3.12 slate_report.py "glm53flash-<rev>=GLM-5.3 Flash" "kimi7-<rev>=Kimi K2.7"
+ gaia eval tasks controls
```
## Reading a result honestly
- - **One run is noise.** A single model's quality spans about 0.3 across repeat runs
- (Kimi measured 4.50–4.83 on an unchanged branch). Never report a delta below that from
- n=1. `run_repeats.sh` queues repeat runs behind whatever is in flight.
- - **Per-task deltas locate a regression**; the aggregate only tells you one exists. Diff
- the judge's per-task scores between two tags and read the `one_line` for the worst.
- - **Then check the transcript, not your theory.** `judge2._gaia_calls(transcript)` and
- `_cc_calls` yield `(name, args, result)` per call. Count what the agent actually did
- before blaming a change: twice I had a confident hypothesis (trimmed tool descriptions;
- tool ordering) and both died on inspection — the descriptions kept their key
- instructions, and the tool order was byte-identical.
- - **Watch for confounds.** After compound shell commands landed, `gh` invocations halved
- while the same work got done in chained calls. A raw call-count drop is not a
- behaviour regression.
+ - **One run is noise.** A single model's quality spans about 0.3 across repeat
+ runs. Never report a delta below that from n=1 — use `--repeats 3`, and read
+ the min–max the table prints under each mean. A gap smaller than the range it
+ sits in is noise.
+ - **Cost is three different kinds of dollar.** `metered` is real money;
+ `api_equivalent` is Claude Code's list-price figure on a subscription, which
+ is a price of compute and not money spent; `harness_counts` is a rate-carded
+ estimate. The reports label them. Never sum or rank them together without
+ saying which is which.
+ - **Per-task deltas locate a regression**; the aggregate only tells you one
+ exists. Diff the per-task rows between two runs and read the judge's
+ `one_line` for the worst.
+ - **Then check the transcript, not your theory.**
+ `gaia.eval.bench.transcripts.tool_calls(transcript)` yields `(name, args,
+ result)` for either harness. Count what the agent actually did before blaming
+ a change: twice a confident hypothesis (trimmed tool descriptions; tool
+ ordering) died on inspection.
+ - **Watch for confounds.** After compound shell commands landed, `gh`
+ invocations halved while the same work got done in chained calls. A raw
+ call-count drop is not a behaviour regression.
## Traps that cost real time
- **`RUN_TIMEOUT` is a wall-clock cap, so it silently penalises slow models.** At the old
- 420 s default, GLM-5.3 full averaged 182 s per task and got cut off three times — it
- scored 40% and looked incapable. Use 900 s for every model in a comparison, high enough
- that none of them binds. (GLM-5.3 full still took 864 s at the higher cap; it really is
- slow. The point is you cannot tell which until the cap stops binding.)
+ **A wall-clock cap silently penalises slow models.** At a 420 s cap, GLM-5.3
+ full averaged 182 s per task, got cut off three times, scored 40% and looked
+ incapable. The default is 1800 s; keep `--run-timeout` the same for every model
+ in a comparison and high enough that none of them binds.
- **Rate cards drift, and a wrong one silently rewrites every cost claim.** My table had
- DeepSeek V4.1 Flash at `$0.22/$0.007/$0.66` when the published card was
- `$0.30/$0.006/$1.20` — every Flash cost I had reported was ~30% low. Verify against
- `docs.fireworks.ai/serverless/pricing` before quoting cost, and price a dated snapshot
- that has fallen off the list (`deepseek-v4-flash-0731`) at its successor's card so the
- comparison never flatters it. Cards live in `ab_report.py::RATES`; a model with no
- published card must print `n/a`, never a guess.
+ **Rate cards drift, and a wrong one rewrites every cost claim.** A table once
+ had DeepSeek V4.1 Flash at `$0.22/$0.007/$0.66` when the published card was
+ `$0.30/$0.006/$1.20` — every Flash cost was ~30% low. Verify against
+ docs.fireworks.ai/serverless/pricing before quoting cost. Cards live in
+ `src/gaia/eval/bench/metering.py::RATES`; a model with no published card prints
+ `n/a`, never a guess. `--meter fireworks` bypasses the question entirely.
- **Graders must not write into the agent's workdir.** A probe that writes `data.txt` makes
- the judge blame the agent for a stray file. `run_task.graded_copy()` runs tests and
- probes in a throwaway copy — keep new probes inside it.
+ **A metered run is account-wide.** Two metered runs on one account at the same
+ time pollute each other's deltas, and the meter trails by ~90 s.
- **Keyword rubrics are both lenient and brittle.** They pass an answer containing the
- right word without it establishing the point, and fail a correct answer that phrases it
- differently. Judged criteria (`must_establish`, graded via `judge2.py`'s
- `answers_correctly`) replaced them; switching moved the adversarial score by several
- points in both directions. Do not add new keyword checks.
+ **Keyword rubrics are both lenient and brittle.** They pass an answer
+ containing the right word without it establishing the point, and fail a correct
+ answer phrased differently. `must_establish` judged by the LLM replaced them.
+ Do not add new keyword checks.
**The judge often costs more than the run it grades.** A 14-task judge pass is
- $0.4–0.9; GLM-5.3 Flash's entire run is $0.083. Batch it (`--batch 6`) and do not re-judge
- a tag you have already judged.
+ $0.4–0.9; GLM-5.3 Flash's entire run is $0.083. Do not re-judge a run you have
+ already judged — `judge` is a separate command for that reason.
- **Fresh input dominates GAIA's cost**, and the first call of every task is uncached
- because each task is a new process. Within a process caching reaches ~95%; across
- processes Fireworks gives a new one nothing, while Claude Code's account-wide cache hits
- 91% across its runs. That asymmetry flatters Claude Code's token efficiency and is worth
- stating when reporting.
+ **Fresh input dominates GAIA's cost**, and the first call of every task is
+ uncached because each task is a new process. Within a process caching reaches
+ ~95%; across processes Fireworks gives a new one nothing, while Claude Code's
+ account-wide cache hits 91% across its runs. That asymmetry flatters Claude
+ Code's token efficiency and is worth stating when reporting.
- **Diff the actual requests when caching looks wrong.** `GAIA_BENCH_DUMP=<file>` records
- every chat request and response; compare consecutive system prompts and tool arrays
- byte-for-byte. That is how the random per-process scratch-directory name — one line in an
- otherwise identical system prompt — turned out to be defeating the prefix cache.
- `check_wire.py <dump>` reads the same log for a different question: whether the model was
- shown its own tool calls and got each result back correctly paired, exiting non-zero on a
- request an OpenAI-style server would reject.
+ **Graders must not write into the agent's workdir.** A probe that writes
+ `data.txt` makes the judge blame the agent for a stray file. Grading runs after
+ the agent exits, and the diff is taken before the probes run — keep it that way
+ when adding a probe.
## Adding a model
- 1. Confirm it is live: `curl -s "$LEMONADE_BASE_URL/models" -H "Authorization: Bearer $LEMONADE_API_KEY"`.
- 2. Add its published card to `ab_report.py::RATES`, keyed by the bare model name.
- 3. Run it through `run_slate.sh` alongside at least one model you already have numbers
- for, so a harness change cannot be mistaken for a model difference.
+ 1. Confirm the gateway can reach it: `gaia eval tasks run --suite everyday
+ --tasks 21-qa --model <id> --no-judge`.
+ 2. Add its published card to `metering.py::RATES`, keyed by the bare model
+ name, or meter it with `--meter fireworks`.
+ 3. Run it alongside at least one model you already have numbers for, so a
+ harness change cannot be mistaken for a model difference.
## What good looks like
- Numbers below are a snapshot from September 2026 — treat them as the shape to expect, and
- re-measure rather than quoting them.
-
- **Run Claude Code at Opus 5 as the reference, not Sonnet 5.** Opus is the strongest leg
- available, so a cost share reads as a fraction of the best agent rather than of a
- mid-tier one, and it sets an honest quality ceiling. On the 14-task battery Opus scored
- 14/14 at quality 4.97, truthfulness 5.00, 83 steps, 456 s, $3.89; Sonnet 5 scored 4.91 for
- $1.26, which is 32% of Opus.
+ Numbers below are a snapshot from September 2026 — treat them as the shape to
+ expect, and re-measure rather than quoting them.
- Against that ceiling, GLM-5.3 Flash reached 14/14 at quality 4.96 for $0.083 — within
- 0.01 of Opus at 2.1% of its cost — and DeepSeek V4.1 Flash 4.88 for $0.105. A model that
- costs more than the Sonnet reference and scores below it (Qwen3.8 Max, 4.57, $1.325) is
- not a candidate, however good its public benchmarks look.
+ On the 14-task suite, Claude Code at Opus 5 scored 14/14 at quality 4.97,
+ truthfulness 5.00, 83 steps, 456 s. Against that ceiling GLM-5.3 Flash on GAIA
+ reached 14/14 at quality 4.96 for $0.09 of real Fireworks charges, and DeepSeek
+ V4.1 Flash 4.88 for $0.105. A model that costs more than the Sonnet reference
+ and scores below it is not a candidate, however good its public benchmarks
+ look.
- Truthfulness is where the daylight is: Opus and DeepSeek V4 Pro both score a clean 5.00,
- most others sit at 4.86. That axis — never claiming what the tool record does not
+ Truthfulness is where the daylight is: the best legs score a clean 5.00, most
+ others sit at 4.86. That axis — never claiming what the tool record does not
support — is the one worth optimising next.