benchmaker git:20260923.61fb7efA

Immutable. This exact content never changes and is served at /api/v1/blob/327e06f8b02b2ca0.

---
name: benchmaker
description: Build benchmarks of hard agent tasks, admitting only verified, shortcut-resistant, calibrated tasks and reporting how well the suite separates systems.
disable-model-invocation: true
---

Apply [library context](../../references/library-context.md). Accept a target: path, callable, command, endpoint, native workflow, skill or capability description. Also accept an optional claim, comparisons, budget and output location. Infer ordinary choices; clarify only material claim, permission or budget decisions. Descriptions support drafts; measurement requires execution.

A benchmark samples hard, valuable work and separates systems that differ in the claimed ability. Most candidate tasks should fail admission. A smaller budget admits fewer tasks and reaches a lower stage; it never weakens tasks or skips admission checks.

The coordinator owns every assignment and every target, comparison, calibration and adversary launch. Other agents return results and execution requests without delegating. Composing targets run in separate top-level sessions under core `docs/hosts.md#workflow-trials`. Before launching, declare and show planned candidates, per-task revision limit, repeats, concurrency, deadlines and estimated spend. Caller limits on launches or sessions count every launch they name. When a limit names only target runs, report authoring, auditing, adversary and calibration launches separately. Record measurement conditions separately from caller limits; failed attempts consume execution limits.

1. **Claim.** Inspect the target, representative user work, [research lessons](../../references/research.md) and relevant primary precedents. Record the claim entries of the [quality card](../../references/quality-card.md). Choose the comparison set and the calibration system under benchmarking guidance and the card.
2. **Source.** Apply common and Make benchmarking guidance. Gather more candidate work than the suite needs, with provenance, from real artifacts, practitioners, the target's own failures and constructive methods. Each candidate states its value, why it is hard for the claimed ability and an estimated expert time. Spread candidates across families and independent source groups before expanding any one.
3. **Build and admit.** Authors build each candidate under the [benchmark contract](../../references/benchmark-contract.md): public instruction and interface, environment, reference solution and verifier. Freeze each built task, then gather its admission evidence:
   - run the reference and trivial attempts through the verifier;
   - a fresh auditor with no authoring history or evaluator material solves from public inputs. The coordinator keeps a copy of each saved outcome; an auditor covering several tasks saves all of them before any disclosure. It then receives the evaluator material and reports unstated requirements, rejected valid outcomes and accepted wrong ones;
   - run labeled outcomes through the verifier: an independent valid alternative, which may be the auditor's when valid, and plausible wrong outcomes, including broken copies of the reference;
   - a fresh adversary with the solver's material and access plus the exploit catalogue in research, and never evaluator material, tries to earn credit without doing the work;
   - the calibration system attempts the task repeatedly.

   Auditors and adversaries may cover several tasks they did not author. Revise or reject by the admission criteria; a revision reruns the checks it affects and counts toward the per-task limit.
4. **Measure.** Freeze the admitted suite. Run the comparison set with predeclared repeats, measuring the actual target, or a named representative when none exists and budget permits. Compute the card. Classify failures from transcripts as ability, task defect, grading, infrastructure or unknown.
5. **Review and repair.** Apply `shared:review-revise-once` with benchmarking and task guidance, using a fresh reviewer who neither authored nor audited tasks. Its inputs are the frozen suite, card, rejection log, admission evidence and a transcript sample that includes successes. Scope repairs and checks to the package. Review repairs count toward each task's revision limit. Changed tasks receive new identities and rerun affected admission and measurement within remaining allowance. A repaired task that fails re-admission leaves the suite, which then receives a new identity, or stays draft.
6. **Deliver.** Return the suite, commands, card, rejection log, per-family results with denominators, measured time and cost, requested and achieved stage, and gaps. State whether measured headroom and separation support the claim. Missing required execution, judgment or review, or an unresolved validity defect, leaves affected claims draft. Name the next unmet stage.