git:20260923.4056448 to git:20260923.61fb7ef

18 added, 9 removed. Audit A to A.

---

name: benchmaker

- description: Build staged benchmarks for substantial agent work, with outcome credit, baseline calibration and evidence-based acceptance.

+ description: Build benchmarks of hard agent tasks, admitting only verified, shortcut-resistant, calibrated tasks and reporting how well the suite separates systems.

disable-model-invocation: true

---

- Apply [library context](../../references/library-context.md). Accept a target path, callable, command, endpoint, native workflow or capability description; optional comparisons, budget and output location. Infer ordinary choices; clarify only material claim, permission or budget decisions. Descriptions support drafts; measurement requires execution.

+ Apply [library context](../../references/library-context.md). Accept a target: path, callable, command, endpoint, native workflow, skill or capability description. Also accept an optional claim, comparisons, budget and output location. Infer ordinary choices; clarify only material claim, permission or budget decisions. Descriptions support drafts; measurement requires execution.

- The coordinator owns authoring assignments and every target-agent launch or continuation. Preserve a fresh pilot isolated from authoring history. Record measurement conditions separately from caller resource limits; failed attempts consume execution limits. Gather pilot evidence and step 5's review before repairs.

+ A benchmark samples hard, valuable work and separates systems that differ in the claimed ability. Most candidate tasks should fail admission. A smaller budget admits fewer tasks and reaches a lower stage; it never weakens tasks or skips admission checks.

- 1. **Frame and research.** Inspect the target, representative user work, [research lessons](../../references/research.md) and relevant primary precedents. Apply the [quality profile](../../references/quality-profile.md): record requested stage, track, bounded claim, population, complete system boundary, observable outcomes, assumptions and budget in the card. Missing research or source work leaves a provisional-design gap.

- 2. **Design.** Apply common and Make benchmarking guidance. Build work dossiers and coverage before expanding cases. Predefine anchored outcome credit, full success, critical failures, valid alternatives, source-group splits and conditions. Inspect representative inputs in every required modality. Declare the staged deliverable, baseline calibration, pilot sample, target executions, repetitions, time/spend limits and stopping rules.

- 3. **Construct.** Follow the [benchmark contract](../../references/benchmark-contract.md). Reuse a suitable harness or build a task-specific runner. Produce public tasks/fixtures, evaluator material, controls, actual adapters and reproducible commands in the caller workspace. Freeze and identify the package.

- 4. **Pilot.** Launch a fresh worker with no authoring history or evaluator content. Initially supply only public audit inputs, conditions, guidance, budget and output directory. Require saved answers and notification before disclosing evaluator/package paths to that worker. It then compares answers, exercises controls and applicable [boundary checks](../../references/benchmark-contract.md#boundary-checks), and returns findings and execution requests without repairs or delegation. The coordinator runs declared target measurements/calibration; composing targets use separate top-level sessions under core `docs/hosts.md#workflow-trials`. Measure the actual target, or a named representative when none exists and budget permits. Record disclosure and access limits; inherited answers leave independent audit incomplete. Retain disagreements, classify failures, join the evidence and freeze proposed corrections for review.

- 5. **Review and repair.** Apply `shared:review-revise-once` to the frozen package, card, pilot evidence and corrections with benchmarking/task guidance and a fresh reviewer other than the pilot auditor. Scope repairs and checks to the package. Assess task/grader validity separately from performance. Preserve the review's inspected identity; give changed definitions a new identity. Rerun affected independent trials within remaining allowance or report the gap.

- 6. **Deliver.** Return package, commands, measured timing/cost, coverage and gaps. Report requested/achieved stage and gate evidence; separate harness checks, benchmark validation and measurements. Report full success, outcome credit and critical errors with denominators and per-family results. Missing required execution, judgment, review or unresolved validity defects leave affected claims draft/partial. Name exercised profiles/failure paths and the next unmet stage.

+ The coordinator owns every assignment and every target, comparison, calibration and adversary launch. Other agents return results and execution requests without delegating. Composing targets run in separate top-level sessions under core `docs/hosts.md#workflow-trials`. Before launching, declare and show planned candidates, per-task revision limit, repeats, concurrency, deadlines and estimated spend. Caller limits on launches or sessions count every launch they name. When a limit names only target runs, report authoring, auditing, adversary and calibration launches separately. Record measurement conditions separately from caller limits; failed attempts consume execution limits.

+ 1. **Claim.** Inspect the target, representative user work, [research lessons](../../references/research.md) and relevant primary precedents. Record the claim entries of the [quality card](../../references/quality-card.md). Choose the comparison set and the calibration system under benchmarking guidance and the card.

+ 2. **Source.** Apply common and Make benchmarking guidance. Gather more candidate work than the suite needs, with provenance, from real artifacts, practitioners, the target's own failures and constructive methods. Each candidate states its value, why it is hard for the claimed ability and an estimated expert time. Spread candidates across families and independent source groups before expanding any one.

+ 3. **Build and admit.** Authors build each candidate under the [benchmark contract](../../references/benchmark-contract.md): public instruction and interface, environment, reference solution and verifier. Freeze each built task, then gather its admission evidence:

+ - run the reference and trivial attempts through the verifier;

+ - a fresh auditor with no authoring history or evaluator material solves from public inputs. The coordinator keeps a copy of each saved outcome; an auditor covering several tasks saves all of them before any disclosure. It then receives the evaluator material and reports unstated requirements, rejected valid outcomes and accepted wrong ones;

+ - run labeled outcomes through the verifier: an independent valid alternative, which may be the auditor's when valid, and plausible wrong outcomes, including broken copies of the reference;

+ - a fresh adversary with the solver's material and access plus the exploit catalogue in research, and never evaluator material, tries to earn credit without doing the work;

+ - the calibration system attempts the task repeatedly.

+ Auditors and adversaries may cover several tasks they did not author. Revise or reject by the admission criteria; a revision reruns the checks it affects and counts toward the per-task limit.

+ 4. **Measure.** Freeze the admitted suite. Run the comparison set with predeclared repeats, measuring the actual target, or a named representative when none exists and budget permits. Compute the card. Classify failures from transcripts as ability, task defect, grading, infrastructure or unknown.

+ 5. **Review and repair.** Apply `shared:review-revise-once` with benchmarking and task guidance, using a fresh reviewer who neither authored nor audited tasks. Its inputs are the frozen suite, card, rejection log, admission evidence and a transcript sample that includes successes. Scope repairs and checks to the package. Review repairs count toward each task's revision limit. Changed tasks receive new identities and rerun affected admission and measurement within remaining allowance. A repaired task that fails re-admission leaves the suite, which then receives a new identity, or stays draft.

+ 6. **Deliver.** Return the suite, commands, card, rejection log, per-family results with denominators, measured time and cost, requested and achieved stage, and gaps. State whether measured headroom and separation support the claim. Missing required execution, judgment or review, or an unresolved validity defect, leaves affected claims draft. Name the next unmet stage.