git:20260922.6f31a92 to git:20260922.d9b2153

2 added, 2 removed. Audit A to A.

---

name: benchmaker

description: Build staged benchmarks for substantial agent work, with outcome credit, baseline calibration and evidence-based acceptance.

disable-model-invocation: true

---

Apply [library context](../../references/library-context.md). Accept a target path, callable, command, endpoint, native workflow or capability description; optional comparisons, budget and output location. Infer ordinary choices; clarify only material claim, permission or budget decisions. Descriptions support drafts; measurement requires execution.

- The orchestrator owns authoring assignments and every target-agent launch or continuation. Preserve a fresh pilot isolated from authoring history. Record measurement conditions separately from caller resource limits; failed attempts consume execution limits. Gather pilot evidence and step 5's review before repairs.

+ The coordinator owns authoring assignments and every target-agent launch or continuation. Preserve a fresh pilot isolated from authoring history. Record measurement conditions separately from caller resource limits; failed attempts consume execution limits. Gather pilot evidence and step 5's review before repairs.

1. **Frame and research.** Inspect the target, representative user work, [research lessons](../../references/research.md) and relevant primary precedents. Apply the [quality profile](../../references/quality-profile.md): record requested stage, track, bounded claim, population, complete system boundary, observable outcomes, assumptions and budget in the card. Missing research or source work leaves a provisional-design gap.

2. **Design.** Apply common and Make benchmarking guidance. Build work dossiers and coverage before expanding cases. Predefine anchored outcome credit, full success, critical failures, valid alternatives, source-group splits and conditions. Inspect representative inputs in every required modality. Declare the staged deliverable, baseline calibration, pilot sample, target executions, repetitions, time/spend limits and stopping rules.

3. **Construct.** Follow the [benchmark contract](../../references/benchmark-contract.md). Reuse a suitable harness or build a task-specific runner. Produce public tasks/fixtures, evaluator material, controls, actual adapters and reproducible commands in the caller workspace. Preserve native target composition; do not substitute an API or require a new candidate protocol unnecessarily. Freeze and identify the package.

- 4. **Pilot.** Launch a fresh worker with no authoring history or evaluator content. Initially supply only public audit inputs, conditions, guidance, budget and output directory. Require saved answers and notification before disclosing evaluator/package paths to that worker. It then compares answers, exercises controls and applicable [boundary checks](../../references/benchmark-contract.md#boundary-checks), and returns findings and execution requests without repairs or delegation. The orchestrator runs declared target measurements/calibration; composing targets use separate top-level sessions under core `docs/hosts.md#workflow-trials`. Measure the actual target, or a named representative when none exists and budget permits. Record disclosure and access limits; inherited answers leave independent audit incomplete. Retain disagreements, classify failures, join the evidence and freeze proposed corrections for review.

+ 4. **Pilot.** Launch a fresh worker with no authoring history or evaluator content. Initially supply only public audit inputs, conditions, guidance, budget and output directory. Require saved answers and notification before disclosing evaluator/package paths to that worker. It then compares answers, exercises controls and applicable [boundary checks](../../references/benchmark-contract.md#boundary-checks), and returns findings and execution requests without repairs or delegation. The coordinator runs declared target measurements/calibration; composing targets use separate top-level sessions under core `docs/hosts.md#workflow-trials`. Measure the actual target, or a named representative when none exists and budget permits. Record disclosure and access limits; inherited answers leave independent audit incomplete. Retain disagreements, classify failures, join the evidence and freeze proposed corrections for review.

5. **Review and repair.** Apply `shared:review-revise-once` to the frozen package, card, pilot evidence and corrections with benchmarking/task guidance and a fresh reviewer other than the pilot auditor. Scope repairs and checks to the package. Assess task/grader validity separately from performance. Preserve the review's inspected identity; give changed definitions a new identity. Rerun affected independent trials within remaining allowance or report the gap.

6. **Deliver.** Return package, commands, measured timing/cost, coverage and gaps. Report requested/achieved stage and gate evidence; separate harness checks, benchmark validation and measurements. Report full success, outcome credit and critical errors with denominators and per-family results. Missing required execution, judgment, review or unresolved validity defects leave affected claims draft/partial. Name exercised profiles/failure paths and the next unmet stage. File creation does not register a native skill.