---
name: benchmaker
description: Build staged benchmarks for substantial agent work, with outcome credit, baseline calibration and evidence-based acceptance.
disable-model-invocation: true
---

Apply [library context](../../references/library-context.md). Accept a target path, callable, command, endpoint, native workflow, or capability description, plus optional comparisons, budget and output location. Infer ordinary choices; ask only about unresolved choices that materially change the claim, permissions or budget. A description suffices for a draft; agent measurement requires actual execution.

Use the caller context for research, design and construction. Plan two fresh children: one pilot worker through `orch-work`, then one reviewer through `orch-review`. One affected trial rerun after repair may use one additional `orch-work` child. Benchmarked agent executions are additional workload; declare and count them separately before launch. Interrupted children consume their allocation; resume the same child if possible, otherwise report the gap when no allocation remains. No automatic second review or repair loop. Preserve scoped caller model/effort choices through the primitives.

1. **Frame and research.** Inspect the target and representative user work. Read [research lessons](../../references/research.md) and inspect relevant primary precedents for this domain. Apply the [quality profile](../../references/quality-profile.md): record requested stage, track, bounded claim, task population, complete system boundary, observable outcomes, assumptions and resource budget in a benchmark card. Unavailable research or source work stays an explicit provisional-design gap.
2. **Design.** Apply benchmarking Make guidance. Build the profile's work dossiers and coverage matrix before expanding cases. Define anchored outcome credit, separate full success and critical failures, valid alternatives, source-group splits and run conditions before measurement. Justify binary-only scoring when useful partial work cannot be graded defensibly. Inspect one representative input in each required modality. Plan staged authoring and baseline calibration at a scale appropriate to the requested claim; declare the current deliverable, pilot sample, target executions, repetitions, time/spend limits and stopping rules. A smaller affordable stage must retain the larger request and unmet gates.
3. **Construct.** Follow the [benchmark contract](../../references/benchmark-contract.md). Reuse a suitable harness or write a small runner tailored to this benchmark. Produce tasks, public fixtures, evaluator material, controls, actual invocation adapters and reproducible commands in the caller workspace. Keep target execution faithful to its native workflow. Do not invent an API stand-in for a native workflow or make the candidate emit a new protocol unnecessarily. Freeze and identify the candidate before the pilot.
4. **Pilot.** Launch the fresh worker in a context that excludes authoring history and evaluator content, using the host's no-history control (for example Codex `fork_turns="none"`). Initially supply only the chosen public audit sample, conditions, guidance, budget and output directory. Have it save its answers and notify the coordinator before disclosing package/evaluator paths to the same worker; it then compares answers, exercises controls and applicable [boundary checks](../../references/benchmark-contract.md#boundary-checks), and runs the declared pilot. Record the handoff and actual access limits. If the host cannot exclude inherited answers, report the independent audit as incomplete; asking the child to ignore them does not restore independence. Use the actual target, or a named representative agent when no target exists and the budget permits. Offline controls are still useful when execution is unavailable; report the missing measurement. Execute declared calibration, retain disagreements and classify failures before interpreting difficulty. The worker returns evidence and findings without repairing. Join the evidence; freeze the candidate and any proposed corrections for review.
5. **Review and repair.** Give one fresh reviewer the frozen package, card, pilot evidence, proposed corrections and benchmarking Review guidance. It assesses task and grader validity separately from agent performance, without repairing. Wait for it to finish. Make one bounded repair pass from joined findings, version changed definitions and rerun affected checks or trials within the remaining budget. Use the optional fresh worker only when an independent rerun is needed; do not rerun failed answers until they pass or reuse exposed cases as held-out evidence.
6. **Deliver.** Return the package, commands, measured timing/cost and coverage, and remaining gaps. Report requested versus achieved stage with evidence for each gate; keep harness checks, benchmark validation and agent measurements separate. Publish full success, outcome credit and critical errors separately, with their denominators and per-family results. A missing required execution, judgment, review or unresolved validity defect leaves the affected claim draft or partially validated. State which profiles and failure paths were actually exercised, and the next unfulfilled stage. File creation does not register the native skill.