benchmaker git:20260918.4a145aeA

Immutable. This exact content never changes and is served at /api/v1/blob/ef8d5e69e3f2e1a2.

---
name: benchmaker
description: Build staged benchmarks for substantial agent work, with outcome credit, baseline calibration and evidence-based acceptance.
disable-model-invocation: true
---

Apply [library context](../../references/library-context.md). Accept a target path, callable, command, endpoint, native workflow, or capability description, plus optional comparisons, budget and output location. Infer ordinary choices; ask only about unresolved choices that materially change the claim, permissions or budget. A description suffices for a draft; agent measurement requires actual execution.

Apply core execution rules to authoring and validation. Preserve a fresh pilot audit isolated from authoring history. The orchestrator owns every target-agent launch and continuation as well as authoring assignments. Declare measurement conditions and caller resource limits separately. Failed target attempts consume declared execution limits. Pilot findings wait for step 5's separate review before any package repair.

1. **Frame and research.** Inspect the target and representative user work. Read [research lessons](../../references/research.md) and inspect relevant primary precedents for this domain. Apply the [quality profile](../../references/quality-profile.md): record requested stage, track, bounded claim, task population, complete system boundary, observable outcomes, assumptions and resource budget in a benchmark card. Unavailable research or source work stays an explicit provisional-design gap.
2. **Design.** Apply benchmarking's common criteria and Make guidance. Build the profile's work dossiers and coverage matrix before expanding cases. Define anchored outcome credit, separate full success and critical failures, valid alternatives, source-group splits and run conditions before measurement. Justify binary-only scoring when useful partial work cannot be graded defensibly. Inspect one representative input in each required modality. Plan staged authoring and baseline calibration at a scale appropriate to the requested claim; declare the current deliverable, pilot sample, target executions, repetitions, time/spend limits and stopping rules. A smaller affordable stage must retain the larger request and unmet gates.
3. **Construct.** Follow the [benchmark contract](../../references/benchmark-contract.md). Reuse a suitable harness or write a small runner tailored to this benchmark. Produce tasks, public fixtures, evaluator material, controls, actual invocation adapters and reproducible commands in the caller workspace. Keep target execution faithful to its native workflow. Do not invent an API stand-in for a native workflow or make the candidate emit a new protocol unnecessarily. Freeze and identify the candidate before the pilot.
4. **Pilot.** Launch the fresh worker in a context that excludes authoring history and evaluator content, using the host's no-history control (for example Codex `fork_turns="none"`). Initially supply only the chosen public audit sample, conditions, guidance, budget and output directory. Have it save its answers and notify the orchestrator before disclosing package/evaluator paths to the same worker; it then compares answers and exercises controls and applicable [boundary checks](../../references/benchmark-contract.md#boundary-checks). It returns evidence, findings and any further execution requirements without repairing or launching agents. The orchestrator runs the declared target measurements and calibration, using separate top-level sessions for composing targets per core `docs/hosts.md#workflow-trials`. Use the actual target, or a named representative when no target exists and the budget permits. Record disclosure and actual access limits. If the host cannot exclude inherited answers, report the independent audit as incomplete. Offline controls do not establish target measurement. Retain disagreements and classify failures before interpreting difficulty. Join the evidence; freeze the candidate and proposed corrections for review.
5. **Review and repair.** Apply `orchflows:orch-review-revise-once` to the frozen package, card, pilot evidence and proposed corrections, using applicable benchmarking and task-domain guidance and a fresh reviewer other than the pilot auditor. Scope repairs to the benchmark package and include its required checks. Assess task and grader validity separately from agent performance. Preserve the original review and inspected state separately from the delivered revision; version changed definitions. Rerun affected independent trials only within the remaining declared allowance; otherwise report the validation gap. Do not retry failed answers until they pass or call exposed cases held out.