benchmaker git:20260918.c9794adA

Immutable. This exact content never changes and is served at /api/v1/blob/203bdaea49753b37.

---
name: benchmaker
description: Build staged benchmarks for substantial agent work, with outcome credit, baseline calibration and evidence-based acceptance.
disable-model-invocation: true
---

Apply [library context](../../references/library-context.md). Accept a target path, callable, command, endpoint, native workflow or capability description; optional comparisons, budget and output location. Infer ordinary choices; clarify only material claim, permission or budget decisions. Descriptions support drafts; measurement requires execution.

The orchestrator owns authoring assignments and every target-agent launch or continuation. Preserve a fresh pilot isolated from authoring history. Record measurement conditions separately from caller resource limits; failed attempts consume execution limits. Gather pilot evidence and step 5's review before repairs.

1. **Frame and research.** Inspect the target, representative user work, [research lessons](../../references/research.md) and relevant primary precedents. Apply the [quality profile](../../references/quality-profile.md): record requested stage, track, bounded claim, population, complete system boundary, observable outcomes, assumptions and budget in the card. Missing research or source work leaves a provisional-design gap.
2. **Design.** Apply common and Make benchmarking guidance. Build work dossiers and coverage before expanding cases. Predefine anchored outcome credit, full success, critical failures, valid alternatives, source-group splits and conditions. Justify binary-only scoring when partial work lacks defensible credit. Inspect representative inputs in every required modality. Declare the staged deliverable, baseline calibration, pilot sample, target executions, repetitions, time/spend limits and stopping rules. Smaller stages retain the larger request's unmet gates.
3. **Construct.** Follow the [benchmark contract](../../references/benchmark-contract.md). Reuse a suitable harness or build a task-specific runner. Produce public tasks/fixtures, evaluator material, controls, actual adapters and reproducible commands in the caller workspace. Preserve native target composition; do not substitute an API or require a new candidate protocol unnecessarily. Freeze and identify the package.
4. **Pilot.** Launch a fresh worker with no authoring history or evaluator content, using native no-history controls, such as Codex `fork_turns="none"`. Initially supply only public audit inputs, conditions, guidance, budget and output directory. Require saved answers and notification before disclosing evaluator/package paths to that worker. It then compares answers, exercises controls and applicable [boundary checks](../../references/benchmark-contract.md#boundary-checks), and returns findings and execution requests without repairs or delegation. The orchestrator runs declared target measurements/calibration; composing targets use separate top-level sessions under core `docs/hosts.md#workflow-trials`. Authoring trials use synthetic inputs and simulated external effects while preserving real agent executions and judgments. Measure the actual target, or a named representative when none exists and budget permits. Record disclosure and access limits; inherited answers leave independent audit incomplete. Offline controls establish no target measurement. Retain disagreements, classify failures, join the evidence and freeze proposed corrections for review.
5. **Review and repair.** Apply `orchflows:orch-review-revise-once` to the frozen package, card, pilot evidence and corrections with benchmarking/task guidance and a fresh reviewer other than the pilot auditor. Scope repairs and checks to the package. Assess task/grader validity separately from performance. Preserve the review's inspected identity; version changed definitions. Rerun affected independent trials within remaining allowance or report the gap. Never retry wrong answers until passing or call exposed cases held out.
6. **Deliver.** Return package, commands, measured timing/cost, coverage and gaps. Report requested/achieved stage and gate evidence; separate harness checks, benchmark validation and measurements. Report full success, outcome credit and critical errors with denominators and per-family results. Missing required execution, judgment, review or unresolved validity defects leave affected claims draft/partial. Name exercised profiles/failure paths and the next unmet stage. File creation does not register a native skill.