benchmaker git:20260916.32892e8A

Immutable. This exact content is served forever at /api/v1/blob/20400596e6a8e1dd.

---
name: benchmaker
description: Build and pilot a small runnable benchmark for an agent, workflow, or described capability, with task-specific grading and explicit measurement limits.
disable-model-invocation: true
---

Apply [library context](../../references/library-context.md). Accept a target path, callable, command, endpoint, native workflow, or capability description, plus optional comparisons, budget and output location. Infer ordinary choices; ask only about unresolved choices that materially change the claim, permissions or budget. A description suffices for a draft; agent measurement requires actual execution.

Use the caller context for research, design and construction. Plan two fresh children: one pilot worker through `orch-work`, then one reviewer through `orch-review`. One affected trial rerun after repair may use one additional `orch-work` child. Benchmarked agent executions are additional workload; declare and count them separately before launch. Interrupted children consume their allocation; resume the same child if possible, otherwise report the gap when no allocation remains. No automatic second review or repair loop. Preserve scoped caller model/effort choices through the primitives.

1. **Frame and research.** Inspect the target and representative user work. Read [research lessons](../../references/research.md) and inspect relevant primary precedents for this domain. Record the decision, bounded claim, task population, complete system boundary, observable outcomes, assumptions and resource budget in a benchmark card. Unavailable research stays an explicit provisional-design gap.
2. **Design.** Apply benchmarking Make guidance. Map families, sources and splits; define criteria, mandatory constraints, valid alternatives, weights and run conditions before comparative measurement. Inspect one representative input in each required modality. Propose a modest authoring budget, usually 12–24 cases, adjusting to the request; this count establishes no statistical sufficiency. Declare the pilot sample, target executions, repetitions, time/spend limits and stopping rules.
3. **Construct.** Follow the [benchmark contract](../../references/benchmark-contract.md). Reuse a suitable harness or write a small runner tailored to this benchmark. Produce tasks, public fixtures, evaluator material, controls, actual invocation adapters and reproducible commands in the caller workspace. Keep target execution faithful to its native workflow. Do not invent an API stand-in for a native workflow or make the candidate emit a new protocol unnecessarily. Freeze and identify the candidate before the pilot.
4. **Pilot.** Launch the fresh worker in a context that excludes authoring history and evaluator content, using the host's no-history control (for example Codex `fork_turns="none"`). Initially supply only the chosen public audit sample, conditions, guidance, budget and output directory. Have it save its answers and notify the coordinator before disclosing package/evaluator paths to the same worker; it then compares answers, exercises controls and runs the declared pilot. Record the handoff and actual access limits. If the host cannot exclude inherited answers, report the independent audit as incomplete; asking the child to ignore them does not restore independence. Use the actual target, or a named representative agent when no target exists and the budget permits. Offline controls are still useful when execution is unavailable; report the missing measurement. The worker returns evidence and findings without repairing. Join the evidence; freeze the candidate and any proposed corrections for review.
5. **Review and repair.** Give one fresh reviewer the frozen package, card, pilot evidence, proposed corrections and benchmarking Review guidance. It assesses task and grader validity separately from agent performance, without repairing. Wait for it to finish. Make one bounded repair pass from joined findings, version changed definitions and rerun affected checks or trials within the remaining budget. Use the optional fresh worker only when an independent rerun is needed; do not rerun failed answers until they pass or reuse exposed cases as held-out evidence.
6. **Deliver.** Return the package, commands, measured timing/cost and coverage, and remaining gaps. Report harness checks, benchmark validation and agent measurements separately. A missing required execution, judgment, review or unresolved validity defect leaves the affected claim draft or partially validated. State which profiles and failure paths were actually exercised. File creation does not register the native skill.