git:20260922.df9823f to git:20260923.61fb7ef
37 added, 23 removed. Audit A to A.
# Benchmaker design
- Benchmaker builds a task-specific benchmark around the system's actual model, instructions, tools, orchestration and memory. It supplies an authoring process and execution contract rather than a universal runner.
+ Benchmaker builds a task-specific benchmark around a system's actual model, instructions, tools, orchestration and memory. It supplies an authoring process, admission criteria and an execution contract rather than a universal runner.
+ ## A benchmark is not a test suite
+
+ A test suite checks that known behavior still works. A benchmark samples hard, valuable work and exists to separate systems that differ in an ability. That changes what matters. Tasks should take a capable practitioner substantial effort. Verifiers must accept every valid outcome and reject every shortcut. The suite as a whole must show headroom and discrimination, measured rather than asserted.
+
+ Public benchmarks show how easily this goes wrong. Terminal-Bench 2.0 kept 89 of 229 submitted tasks and still repaired 28 later. An empty-response agent scored 38% on τ-bench's airline domain. Frontier models cheated on about half of ImpossibleBench's deliberately impossible tasks. Benchmaker's design follows from those failures: most candidate tasks are expected to be rejected, and every admitted task carries evidence.
+
## Three kinds of evidence
| Evidence | Establishes |
| --- | --- |
| Harness checks | Execution, grading, budgets and recovery work on the exercised paths |
- | Benchmark validation | Tasks are feasible and graders accept valid alternatives while rejecting relevant failures |
- | Agent measurement | The declared system's observed performance under recorded conditions |
+ | Benchmark validation | Each task is solvable, fully specified, fairly graded, shortcut-resistant and hard for the claimed ability |
+ | Agent measurement | The declared systems' observed performance under recorded conditions |
- A simulator can exercise machinery; capability claims require actual agent execution. Synthetic authoring fixtures and simulated external effects follow core's trial policy.
+ A simulator can exercise machinery; capability claims require actual agent execution.
## From request to defensible result
- The coordinator frames the claim, studies representative work, constructs a package and freezes it. A fresh pilot first solves public inputs without authoring history or evaluator answers, then receives evaluator material to inspect discrepancies and controls. The coordinator owns target-agent execution. A separate reviewer assesses the frozen package and complete pilot evidence before the single permitted repair pass.
-
```mermaid
flowchart LR
- A[Frame and construct] --> B[Freeze package]
- B --> C[Fresh pilot and measurements]
- C --> D[Separate review]
- D --> E[At most one repair and affected checks]
- E --> F[Deliver stage and evidence gaps]
+ A[Claim and comparison set] --> B[Source more candidates than needed]
+ B --> C[Build each task]
+ C --> D[Admission: reference, trivial attempts, blind auditor, adversary, calibration]
+ D -->|revise within limit| C
+ D -->|reject and log| R[Rejection log]
+ D -->|admit| E[Freeze suite and measure comparison set]
+ E --> F[Quality card]
+ F --> G[Separate review, at most one repair]
+ G --> H[Deliver suite, card and gaps]
classDef make fill:#dbeafe,stroke:#2563eb,color:#172554
classDef check fill:#dcfce7,stroke:#16a34a,color:#14532d
classDef result fill:#fef3c7,stroke:#d97706,color:#78350f
- class A,B make
- class C,D,E check
- class F result
+ class A,B,C make
+ class D,E,G check
+ class F,H,R result
```
- A benchmark can stop at prototype, calibrated pilot or evaluation suite according to its evidence and budget. Smoke, quick and full are execution profiles, not maturity levels. Requested scope and unmet gates remain visible when only a smaller stage is affordable.
+ The loop sits at the task, not the package. Each task passes the admission criteria or is revised within a declared limit or rejected. Only the admitted suite is measured and reviewed as a whole.
- Outcome credit represents useful partial work. Full success and critical failures remain separate. Missing judgments stay unknown; infrastructure failures do not silently become task failures or disappear from coverage. Changed definitions receive a new benchmark identity, while saved outcomes can be re-scored under an explicit scorer revision.
+ Skills, workflows and harnesses default to an uplift claim: the same tasks with and without the component, at matched budget, beside a simple baseline. That is usually the question someone deciding whether to adopt them needs answered.
+ The quality card replaces paperwork maturity with measured properties: validity, headroom, discrimination, reliability, coverage, integrity, cost and yield. Stages are named by the evidence they hold. A draft is incomplete. A development suite has every task admitted and measured. An evaluation suite adds held-out tasks measured once. Smoke, quick and full are execution profiles, not maturity levels.
+
+ ## Benchmarking Benchmaker
+
+ Because the benchmarks it delivers are executable, Benchmaker can be benchmarked. A task asks it to build a benchmark for a subject system. The verifier runs the delivered benchmark against a hidden pool of subject variants with a known order: a stronger variant, the subject, variants with planted defects, a do-nothing variant and a cheating variant. A good benchmark recovers the order, catches the defects and gives the cheater nothing. The verifier recomputes those results by running the delivered benchmark, so a package cannot pass by claiming them. The [trial catalog](https://github.com/DanMcInerney/orchflows/blob/main/example-workflows/benchmaker/trials/README.md) includes this design, including a scenario where Benchmaker builds a benchmark of benchmark builders and is itself one of the builders measured.
+
## Contract owners
| Owner | Responsibility |
| --- | --- |
- | [Workflow](skills/benchmaker/SKILL.md) | Inputs, phases, independence, repair bound and delivery |
- | [Benchmarking guidance](guidance/benchmarking.md) | Task validity, grading quality and interpretation |
- | [Quality profile](references/quality-profile.md) | Stage gates, substantial work and challenge calibration |
- | [Benchmark contract](references/benchmark-contract.md) | Records, identity, execution, recovery and aggregation |
- | [Research lessons](references/research.md) | Primary precedents for design choices |
+ | [Workflow](skills/benchmaker/SKILL.md) | Inputs, phases, independence, bounds and delivery |
+ | [Benchmarking guidance](guidance/benchmarking.md) | Task, specification, verification and integrity quality; interpretation |
+ | [Quality card](references/quality-card.md) | Claim, admission criteria, measured properties and stages |
+ | [Benchmark contract](references/benchmark-contract.md) | Package, records, identity, execution, recovery and aggregation |
+ | [Research lessons](references/research.md) | Precedents and observed failure rates behind the design |
- Generated packages and evidence live outside this library. Use installed tooling and the lightest environment that preserves fidelity and required access controls. A local directory is not a secrecy boundary. Shared runner templates or optional harness integrations need demonstrated cross-domain reuse and outcome parity before adoption.
+ Generated packages and evidence live outside this library. Use installed tooling and the lightest environment that preserves fidelity and required access controls. A local directory is not a secrecy boundary.
- The library remains experimental. [Trial scenarios and validation gaps](https://github.com/DanMcInerney/orchflows/blob/main/example-workflows/benchmaker/trials/README.md) define what to exercise; a passing pilot does not establish broad readiness.
+ The library remains experimental. [Trial scenarios](https://github.com/DanMcInerney/orchflows/blob/main/example-workflows/benchmaker/trials/README.md) define what to exercise; a passing trial does not establish broad readiness.