git:20260904.fbb98b4 to git:20260907.a729050

50 added, 39 removed. Audit A to A.

---

name: benchmaker

- description: Build and qualify one runnable benchmark for any target with an observable outcome.

+ description: Build, independently qualify and empirically calibrate a runnable benchmark for one pinned target configuration.

disable-model-invocation: true

---

- Require: `target` and `outcome` — the identity and its intended observable

- outcome, which stay opaque to every call — `sources` (the source policy),

- `rigor` (the confidence each load-bearing claim must reach, stated as the

- evidence that must exist for it), `standard` (the construction calls' stamp)

- and `package` (where the benchmark is written). Construction standard that no

- rule or contract owns is [the protocol](../references/benchmaker-protocol.md);

- the manifest's field set is [its own](../references/benchmaker-manifest.md);

- acquisition's lane cut is [the charter's](../references/benchmaker-research.md).

+ Require: `target` (fixed identity), `outcome` (intended observable result),

+ `sources` (licensed source policy and access/recency limits), evidence-defined

+ `rigor`, domain construction `standard`, destination `package`, fixed

+ `target_configuration` and preregistered `calibration_policy` using the

+ [manifest fields](references/manifest.md). Supply git `workspace`, external git

+ `evidence_workspace`, research evidence-store root, native execution mechanism,

+ candidate `access_policy`, qualification/reference-audit allocation and total

+ source/tool/execution bounds. Missing configuration, policy or other required

+ input returns UNVERIFIED with available artifacts and named gaps before dependent

+ work; do not infer an experiment from a target name.

tickets.py frame-open <run> --goal-file <benchmark-goal> --workflow benchmaker

+ The three private invocations retain this pinned package scope, including

+ benchmark-quality and benchmark-evidence. Execute each body's contract; a frame

+ alone is no invocation. Each goal carries original semantic inputs and remaining

+ bounds. Bind every helper's `workspace` to the supplied construction repository

+ and `evidence-workspace` to `evidence_workspace`; records stay outside frozen

+ benchmark bytes. Resolve the domain standard in this callee's scope before making.

- Three making calls, each on the one before it. **Acquire**, `do` with

- `--standard orch-research`: one converged synthesis about `target` and its

- class, frozen with its sources at one identity, carrying every artifact the

- charter names. **Design**, `do --standard <standard>`: one evaluation frozen at one

- package-owned identity — case specifications with their execution tiers and

- anchors, measurable criteria and evidence classifications, scoring and

- aggregation, intended coverage, expected execution cost. **Materialize**,

- `do --standard <standard> --isolation required`: every specification the frozen

- design names, materialized exactly into `package` as runnable cases, runner,

- scoring data and provenance, each at a preserved identity.

+ **Construct.** Supply target/outcome, sources/rigor, standard, configuration/policy,

+ package/workspace, evidence-store root and bounds to benchmark-construct:

- Then three read-only calls, and independence is what the chain is for.

- **Qualify**, `judge --standard <standard>` over the materialized artifact in a

- delivery disjoint from every builder: oracle failability, coverage,

- discrimination, reproducibility, redundancy, provenance and execution cost

- each checked independently, a verdict per required criterion. **Audit**,

- `judge` over the package and that qualification in a context disjoint from

- both: every evidence-backed blocker under the standard's lens. **Measure**,

- `judge`: the manifest recorded and the measurement pass beside it — what the

- candidates scored over the candidate-accessible scope at the declared rungs,

- on [§Measurement pass](../references/benchmaker-protocol.md#measurement-pass)'s

- terms.

+ tickets.py frame-open <run> --parent <frame> --goal-file <construction-goal> --workflow benchmark-construct

- Never: mutate `target`; generate, rank, promote or activate a candidate;

- select, add, remove, rewrite or substitute a case; let a candidate or search

- context read, choose, retire or receive item-level feedback from protected

- evidence; let unsupported semantics become invented target truth; move the

- declared coverage floor with the target's execution cost; buy speed from the

- coverage floor, the oracle or the horizon; or return a self-qualified

- verdict where the builder-disjoint context is unreachable.

+ Retain the committed draft/design and actual research identities, component

+ locators, builder contexts and gaps. A construction partial stops dependent work.

- Return: `tickets.py frame-close <run> <frame> --done <check>` over the

- recorded manifest and its measurement pass.

+ **Qualify.** Supply the fixed draft, coverage/rigor, standard, builder identities,

+ qualification budget, candidate access/protected scope and preregistered

+ reference-audit sample to benchmark-qualify:

+ tickets.py frame-open <run> --parent <frame> --goal-file <qualification-goal> --workflow benchmark-qualify

+ **Calibrate.** Supply draft and revision-bound qualification, coverage/rigor,

+ standard, unchanged configuration/policy, native mechanism, access scope,

+ builder identities, qualification budget, both git workspaces and remaining

+ bounds to benchmark-calibrate:

+ tickets.py frame-open <run> --parent <frame> --goal-file <calibration-goal> --workflow benchmark-calibrate

+ Its validity gate owns whether attempts or an evidenced bounded repair can

+ proceed. Preserve validity, development calibration and separate final

+ score/drift; required UNVERIFIED never becomes eligibility by omission.

+ Never: mutate target, optimize/rank/promote configurations, activate results,

+ substitute controls for agent attempts, disclose protected evidence to candidate

+ contexts, or revise frozen evaluation after final observations. Development

+ revisions belong solely to the declared calibration policy.

+ Return: `tickets.py frame-close <run> <frame> --done <check>` over available

+ artifact/research identities, manifest, validity and independent findings,

+ all calibration rounds/decision/ledger, eligible frozen identity when present,

+ external final-evaluation record or not-performed reason, spend and gaps

+ (`[]` when none). INVALID/UNVERIFIED/OUT_OF_BAND are useful partial results,

+ never a qualified calibrated success with required criteria unmet.