git:20260730.e47a8e1 to git:20260822.97934bf

100 added, 82 removed. Audit A to A.

---

name: implement-and-check

- description: The HARD GATE that must pass before any optimization budget is spent. Use right after intake. Walks the agent through implementing the 3 required adapter methods plus any defaulted hooks that need overriding (and any selected skill's abstract methods), then runs `cap-evolve check` on the project plus each involved skill's check.py, refusing to proceed until everything is implemented and deterministic — and listing exactly what is still stubbed.

+ description: Runs the hard gate that has to pass before any optimization budget is spent. Use right after intake. Walks the agent through implementing the 3 required adapter methods plus any defaulted hooks that need overriding (and any selected skill's abstract methods), then runs `cap-evolve check` on the project plus each involved skill's check.py, listing exactly what is still stubbed or non-deterministic and what to do about each kind of failure.

component: phase

argument-hint: "--project .capevolve/project --skill-check PATH"

allowed-tools: Read, Write, Edit, Bash

provides: [checked]

needs: [project]

sources: [skillopt]

---

# implement-and-check — make the contract real

- Optimizing against a half-wired adapter produces a number that means nothing: if

- the scorer is a stub, every candidate scores the same; if `tasks()` is empty, the

- mean is computed over nothing; if `run_target` is non-deterministic in a way the

- scorer can't see, the gate chases ghosts. This phase is the **hard gate** that

- ensures the contract holds *before* a single unit of budget is spent. It is

- cheaper to fail here than after a full optimization run.

- ## Inputs / outputs (manifest tokens)

- - **needs:** `project` — the scaffolded `.capevolve/project/` from intake.

- - **provides:** `checked` — the proof that the adapter (and any involved skill)

- is fully implemented and deterministic. `baseline` will not run without it.

+ Optimizing against a half-wired adapter produces a number that means nothing: a stub

+ scorer gives every candidate the same reward, an empty `tasks()` averages over nothing,

+ a non-deterministic scorer makes the gate chase measurement noise. This phase proves the

+ measurement apparatus works *before* budget is spent. It is cheaper to fail here than

+ after a full run.

## Steps

1. **Implement the 3 required adapter methods** in

- `.capevolve/project/adapters/adapter.py` (see `docs/ADAPTER_CONTRACT.md`). These

- three are `@abstractmethod` — the gate refuses to run until all three are real:

- - `tasks(split)` — yield the evaluation tasks (non-empty, stable across calls).

- - `run_target(task, ctx, *, seed=0)` — run the agent under test with the candidate

- live as `ctx`; capture output + trace into a `Rollout`. Forward `seed` if the

- runner is stochastic.

- - `score(task, rollout)` — return a reward in `[0,1]` + general feedback (no

- gold-answer leakage — it becomes the diagnosis signal).

+ `.capevolve/project/adapters/adapter.py`. These are the `@abstractmethod`s

+ (`core/cap_evolve/adapter.py:77-106`) — the gate refuses to run until all three are real:

+ - `tasks(split)` → `list[Task]` for `'train'|'val'|'test'|'all'`; non-empty, same list

+ every call.

+ - `run_target(task, ctx, *, seed=0)` → `Rollout`. Run the agent under test with the

+ candidate live as `ctx`; capture output + trace + tool calls + cost. Do not score

+ here. Forward `seed` if the runner is stochastic; set `Rollout.error` on an infra

+ failure so the engine treats it as noise, not as a low score.

+ - `score(task, rollout)` → `Score`: reward in `[0,1]` + general feedback (it becomes

+ the diagnosis signal, so never leak the gold answer).

- Then override a **defaulted hook** only if its default does not fit your capability:

- `materialize(candidate_dir, edits=None)` (default: pure write of each

- `{component: text}` edit as a file under `candidate_dir` — override to support a

- capability-specific patch format),

- `live(candidate_dir)` (context manager yielding `ctx`), `apply(candidate_dir, edits=None)`

- (back-compat inject), `trajectories(split, ctx=None)` / `runner_model()` (both default

- to `None`). Separately, `run_batch` / `run_trials` / `score_batch` are not on the base

- class — the harness feature-detects them with `hasattr` and uses them if you define them.

- 2. **Implement any selected skill's `scripts/abstract.py`** (most are concrete and

- need nothing).

+ Override a **defaulted hook** only when its default does not fit:

+ `materialize(candidate_dir, edits=None)` (pure write of `{component: text}`),

+ `live(candidate_dir)` (context manager yielding `ctx`),

+ `apply(candidate_dir, edits=None)` (back-compat inject),

+ `trajectories(split, ctx=None)` and `runner_model()` (both default `None`).

+ Three **optional fast paths** are not on the base class at all — the harness

+ feature-detects them with `hasattr` and uses them only if you define them:

+ `run_batch(tasks, ctx, *, seed)` (drive a benchmark's own batch runner *instead of*

+ `run_target`), `run_trials(tasks, ctx, *, n_trials, base_seed)` (all trials in one

+ concurrent run), `score_batch(tasks, rollouts)` (score a whole trial in one external

+ harness call). `docs/ADAPTER_CONTRACT.md` is the full contract, including the

+ shown-only `metrics` catalog `score()` may return.

+ Note `capability_sources` is **not** an adapter method — it is a `capevolve.yaml` key

+ (the data-model/types files copied into the optimizer's context), owned by intake.

+ 2. **Implement any selected skill's `scripts/abstract.py`** (most are concrete and need

+ nothing).

3. **Run the gate:**

```

python scripts/run.py --project .capevolve/project \

--skill-check <skills>/capabilities/<cap>/scripts/check.py

```

- It runs `cap-evolve check` (adapter: no stubs, `tasks` non-empty + stable, scorer

- deterministic, `materialize()` probed) and each named skill's `check.py`. **Exit 0 =

- green; the JSON lists exactly what is still stubbed or non-deterministic.**

- 4. **Pipeline-wiring self-test (runs automatically once the check is green).** A

- green adapter is necessary but not sufficient — the optimizer also needs its

- *context* wired. After the check passes, `run.py` runs `pipeline_selftest.py`,

- a cheap, benchmark-agnostic check (no API cost) that asserts the plumbing the

- optimizer depends on:

- - the optimizer-prompt template is scaffolded at `optimizer/INSTRUCTIONS.md`

- and still carries its `{{...}}` placeholders (intake must not delete them);

- - `capevolve.yaml::optimizer_instructions_file` points at a file that EXISTS;

- - rendering that template through the REAL harness renderer leaves NO `{{`

- placeholder behind (every dynamic block substitutes);

- - the adapter either DEFINES `trajectories()` (native traj dir → copied verbatim

- into the optimizer's `./trajectories/`) or intentionally inherits the base

- default (cap-evolve falls back to its per-rollout JSON) — both valid, reported.

+ Exit 0 = green. The JSON has three fields with three different meanings — see the

+ table below before you react to it.

- It reports the precise missing/broken artifact so you can iterate until green.

- (Pass `--no-pipeline-selftest` to skip it; run it standalone with

- `python scripts/pipeline_selftest.py --project .capevolve/project`.)

+ 4. **Pipeline-wiring self-test (automatic once the check is green).** A green adapter is

+ necessary but not sufficient — the optimizer also needs its *context* wired. `run.py`

+ then runs `pipeline_selftest.py` (zero API cost): the optimizer-prompt template named

+ by `capevolve.yaml::optimizer_instructions_file` exists, still carries its `{{...}}`

+ placeholders, and renders through the real harness renderer with none left over; and

+ whether the adapter defines `trajectories()` or inherits the base default (both valid,

+ both reported). The template checks are **skipped with a note** for an algorithm that

+ never reads the template — only `hill-climb` is passed `--instructions-file`

+ (`cli.py:869-876`). `--no-pipeline-selftest` skips it; it also runs standalone.

- A full one-iteration mock run is intentionally NOT done here: it would need a

- baseline + frozen split + run dir that do not exist yet at gate time, and those

- are benchmark-specific. This self-test exercises the same workdir-building and

- prompt-rendering code paths instead — the wiring an optimizer actually consumes.

+ A full one-iteration mock run is deliberately not attempted: it would need a baseline,

+ a frozen split and a run dir that do not exist yet at gate time, and building them is

+ benchmark-specific. This exercises the same workdir-building and prompt-rendering paths.

- ## What the check actually verifies (and why)

- - **No stubs** — a `NotImplementedError`/`pass` body means the method silently

- returns nothing; the resulting score is meaningless.

- - **`tasks` non-empty and stable** — an empty or shuffling task set makes the

- mean and the split irreproducible.

- - **Scorer determinism** — score the same rollout twice; differing rewards mean

- the "reward" includes scorer noise the optimizer cannot learn from. (Target

- *stochasticity* is fine and is handled by multi-trial evaluation; *scorer*

- nondeterminism is a bug.)

- - **`materialize()` probed** — an edit that cannot be materialized cannot be

- evaluated, so the check calls it against a temp copy (pure, so the host is untouched).

- This one is a probe, not an assertion: a raise is reported as a **note** and does not

- fail the check (`core/cap_evolve/check.py:166-167`), because a real adapter may need its

- full environment. Green here means "callable or explained", not "edit path verified".

+ ## When it is red — what to do, per failure kind

- ## Do not proceed until green

- If it reports stubs or non-determinism, fix them and re-run. This is the standard

- validation-gate discipline for self-improving systems: prove the measurement

- apparatus works before you trust any measurement it produces. A green check is

- the only honest entry into `baseline`.

+ `CheckReport` has three fields (`core/cap_evolve/check.py:30-38`) and only `problems`

+ affects `ok`. Treating a note as a failure is how an agent gets stuck in a loop.

- ## What good vs bad looks like

- - **Good:** `{"ok": true}` with all 3 required methods concrete, a deterministic scorer,

- and every involved skill's check green.

- - **Bad:** proceeding on a red check "to save time"; a scorer that returns

- different rewards for the same rollout; an empty/placeholder `tasks()`; feedback

- that leaks the gold answer (passes the wiring check but corrupts diagnosis).

+ | report field / message | what it means | do this |

+ |---|---|---|

+ | `stubs: ["<name>"]` | that method still raises the `IMPLEMENT ME` marker | write the method in `adapters/adapter.py`; nothing later was even probed (`check.py:102-107`) |

+ | `"could not load adapter: ..."` | import/instantiation failed — often an unimplemented `@abstractmethod` (`TypeError`) or a bad sibling import | fix the import or define all three abstract methods; the adapter's own dir is on `sys.path`, so sibling helpers import plainly |

+ | `"tasks('val') raised: ..."` | the data path is wrong | point `tasks()` at real data; check the `split` argument is being honored |

+ | `"tasks('val') returned an empty list"` | the split has no tasks | usually a filter or path that matched nothing — print the list before returning |

+ | `"tasks('val') is not stable across calls"` | ids differ between two calls | remove `set`/`dict` iteration order and any per-call shuffle; sort explicitly |

+ | `"scorer is non-deterministic: X vs Y"` | `score()` returned two rewards for one rollout | remove the RNG, or pin an LLM judge's decoding (temperature 0) and cache nothing that hides the variance |

+ | `"score(...) raised on a probe rollout"` | the scorer cannot survive an unfamiliar output | make `score()` total — an unparseable output is reward 0 with feedback, not an exception |

+ | `notes: [...]` | informational, incl. the `materialize()` probe raise and the consuming-model tier mismatch | read; do **not** treat as failure |

+ | skill `check.py` red | that capability/algorithm skill's own contract is unmet | run its `check.py` directly; its JSON names the assertion |

+ Re-run until green. Green is the entry condition for `baseline`.

+ ## What the gate does and does not guarantee

+ Determinism is genuinely executed, not asserted: `check.py:133-142` scores one fixed

+ rollout twice and reports a problem when the rewards differ. Do not read more into green

+ than that. Measured on this checkout (issue #358):

+ - **`run_target` is never called** on the default path, so a `pass`-body runner goes

+ green and fails later, after the split is frozen.

+ - The scorer probe uses a **synthetic** rollout (`output="__probe_output__"`), so a

+ scorer that short-circuits on unrecognizable output — every LLM-judge scorer — is not

+ really tested. Both `score()` calls happen on one in-process instance, so a memoized

+ scorer is unfalsifiable here.

+ - `materialize()` is a **probe, not an assertion**: a raise is a note and does not fail

+ the check (`check.py:166-167`), because a real adapter may need its full environment.

+ Green means "callable or explained", not "edit path verified".

+ - `cap-evolve run` fails closed: `cli.py:721-726` returns 1 before creating a run dir, so

+ no split and no spend. The **standalone** `/cap-evolve:*` chain does not —

+ `baseline/scripts/run.py` contains no check, and `provides: checked` is a static

+ ordering token, not a runtime precondition.

+ If your scorer calls a judge, say so in `PROJECT.md` along with how its decoding is

+ pinned — the gate cannot see it. The one failure mode nothing here can catch: feedback

+ that leaks the gold answer passes every wiring check and still corrupts diagnosis.

## Dual-mode

- This phase runs two ways from the **same** SKILL.md: standalone as the slash command `/cap-evolve:implement-and-check` (the `argument-hint` shows its run.py args), and orchestrator-callable — `cap-evolve run` / the `orchestrate` skill invokes the same `scripts/run.py` headlessly and threads the run dir between phases.

+ Standalone as `/cap-evolve:implement-and-check`; orchestrator-callable — but uniquely for

+ this phase, `cap-evolve run` does **not** invoke `scripts/run.py`. It calls the core check

+ inline and shells straight to `baseline`, so `--skill-check` and the pipeline self-test run

+ in standalone mode only. Run this phase yourself before either `cap-evolve run` or

+ `/cap-evolve:baseline` if you want them — and, per the previous section, the standalone

+ chain has no gate of its own.

## References

- - `references/concepts.md` — the adapter contract, why each check exists, the

- scorer-determinism-vs-target-stochasticity distinction, and the

- validation-gate-before-budget rationale, with sources.

+ - `references/concepts.md` — why each check exists, the

+ scorer-determinism-vs-target-stochasticity distinction (load this when deciding whether

+ your scorer's variance is a bug or a measurement), and the sources.