test-plugin-harnessing git:20260907.f384104A

Immutable. This exact content is served forever at /api/v1/blob/d5d7e98a7f351b3f.

---
name: test-plugin-harnessing
description: Measure and optimise one phase of an oddyssey run - preflight, drive, observation - against the published benchmark row for that model. Use when a phase is too slow or too expensive, when a harnessing change must be proven rather than asserted, or when a run is suspected of composing work the package should ship. Drives opencode headless, measures the phase, names where the time went, and separates what the package controls from provider latency. Never a substitute for launch-llms-benchmark, which grades findings; this grades the harness.
---

# Testing a phase of the plugin's harness

`launch-llms-benchmark` asks *how good is this model's report*. This asks
a different question: **how much work does the package still make the
model compose before it can do anything** — and it answers it for one
named phase at a time. Do not run the benchmark protocol for this; it
grades findings, costs a full run, and its row is not what moves when a
skill stops making the model write a script.

The rules a change here must satisfy are `AGENTS.md`'s **Plugin
harnessing** section. This skill is how you prove one landed.

## What you need before starting

- **The phase**, named by the caller: `preflight`, `drive`,
  `observation`, or `whole`. Ask if it is not named — measuring the
  wrong phase wastes the run.
- **The baseline**, which is the published row in
  `.llms-benchmark/README.md` for the model you will use: its phase
  durations, its turn count and its **median turn**. Read it from
  `origin/main`, never from the working tree.
- **A model whose median turn is small.** The published median is the
  instrument's precision: a model at 3 s per turn measures the harness,
  one at 20 s measures the provider. Prefer the fastest row in the
  table, whatever its findings score — this is not a quality test.

## The procedure

1. **Rebuild the generated tree.** A CLI agent loads
   `marketplace/`, not `.apm/`, so an unregenerated tree runs the
   *previous* version of every prompt, agent and skill — the run then
   measures the old package while looking exactly like it measures the
   new one. `bash scripts/build-marketplace.sh`, then grep the change in
   the generated file the run will read. This is the single most
   expensive mistake available here; make it once and every number since
   the change is void.

2. **Deploy to every scope the host reads, and prove they match.**
   `uvx --from 'apm-cli==0.29.1' apm install --target opencode` for the
   repository, and copy `.apm/skills/*` and `.apm/agents/*` over
   `~/.claude/skills` / `~/.claude/agents` when the host also reads a
   user scope. Back the user scope up first and restore it at the end —
   it is the user's install, not yours. `diff -rq` the two scopes: a run
   that finds them different spends turns comparing them.

3. **Clean what the next run must not read.** Any report a previous run
   stored, and any leftover container, process or scratch directory. A
   run that reads the last run's conclusions is not measuring anything.

4. **Measure**, with `scripts/measure_phase.py`:

   ```bash
   python3 <this skill's directory>/scripts/measure_phase.py \
     --model <openrouter id> --tag <short label> --phase <phase> \
     --prompt-file <mission> --out <study dir>
   ```

   Its whole surface, so `--help` has nothing to add: `--model`,
   `--tag`, `--phase`, one of `--prompt` / `--prompt-file`, `--out`,
   plus `--end-pattern` (a regular expression over the run's own log
   lines, for a mission with no k6 drive to mark the phase - a post-hoc
   observation, a scenario the mission names), `--variant` (default
   `medium`), `--timeout` (default 2700 s) and `--keep-running` to let
   the run continue past the phase. It records
   the run's own id at launch, stops at the phase's marker, and exits
   non-zero rather than return a fast wrong number when the phase never
   closed.

5. **Analyse before concluding**, with `scripts/analyze_run.py`:

   ```bash
   python3 <this skill's directory>/scripts/analyze_run.py --record <study dir>/<tag>.record.json
   ```

   Surface: `--record`, or `--run-id`; `--gap` (default 60 s) sets the
   gap it reports; `--json`. It prints the
   commands, the
   turns, the generation time and the median turn, then the four
   behaviours a harnessing change removes — scripts the run authored,
   stack resets, machine questions already answered upstream, `--help`
   calls on shipped scripts — and every silent gap.

6. **Read the gaps before believing the clock.** A gap with no command
   in it is the model generating. One far above the run's median turn is
   the provider, not the package: the script says so, and when it does,
   the wall clock is not comparable to anything. Compare **generation
   time** and **commands** instead, or measure again later.

7. **Fix one lever, then measure again.** One change per run, rebuilt
   and redeployed per steps 1-2. Two changes in one run cannot be
   attributed, and the run-to-run spread is wide enough to hide a small
   effect either way.

8. **Restore the machine.** The user's skill and agent scopes from the
   backup, the generated trees to `origin/main` (the release workflow
   owns them), `.gitignore` and anything else `apm install` edited, the
   containers down, the stray processes killed.

## Judging what you measured

**Two samples minimum before claiming a wall-clock gain**, and state
both. The spread between two runs of one configuration reached 17 s in
practice; a single sample below the baseline proves nothing.

**A behaviour count is stronger evidence than a duration.** Commands
before the phase closed, scripts authored, resets taken, redundant
questions — these are what the package controls, they do not move with
the provider's mood, and they are what the PR should quote.

**When a run writes its own script, the shipped one is missing a shape
of the work.** Read what it wrote before hardening any instruction: a
wrapper that only chains a shipped command with a wait is the
repository's own helper-file pattern and is not a defect; one that
rebuilds a command the package ships is a gap in that script.

**Report what did not improve.** A lever that cost a run and moved
nothing belongs in the PR body too — it is what stops the next person
from trying it again.