git:20261014.4640d94 to git:20261014.367ba9a

# Agentic Coding Lab

Research framework for studying how TDD workflow structures and prompt styles affect AI-generated code quality. Runs Claude Code CLI in Docker containers against coding katas, then analyzes results via RQ-driven aggregation.

**Full documentation:** `README.md` — methodology, RQ frontmatter schema, workflow descriptions, model configurations, script reference, metrics glossary.

## Skills

### `/run-rq RQ-N`

End-to-end RQ orchestration: validate README → generate fill plan → start Docker batch → monitor progress → aggregate → propose findings updates. Pure orchestration calling existing repo scripts.

- Triggers: "run-rq", "fill RQ-N", "RQ-N voranbringen", "Forschungsfrage N starten"

- Details: `.claude/skills/run-rq/SKILL.md`

### `/reanalyze RQ-N`

Re-run analysis pipeline on all runs matching an RQ, reaggregate metrics, and propose findings updates. No new runs — only refreshes existing data after pipeline changes.

- Triggers: "reanalyze RQ-N", "reanalyse", "Runs neu analysieren"

- Details: `.claude/skills/reanalyze/SKILL.md`

### `/build-overview`

Generates a frozen experiment-overview snapshot across all RQs under `research/reports/`. Runs `generate-snapshot-skeleton.py`, then fills synthesis sections from `findings.md` files.

- Details: `.claude/skills/build-overview/SKILL.md`

### `/exact-coding-baseline-export [line] [date] [source-workflow]`

Mint a dated consumer snapshot under `research/workflow-dev/export/` for either promoted EXACT Coding line. `opus` (default for backward compatibility) resolves the correctness-oriented Hybrid source from `workflow-construction.md`; `sol` resolves the Predictive-TDD source from `model-recommendation-matrix.md` and runs `export-sol.py`. Both remove lab content, restore HITL, and gate invocation. The SOL path emits Claude Code, pi, OpenCode, Cursor and Copilot ports and can explicitly synchronize local `sol/*` branches in `EXACT-Coding-Exercises` with `--sync-distribution`; it never pushes.

- Triggers: "exact-coding baseline export", "neue exact-coding baseline", "exact-coding-baseline-export"

- Details: `.claude/skills/exact-coding-baseline-export/SKILL.md`

### Other skills

- `/simplify` — Review changed code for reuse, quality, and efficiency; fix issues found

- `/review` — Pull-request review

- `/security-review` — Security review of pending branch changes

For one-off analyses (`analyze-run.sh`), debugging, or spot-checks: call scripts directly, don't use skills.

## Directory structure

```

experiments/

katas/ Coding tasks; each has 3 prompt variants (-prose, -user-story, -example-mapping)

workflows/ TDD agent architectures, ordered by category: exact-coding/{opus,sol}/,

baselines/, external/, _archive/. LINEAGE.yaml is the source for arm,

version, lineage, status and former names; read MARKERS.md before editing

runs/ Flat pool of all completed runs (metrics.json, transcript.jsonl, src/)

docker/ Containerized batch execution (Dockerfile, batch.sh, run-batch.sh)

batch-plans/ JSON execution plans (auto-generated by batch-plan-from-rq.py)

*.sh / *.py Lifecycle scripts: record-run, analyze-run, aggregate-by-query, batch-plan-from-rq

research/

questions/ Generic research RQs (prompt/model/TDD/context effects); dirs <chapter>-slug

workflow-dev/ Workflow-evolution RQs (hybrid reduction chain, subagents-v1→v2) + methodology docs + model-recommendation-matrix.md

reports/ Published experiment-overview snapshots (md; pdf gitignored, built on demand)

kata-dev/ Kata validation RQs (methodology, ambiguity probe and workshop material live in the

sister repo kata-builder: github.com/marcoemrich/kata-builder)

```

Each RQ dir holds `README.md`, `findings.md`, `runs.csv`, `summary.md`. The `<chapter>` prefix

(e.g. `2.6-lean-validation`) is an **ordering label, not an id** — like a document section number,

freely renumber-able. The stable identity is the frontmatter `id:` (e.g. `RQ-lean`); all tooling

resolves RQs by that id, never by the directory name. Resolve `RQ-N` → path by id-grep across both

subtrees.

## Conventions

### Language

- **Everything in this repo is written in English** — docs, findings, RQ questions, code comments, todos. The snapshot generator pulls `question:` fields and finding titles verbatim, so German in an RQ source file resurfaces in the published snapshot. Archived snapshots under `research/reports/` from before 2026-07 stay German. German workshop and kata-design material lives in the sister repo kata-builder.

### Git branches

- **Create branches only on explicit request.** Never branch automatically, not even when HEAD is on `main`. Always commit on the current branch. If a branch seems useful, ask instead of creating one unasked.

### Editing workflows

- **No numeric thresholds in workflow/agent prompts** (e.g. `cognitive_max < 15`, `LoC < 50`, `complexity ≤ 10`). Katas, scenarios and environments are too volatile — a threshold that makes sense for game-of-life is nonsense for claim-office. Use qualitative language instead ("reduce cognitive complexity", "extract when a function does multiple things"). This also applies to new refactor/review agents.

- **Read `experiments/workflows/MARKERS.md` first** (hard parser requirements — four markers drive all TDD metrics; altering one silently zeros the corresponding metric, no error). **For content design** (Theory-of-Mind / Why-Block pattern, reduction learnings from RQ-rules/RQ-pep/RQ-emoji/RQ-lean, Skill-vs-Subagent architecture gradient): `research/workflow-dev/workflow-construction.md`.

- Smoke-test after every workflow change:

```bash

jq '.summary_metrics + .final_metrics | {cycles_closed, refactor_events, red_batch_max, predictions_correct, predictions_total, tests_passing}' experiments/runs/<latest>/metrics.json

```

Healthy: `cycles_closed >= 3`, `refactor_events >= 1`, `red_batch_max == 1` for

workflows that write one test at a time, `predictions_total ~ 2 * cycles_closed`

for predictive workflows. A Predictive-TDD run on GPT-6.1 Sol / Claim Office reads

e.g. 26 / 19 / 1 / 97 / 100.

The first three come from the phase chain (the stack's TDD event reporter); `null`

there means the reporter wrote nothing, not that the agent skipped TDD.

`cycle_count` and `refactorings_applied` are gone from new runs — older runs carry

them as `legacy_cycle_count` / `legacy_refactorings_applied`, so querying the old

names returns `null` on every healthy run.

The block merge is load-bearing: the TDD fields live in `summary_metrics`, only

`tests_passing` in `final_metrics`.

### Writing findings

- `findings.md` shows **current state only**. No status tags ("revised"/"confirmed"), no legacy comparisons, no "previously X" references.

- Finding headers: `## F-x.y — Title` (no trailing suffix). The namespace before the last dot may itself carry dots — chapter-numbered ids like `F-4.4.1` or `F-1.12.5` are in use and valid. Em-dash `—` between id and title is mandatory: `generate-snapshot-skeleton.py` parses on it, and a header it cannot match is dropped silently, surfacing downstream as "no findings documented".

- Overview table comes first (pivot of primary outcome).

- **🏆 trophies in the overview table are mandatory.** Winners per column get `**…** 🏆`; on ties, all of them. Note the metric direction in the header or in the table description (`lower = better` / `higher = better`). Full convention (spread threshold, ambivalent metrics without a trophy): `.claude/skills/run-rq/SKILL.md` section "Trophy convention".

- **Always use the `/run-rq` or `/reanalyze` skill for findings updates** — the skills carry the trophy convention, the spot check before aggregation, the plausibility cross-check and the "never write automatically" rule. Writing findings ad hoc without a skill misses these rules (happens reproducibly).

- **Glossary terms from README are binding** — use the exact forms: Correctness (internal) / Correctness (external) / Code Mass (APP) / Smell Total / Production LoC / Test LoC / Mutation Score. "Code-Volumen" and "LoC-Groesse" are forbidden synonyms, as are the German forms these replaced in 2026-07 ("Korrektheit (innen/außen)", "Code-Mass", "Smell-Summe", "Produktiv-LoC", "Spitzen-Komplexität").

- **`tdd_discipline` is not comparable across the test-list boundary.** Affected: `tdd_discipline`, `tdd_discipline_test_first`, `tdd_discipline_closure`, `test_first_rate`, `skip_events`, `cycles_total`, `chain_deviations`. A `Skip` — a test that arrived and passed immediately — is **compliance** in a test-list workflow (EXACT Coding's Predictive TDD: "A test already satisfied by an earlier generalization is legitimate evidence. Do not manufacture a failure.") and a reason to start over in Superpowers ("Delete code"). The metric therefore penalises test-list completeness rather than measuring slack discipline: measured 16.6–25.8 skips per run in the four test-list workflows against 2.4–8.2 in the three ad-hoc ones, at equal or higher red counts. Report these columns **within** an architecture group, never across one, and say why in findings.md. `aggregate-by-query.py` warns when an RQ declares one of them and its cells span the boundary. Always comparable: `tdd_discipline_step`, `red_batch_size`, `red_batch_max`, `green_batch_size`, `refactor_events`, `cycles_closed`, `chain_suite_runs` — for "one test at a time" the column is `red_batch_max`, not the score. Full argument: README "Comparability: the test-list boundary"; evidence in RQ-tdd-workflow-comparison-opus55 F-4.13.4.

- **The complexity metrics have no prose name.** Write `longest_function`, `cognitive_max`, `cognitive_avg` or `mccabe_max` in backticks. "Complexity Peak", "Spitzen-Komplexität", "Cognitive Complexity peak" and "McCabe peak" are all forbidden. The term was retired 2026-09-30 because it had been used for all three: `longest_function` in the kata-dev and workflow-dev RQs, `cognitive_max` in RQ-opus55-current-workflow, and `cognitive_max`/`mccabe_max` jointly in RQ-cost-sol-pi-vs-opus-cc. Two findings tables using the same word were not comparable.

- Open hypotheses live in the RQ's `README.md`, not in findings.md or separate files.

### Running batches

- Start with `./batch.sh <plan>` using Bash tool `run_in_background: true`. Do NOT combine with `nohup` or `&` — that breaks tool tracking.

- `--shards N` for parallel containers (default 5). 5–10 shards are routine — a container

uses ~170 MiB in practice; the `memory: 4G` in `docker-compose.yml` is a cap, not a

requirement (measured 2026-08-17: 5 shards ≈ 865 MiB total, CPU I/O-bound). The real

ceiling is API rate limits, which `batch.sh` already retries against.

- **Drop to 1–2 shards only against a rate-limit risk**, never for memory. The risk is

route- and model-specific: it has repeatedly hit Opus batches, while pi/Sol routes are

uncritical. Memory is never the reason to reduce shard count.

- Rate-limit resilience is built in: 5 retries with backoff (60s → 5min → 30min → 1h → 2h). Do not add manual retry logic.

- **CC routing is container-global.** The `ANTHROPIC_*` env vars apply to the whole batch container, not per run. A plan must therefore not mix CC runs with different CC routing (e.g. CC-requesty + CC-native) — split into two plans, run sequentially. Mixing CC-requesty + OC-requesty + pi-requesty is fine (separate routing channels: `.env` for CC, `opencode.json` for OC, `models.json` for pi).

- **Run completion signal:** `metrics.json | jq .run_status.exit_reason` — NOT presence of `analysis-report.md`.

- **Only `ok` runs count toward `min_replicates`.** A timeout, an infrastructure abort, a provider refusal or an unrecognised `error-N` is not a replicate; the next fill run (`batch-plan-from-rq.py`) plans a replacement. Nothing refills automatically and there is no cap — whether to start another fill run for a cell that keeps timing out is the user's call. A timeout is not retried inside the batch. Rule: `is_replicate()` in `experiments/rq_facts.py`.

- **No in-place edits to `run-batch.sh` (or other mounted scripts) while a batch is running.** Bash reads a script incrementally by byte offset, so rewriting one in place shifts the offsets of the lines not yet read: the running container aborts with `syntax error near unexpected token` or `0: command not found` on a line that is perfectly intact on disk (`bash -n` says "ok"). The run already finished stays valid; the loop after it dies. Wait until the batch is done.

- **Only in-place writers are dangerous, and that is a smaller set than it looks.** `sed -i` without a temp file, an editor that truncates and rewrites, a redirect onto the live path — those hit the case above. A write that lands via **rename** does not: `git checkout`, `git merge`, `git pull`, or `mv` of a temp file. Each script is bind-mounted as a *single file*, which pins the inode at mount time, so the rename creates a new inode on the host while the container keeps reading the one it opened. Measured 2026-10-01 during a 5-shard batch: a merge rewrote `run-batch.sh` and all five containers ran on unaffected — host inode 690646 against container inode 690552, different md5. **The corollary cuts both ways.** A deliberate fix to a mounted script does not reach a running container either, and "restoring" the previous content to protect a batch is the one move that can actually break it. Diagnose instead of guessing: `docker exec <container> md5sum /home/experimenter/experiments/run-batch.sh` against the host file.

### Stacks

- A stack is a directory under `experiments/stacks/<slug>/` with an

executable `install.sh`. `run-batch.sh` copies its content into the run directory

(without `install.sh` itself) and calls the script from there — there is no longer any

language branching in the runner. Four stacks: `typescript-vitest`,

`java-junit-maven`, `python-pytest`, `rust-cargo`.

- **A run's language is in `metrics.json.stack`**, written when the run is

created. `analyze-run.sh` (`run_stack`) and `compute-mutation-score.py`

dispatch on it. Do not add new `test -f pom.xml` checks: they

contradict `metrics.json` as soon as a run is analyzed outside its container,

and each of them is one more place the next language has to touch.

The file sniff remains only as a fallback for runs from before the `stack` field.

- New language = stack directory **including a TDD event reporter** hooked into the

test framework, kata variants (`<kata>-<language>-example-mapping`

plus `-verification`), a stack profile in the four maintained `exact-ptdd-*` workflows,

one arm each in `analyze-run.sh` and `compute-mutation-score.py`, and `required_stacks`

in `export-sol.py`.

- **Without the reporter every TDD-discipline column is `null`, silently.** The phase

chain is the only source of TDD discipline, and it reads nothing but the

`tdd-events.jsonl` the stack's reporter appends on each suite invocation. The run

still completes and every quality metric is filled, so nothing fails — this is exactly

how Java and Python looked until 2026-10-01. The reporter must honour

`TDD_REPORTER_OFF` (otherwise `analyze-run.sh` and mutation testing append their own

events), write nothing to stdout/stderr, and swallow its own exceptions. Existing

hooks per stack and the Java compile-step gap: README, "Phase chain metrics".

The regexes in `measure-tdd-rigour.py` only feed that legacy tool and are optional.

- **Quality values are never comparable across stacks.** ESLint/SonarJS, PMD,

ruff/complexipy and clippy/rust-code-analysis produce different sets of findings;

the same holds for Stryker/PIT/mutmut/cargo-mutants. Across RQs only directions and

orderings are compared.

### Aggregation

- **Never average across katas.** string-calculator (3 LoC) vs. game-of-life (30+ LoC) would smear the signal.

- Aggregation is query-based: all runs in `experiments/runs/` matching the RQ selector count, regardless of which batch produced them.

## Metrics quick reference

| Prefix | Meaning | NOT |

|--------|---------|-----|

| `production_loc`, `functions`, `longest_function`, `*_loc_per_function` | **Clean Code** surface metrics (Production LoC, function count, longest function, avg/median LoC per function), from `clean_code.*` in `metrics.json`. The former `cc_` prefix is gone | McCabe / Cyclomatic |

| `mccabe_*` | McCabe cyclomatic complexity (max, avg, high_count) | |

| `cognitive_*` | Cognitive complexity (max, avg, high_count) — SonarJS on TS, PMD on Java, complexipy on Python, rust-code-analysis on Rust | |

| `unit_*` | Size of the smallest named unit: `unit_count`, `unit_size_{max,avg,median}`. PMD NCSS statements on Java, lines per function on TS, Python and Rust | the old `java_method_ncss_*` spelling |

- `verification_pct` (0.0–1.0) = external acceptance score for CLI katas (claim-office). `tests_passing` = the internal suite's pass/fail — vitest, Maven Surefire, pytest or `cargo test`, depending on the stack.

- `mutation_score` (0.0–1.0) = mutation score — Stryker on TS, PIT on Java, mutmut on Python, cargo-mutants on Rust (`experiments/compute-mutation-score.py` picks the engine per run from `metrics.json.stack`). **Opt-in per RQ** (must appear in `outcomes:`) and only computed for `tests_passing = true`. Run between batch and aggregation. On TS it is expensive (minutes per run, `pnpm install` per run), so do not add it to `analyze-run.sh` or routine reanalysis; Java, Python and Rust cost seconds per run. **Scores are not comparable across stacks** — the four tools generate different mutant populations, and PIT's default mutator set is the narrowest. TS, Python and Rust exclude the CLI adapter from mutation because only the external acceptance suite exercises it; Java deliberately does not, because Java runs often nest the whole domain inside the CLI class.

- `mutants_total` / `mutants_survived` = the counts behind `mutation_score` (population, and the

part of it the suite missed). Written by the same script and opt-in the same way. **Report the

pair, not the ratio alone, whenever the arms differ in code size** — the score's denominator is

the mutant population, so it can rise while the absolute gap stands still.

- Full metrics table in README section "Metrics".

## Host dependencies

- The analysis scripts are stdlib-only **except `aggregate-by-query.py`**, which needs

`pandas`, `PyYAML` and `tabulate`. Declared, pinned to the host's versions, in

`experiments/pyproject.toml` (uv, lock in `experiments/uv.lock`, venv in the gitignored

`experiments/.venv`). On a machine without them:

`uv run --project experiments python experiments/aggregate-by-query.py <rq-dir>`.

- **Tests for the scripts:** `cd experiments && uv run pytest`. Hyphenated scripts load

through `load_script()` in `experiments/tests/conftest.py`; do not rename them. ruff

(`uv run ruff check <files>`, the Python stack's lab rules) gates only the files a change

touches — the rest of `experiments/` predates the gate.

- **`tabulate` is never imported by our code** — pandas pulls it in for

`DataFrame.to_markdown()`, which writes `summary.md`. Missing it fails the

aggregation *after* `runs.csv` is already written, so the run looks half-successful

and no grep over imports reveals the dependency.

- System Python on this host is PEP 668 externally managed; a plain `pip3 install --user`

is refused. Use the uv environment above instead.

- **A host pnpm newer than the container pin turns green TS runs red.** pnpm ≥ 10

no longer reads the `pnpm` key in `package.json` and writes a warning about it to

**stderr**. Suites that start the CLI via `spawnSync("pnpm", …)` and correctly

assert `stderr === ""` then fail for a reason that has nothing to do with the

agent's code: `analyze-run.sh` writes `tests_passing: false` **and zeroes coverage**,

`exit_reason` stays `ok`, and in the aggregation the cell looks like a workflow

finding. Measured 2026-09-23: host pnpm 11.25.0 against pin 9.15.9, 4 of 40 runs

flipped, all with a CLI stderr assertion; with pnpm 9.15.9 on the PATH the same

suite was 41/41 green.

`analyze-run.sh` and `compute-mutation-score.py` therefore remove the `pnpm` key

for the duration of the test run and restore the file verbatim afterwards.

The warning about the version mismatch stays in place as well — whoever redirects

it into per-run logs does not see it, so after a host reanalysis check

`tests_passing` and `coverage` against the previous state (`git diff` on the

`metrics.json`) instead of only watching for script errors.

- **Java mutation testing needs a JDK ≤ 21 on the host**, independent of the host default.

PIT's bundled ASM cannot read Java 25 class files, and it reads core classes from the

*running* JVM — on a JDK 25 host that fails as `Unsupported class file major version 69`

for some runs and not others. `compute-mutation-score.py` therefore pins `JAVA_HOME` to

`/usr/lib/jvm/java-17-openjdk-amd64` (the same JDK the container compiles with), falling

back to java-21 and warning if neither exists.

- **Rust analysis and mutation testing run in the container, not on the host.** The Rust

toolchain is the one stack's toolchain this host is not expected to carry, so both

wrappers exist: `reanalyze-in-container.sh` for `analyze-run.sh` (clippy,

`cargo-llvm-cov`, `rust-code-analysis-cli`) and **`mutation-in-container.sh <rq_dir>`**

for `compute-mutation-score.py` (`cargo-mutants`). Both use the image the runs were

produced with, so the engine version cannot drift from a second hand-maintained pin.

Then run `aggregate-by-query.py` **on the host** so the new fields reach `summary.md` —

writing the pivots is the one step that needs pandas.

- **A host install is still possible but is the worse path.** `cargo install --locked

cargo-mutants@27.1.0` against a host toolchain at the image's `1.98.1` works, and

`compute-mutation-score.py` warns on a version mismatch rather than mixing engines

silently. It needs a full Rust toolchain on the host and pins the engine in a second

place. On TypeScript, Java and Python the host path stays the cheaper default — no

container start.

- **`compute-mutation-score.py` must stay importable without pandas.** It reaches

`parse_frontmatter()` in `aggregate-by-query.py`, which therefore imports pandas

**lazily** (`_pd()`); PyYAML is a hard import and is installed as `python3-yaml` in the

image. Moving `import pandas` back to module scope silently breaks every in-container

analysis with `ModuleNotFoundError`, and nothing else notices, because the host has

pandas.

## Docker & version pins

- **Rust: `1.98.1`**, with `cargo-mutants@27.1.0`, `cargo-llvm-cov@0.9.1` and `rust-code-analysis-cli@0.0.25`, all built in the Dockerfile's `rust-tools` stage. The registry is baked into the image from `experiments/docker/rust.cache/` (a mirror of the stack's `Cargo.toml`/`Cargo.lock`, kept in step like `python.cache.txt`) and the stack runs Cargo offline. `/usr/local/bin/cargo` is a launcher for the mounted `experiments/rust_tdd.py`, which records `cargo test` as TDD events — so a fix to the recorder needs no rebuild, but **a container without that mount records nothing, silently.**

- **Claude Code CLI: `2.1.280`** — every model generation has raised the floor, and as a hard API 400 rather than a silent fallback: 2.1.170 for Fable 5, 2.1.267 for Fable 5.1, 2.1.280 for Opus 5.5 (a smoke run on 2.1.267 died after 2 s). 2.1.267 also does not carry Opus 5.5 in its catalog, which silently caps auto-compact at 200k on a 1M-context model. Known bad: 2.1.37 hangs on `.claude/agents/` dirs, 2.1.126 requires a missing `.claude.json`. Do not bump without verifying a subagents-arm workflow end-to-end. The reason per version is commented in `experiments/docker/Dockerfile` — that is the source, this line is the summary.

- **Other harness pins:** `opencode-ai@1.15.10`, `@earendil-works/pi-coding-agent@0.81.1`, `cursor-agent` (dashboard API key). All in `experiments/docker/Dockerfile`.

- **pnpm: `9.15.9`** — pnpm 11 breaks builds via `ERR_PNPM_IGNORED_BUILDS`. Pinned via `npm install -g pnpm@9.15.9` in Dockerfile.

- **uv: `0.12.17`** — package manager of the `python-pytest` stack, pinned as `ENV UV_VERSION` in the Dockerfile. `uv venv` brings its own virtualenv and needs **no** `python3-venv`; the package is therefore deliberately not installed. The uv cache is pre-warmed at image build from `experiments/docker/python.cache.txt` (a mirror of `experiments/stacks/python-pytest/requirements-dev.txt`, because the build context is `experiments/docker/`) and lives in the `uv-cache` volume. An install is therefore offline and takes ~0.2 s.

- **`docker compose build experiment` builds only ONE of three images.** The services `experiment`, `batch` and `batch-retry` share context and Dockerfile, but each gets its own tag (`docker-experiment`, `docker-batch`, `docker-batch-retry`). Batches run on `docker-batch` — building only the `experiment` service leaves them running on an arbitrarily old image. After every Dockerfile change: `docker compose --profile batch --profile batch-retry build`. Symptom if you forget: a newly installed tool is `command not found` in the batch while it is present in `docker compose run experiment`.

- Container uses `experiments/docker/claude-config/` for Claude config, **not** the host `~/.claude`. This separation is intentional (host config has fish/MCP spawns that hang in the container).

- **pi models are wired in two places, and only one of them is required to run.** `run-batch.sh` (MODEL_CONFIGS + `pi_model` case map) is enough for a run; if the entry in `pi-config/agent/models.json` is missing, it still runs — pi logs `Model "<id>" not found for provider`, passes the id through and then prices it **at another model's rate**. `exit_reason` stays `ok`, `metrics.json` names the right model, quality metrics are correct; only `cost_usd` is made up. `experiments/check-pi-model-wiring.py` checks both sources against each other **and** against the price table, and runs as a pre-flight warning in `batch.sh`. New pi model = three entries: `run-batch.sh`, `models.json`, and `PRICES` in `compute-cost.py`.

- **`cost_usd` is a list-price comparison value, not an invoice amount** — "what this would have cost via the API". For **all** pi runs on **both** routes, `PRICES` in `compute-cost.py` is the sole source (`cli_model == "pi-only"`); pi's own inline costs are not adopted because they cannot be reproduced from our token counts (fit over 83 codex runs: negative input price, 33.6 % error — presumably the >272k rate steps apply per request). On the flat-rate subscription nothing is billed per token; the number nevertheless stays the comparison value, **not** 0. Long-context rate steps are deliberately not modeled — the values are a consistent lower bound. Rates and sources: `research/model-pricing.md`.

- Routing: all harnesses run via **Requesty** (Portkey shut down 2026-07). Controlled purely via `experiments/docker/.env` (`ANTHROPIC_BASE_URL`/`ANTHROPIC_AUTH_TOKEN` for CC via `env_file`; OC via the requesty provider in `.opencode/opencode.json`; pi via `pi-config/agent/models.json`). No more `batch.sh` auto-detect — `-portkey` suffixes on old model IDs are only a label now. `ANTHROPIC_AUTH_TOKEN` is fed from `${REQUESTY_API_KEY}` in `docker-compose.yml`.

## Model IDs

Use **lab-variant IDs** in RQ frontmatter and batch plans: `opus-4-7`, `opus-4-7-no-thinking`, `sonnet-4-6`, `sonnet-4-6-no-thinking`, `haiku-4-5`, `haiku-4-5-no-thinking`. Add `-portkey` suffix for Portkey-routed variants.

**Do NOT use short aliases:** `opus` resolves to `claude-opus-4-6` (not 4.7!), `sonnet` to `claude-sonnet-4-6`. Always pin the full API ID in scripts.

Full model table: README section "Model Configurations" and `MODEL_CONFIGS` in `experiments/docker/run-batch.sh`.

## RQ methodology

- **Current state:** Active RQs split by harness family — `research/questions-{claude,opencode,pi,cursor-cli}/` (per-harness model/prompt/TDD effects), `research/questions-cross/` (cross-harness comparisons), `research/workflow-dev/` (workflow-evolution chain) and `research/kata-dev/` (kata validation). Published reports live in `research/reports/`.

- **No RQ status field.** Whether an RQ needs work is read off the data: `experiments/rq-status.py` shows per RQ the cells at `min_replicates`, the findings count, and runs newer than `findings.md` (rules: `experiments/rq_facts.py`). "What is still open?" → `experiments/rq-status.py --open`. Runs newer than the findings mean a sister RQ filled a shared cell — reanalyze that RQ. The only hand-set field is `closed: "<reason>"` for an RQ that ends without full data (abandoned, deprecated, accepted replicate gap) — set it only on the user's decision.

- **Constraint order** when user names an early constraint (model pinning, routing, tool availability): plan-context first → RQ schema → reuse-check. Do not reverse.

- `baseline-oneshot-*` / `baseline-iterative-*`: only `prose` prompt style. All other arms

(`baseline-inline-tdd-*`, `exact-subagents-*`, `exact-single-context-*`, `exact-hybrid-*`,

`exact-green-refactor-*`, `exact-sol-*`): all three styles allowed.

- Kata assignment: `claim-office` → correctness (`verification_pct`), `game-of-life`/`mars-rover` → code quality.

- Never mix control values into a controlled RQ — opens it as uncontrolled factor. Open a new RQ instead.

- **Exception: `controls.model: {any: [...]}`** for routing variants of the same underlying model (e.g. Portkey + Direct API of `opus-4-7-no-thinking`). OR-match — all listed values count toward the same cell, first entry is the canonical **cell label**. Use only when routing is assumed not to affect the outcome; document as caveat in the RQ README. **Do not** use `any:` for different models — that would silently smear the model factor.

- For **fill runs** the label is chosen per harness, not from the first entry: `batch-plan-from-rq.py` picks the alternative that the target harness's case-map in `run-batch.sh` actually knows (pi wants bare `opus-4-8`, cursor wants `opus-cursor`) and reports each swap on stderr. So a cross-harness `any:` list should contain a spelling for every harness the RQ covers.