report git:20260726.1a24604A

Immutable. This exact content never changes and is served at /api/v1/blob/83ce706d0777e069.

---
name: report
description: Summarize a run for a human — baseline val → best val → sealed test, the winning candidate, iterations spent, and pass^k. Use after finalize. Writes report.md and prints a compact JSON summary; the source of truth for "did this optimization actually work, and by how much".
component: phase
argument-hint: "--run-dir DIR"
allowed-tools: Read, Write, Bash
provides: [report]
needs: []
sources: [evo]
---

# report — did it work, and by how much?

The result of an optimization run is not "we made edits" — it is a defensible
answer to *did this actually work, and by how much.* report reads the run dir's
`baseline.json` and `final.json` and lays them side by side: where the seed
started (val), where the best candidate landed (val), and the single **held-out
test** number that actually counts. It is the source of truth a human reads to
decide whether to ship the optimized capability.

## Inputs / outputs (manifest tokens)
- **needs:** *(reads the run dir directly — baseline.json / final.json / events /
  per-task rollouts / git store)*.
- **provides:** `report` — `report.md`, a compact JSON summary, a self-contained
  `dashboard.html`, and an on-demand ANSI terminal report.

## How to read the three numbers
The honest reading is always **test vs baseline**, with val as a sanity check in
between:

- **test ≈ baseline** → no real gain. The val improvement was overfitting or
  noise the gate let through. Do not ship; tighten `gate_k_se` or add trials.
- **test ≫ baseline** → genuine improvement on data the optimizer never saw. Ship.
- **val ≫ test** → the classic overfit signature: the optimizer learned the val
  set, not the capability. The gap *is* the overfitting, quantified.
- **wide pass^k drop vs pass^1** → the gain is *fragile* across trials — the agent
  sometimes succeeds but not reliably. A high mean with low pass^k is not a
  dependable win (τ-bench's point).

Always report the **test stderr / CI**, not just the point: "0.71" and
"0.71 ± 0.08" support very different decisions.

When the run declared a **consuming LLM** (the `target_profile` in the run summary —
the runtime model the capabilities were optimized FOR, distinct from the optimizer
model that proposed the edits), print it alongside the optimizer model so the two LLM
roles stay visibly distinct: e.g. "optimized for gpt-oss-120b (tier mid); edits proposed
by claude-opus". Omit the line for profile-agnostic runs (no `target_profile`).

## Dual-mode
This phase runs two ways from the **same** SKILL.md: standalone as the slash command `/cap-evolve:report` (the `argument-hint` shows its run.py args), and orchestrator-callable — `cap-evolve run` / the `orchestrate` skill invokes the same `scripts/run.py` headlessly and threads the run dir between phases.

## How to run
```
python scripts/run.py --run-dir .capevolve/run_XXXX            # JSON summary + report.md + dashboard.html
python scripts/run.py --run-dir .capevolve/run_XXXX --terminal # colored in-chat ANSI report (no browser)
python scripts/run.py --run-dir .capevolve/run_XXXX --no-dashboard
```

### The dashboard (`dashboard.html`)
A **fully self-contained** static file — inline CSS + vanilla JS + inline SVG, **no
CDN, no server, no network**. It is the single shareable artifact; open it by
double-clicking (works from `file://`). The builder first **reduces** the event log
(plus baseline/final, per-task rollouts, and the git store) into a candidate **graph
+ run-summary**, then renders eight panels:

1. **KPI strip** — best / baseline / %Δ-vs-baseline (abs Δ off a zero baseline),
   counts by status (accepted/rejected/failed/seed), frontier, sealed test, wall
   clock, cost split **optimizer vs runner**, tokens.
2. **Cumulative-best stair** over a per-iteration scatter — running-best step
   polyline, champion star, record-holder rings, hover tooltip (id/status/val/Δ).
3. **tasks×iterations pass/fail heatmap** — rows sorted worst-first; reveals
   regressions, persistent failures, and per-task specialists the mean hides; click
   a cell for that task's feedback.
4. **Diff vs parent** — prompt/skill/tools diff of a candidate against its parent
   (split/unified toggle), computed from the candidate dirs.
5. **Lineage tree** — parent→child DAG (merges = multi-parent), best-lineage spine
   highlighted.
6. **Cost / tokens / latency** — per-iteration + cumulative, split optimizer vs
   runner, plus a cumulative-cost-vs-best-score plot.
7. **Annotations & diagnoses** — gate reasons + diagnose/optimizer-error output.
8. **Candidate leaderboard** + the git iteration-store log.

All values pass through a recursive **secret redactor** first, so a shared dashboard
never leaks `RITS_API_KEY` / `BOBSHELL_API_KEY` / `WATSONX_*` etc. Optional panels
**degrade silently** when per-task data, diffs, or finalize are missing.

### The terminal report (`--terminal` / `--ansi`)
A colored cumulative-best chart + one-line KPI strip + top-N candidate table for
in-chat progress without opening the browser. Sized to the terminal width and
**CLAUDECODE-margin-aware** (subtracts ~6 cols when `CLAUDECODE=1` so it doesn't wrap
inside the tool-output frame). `--no-color` / `NO_COLOR=1` for plain text;
`--top-n N` sets the table size.

## What good vs bad looks like
- **Good:** test reported with its uncertainty and pass^k; the baseline→val→test
  story told honestly, including a no-gain or overfit result when that is what
  happened; a no-holdout run clearly labelled as a fit metric.
- **Bad:** reporting val as if it were the result; a bare point estimate with no
  variance; spinning a `val ≫ test` overfit as success.

## References
- `references/concepts.md` — reading baseline/val/test honestly, the val-test gap
  as overfitting, pass^k fragility, and reporting uncertainty, with sources.
- `references/dashboard.md` — the reduced graph + summary schema, what each of the
  eight panels reads from, the `--terminal`/`--ansi` mode, secret redaction, and the
  graceful-degradation rules.