report git:20260617.4c87ed1A

Immutable. This exact content never changes and is served at /api/v1/blob/589cae78ec602fa6.

---
name: report
description: Summarize a run for a human — baseline val → best val → sealed test, the winning candidate, iterations spent, and pass^k. Use after finalize. Writes report.md and prints a compact JSON summary; the source of truth for "did this optimization actually work, and by how much".
component: phase
argument-hint: "--run-dir DIR"
allowed-tools: Read, Write, Bash
provides: [report]
needs: []
sources: [evo]
---

# report — did it work, and by how much?

The result of an optimization run is not "we made edits" — it is a defensible
answer to *did this actually work, and by how much.* report reads the run dir's
`baseline.json` and `final.json` and lays them side by side: where the seed
started (val), where the best candidate landed (val), and the single **held-out
test** number that actually counts. It is the source of truth a human reads to
decide whether to ship the optimized capability.

## Inputs / outputs (manifest tokens)
- **needs:** *(reads the run dir directly — baseline.json / final.json / events /
  per-task rollouts / git store)*.
- **provides:** `report` — `report.md`, a compact JSON summary, a self-contained
  `dashboard.html`, and an on-demand ANSI terminal report.

## How to read the three numbers
The honest reading is always **test vs baseline**, with val as a sanity check in
between:

- **test ≈ baseline** → no real gain. The val improvement was overfitting or
  noise the gate let through. Do not ship; tighten `gate_k_se` or add trials.
- **test ≫ baseline** → genuine improvement on data the optimizer never saw. Ship.
- **val ≫ test** → the classic overfit signature: the optimizer learned the val
  set, not the capability. The gap *is* the overfitting, quantified.
- **wide pass^k drop vs pass^1** → the gain is *fragile* across trials — the agent
  sometimes succeeds but not reliably. A high mean with low pass^k is not a
  dependable win (τ-bench's point).

Always report the **test stderr / CI**, not just the point: "0.71" and
"0.71 ± 0.08" support very different decisions.

## Dual-mode
This phase runs two ways from the **same** SKILL.md: standalone as the slash command `/cap-evolve:report` (the `argument-hint` shows its run.py args), and orchestrator-callable — `cap-evolve run` / the `orchestrate` skill invokes the same `scripts/run.py` headlessly and threads the run dir between phases.

## How to run
```
python scripts/run.py --run-dir .capevolve/run_XXXX            # JSON summary + report.md + dashboard.html
python scripts/run.py --run-dir .capevolve/run_XXXX --terminal # colored in-chat ANSI report (no browser)
python scripts/run.py --run-dir .capevolve/run_XXXX --no-dashboard
```

### The dashboard (`dashboard.html`)
A **fully self-contained** static file — inline CSS + vanilla JS + inline SVG, **no
CDN, no server, no network**. It is the single shareable artifact; open it by
double-clicking (works from `file://`). The builder first **reduces** the event log
(plus baseline/final, per-task rollouts, and the git store) into a candidate **graph
+ run-summary**, then renders eight panels:

1. **KPI strip** — best / baseline / %Δ-vs-baseline (abs Δ off a zero baseline),
   counts by status (accepted/rejected/failed/seed), frontier, sealed test, wall
   clock, cost split **optimizer vs runner**, tokens.
2. **Cumulative-best stair** over a per-iteration scatter — running-best step
   polyline, champion star, record-holder rings, hover tooltip (id/status/val/Δ).
3. **tasks×iterations pass/fail heatmap** — rows sorted worst-first; reveals
   regressions, persistent failures, and per-task specialists the mean hides; click
   a cell for that task's feedback.
4. **Diff vs parent** — prompt/skill/tools diff of a candidate against its parent
   (split/unified toggle), computed from the candidate dirs.
5. **Lineage tree** — parent→child DAG (merges = multi-parent), best-lineage spine
   highlighted.
6. **Cost / tokens / latency** — per-iteration + cumulative, split optimizer vs
   runner, plus a cumulative-cost-vs-best-score plot.
7. **Annotations & diagnoses** — gate reasons + diagnose/optimizer-error output.
8. **Candidate leaderboard** + the git iteration-store log.

All values pass through a recursive **secret redactor** first, so a shared dashboard
never leaks `RITS_API_KEY` / `BOBSHELL_API_KEY` / `WATSONX_*` etc. Optional panels
**degrade silently** when per-task data, diffs, or finalize are missing.

### The terminal report (`--terminal` / `--ansi`)
A colored cumulative-best chart + one-line KPI strip + top-N candidate table for
in-chat progress without opening the browser. Sized to the terminal width and
**CLAUDECODE-margin-aware** (subtracts ~6 cols when `CLAUDECODE=1` so it doesn't wrap
inside the tool-output frame). `--no-color` / `NO_COLOR=1` for plain text;
`--top-n N` sets the table size.

## What good vs bad looks like
- **Good:** test reported with its uncertainty and pass^k; the baseline→val→test
  story told honestly, including a no-gain or overfit result when that is what
  happened; a no-holdout run clearly labelled as a fit metric.
- **Bad:** reporting val as if it were the result; a bare point estimate with no
  variance; spinning a `val ≫ test` overfit as success.

## References
- `references/concepts.md` — reading baseline/val/test honestly, the val-test gap
  as overfitting, pass^k fragility, and reporting uncertainty, with sources.
- `references/dashboard.md` — the reduced graph + summary schema, what each of the
  eight panels reads from, the `--terminal`/`--ansi` mode, secret redaction, and the
  graceful-degradation rules.