skillopt · diff
git:20260722.5d6d913 to git:20260822.65c52f3
112 added, 60 removed. Audit A to A.
---
name: skillopt
- description: Runs the SkillOpt single-lineage optimization loop over epochs x mini-batches with a textual learning rate (an integer edit budget that decays on a constant|linear|cosine schedule), a within-epoch rejected-edit + failure-pattern buffer injected into the optimizer prompt, and a gated epoch-boundary slow/meta update that fixes longitudinal regressions. Parent is always the current best; acceptance is the val significance gate; the test split stays sealed. Use when a run benefits from disciplined annealing and per-epoch consolidation rather than hill-climb's one-shot whole-trainset proposals or gepa's Pareto frontier.
+ description: Runs the SkillOpt single-lineage optimization loop, which organizes a hill-climb into epochs over mini-batches of train tasks under a textual learning rate — an integer edit budget that decays on a constant|linear|cosine schedule — and ends each epoch with one extra gated consolidation step. Parent is always the current best; acceptance is the val significance gate. Use when a run should anneal from broad early edits to small late ones and consolidate once per epoch, rather than hill-climb's one-shot whole-trainset proposals or gepa's Pareto frontier.
component: algorithm
- argument-hint: "--run-dir DIR --project DIR --optimizer CMD [--epochs 4] [--batch-size N] [--edit-budget 4] [--lr-schedule cosine] [--min-edit-budget 2] [--no-slow-update]"
+ argument-hint: "--run-dir DIR --project DIR --optimizer CMD [--epochs 4] [--batch-size N] [--accumulation 1] [--edit-budget 4] [--min-edit-budget 2] [--lr-schedule cosine] [--no-slow-update]"
allowed-tools: Read, Write, Bash
provides: [candidate]
needs: [scores, traces, candidate]
---
# skillopt — annealed single-lineage climb (epochs × mini-batches)
- SkillOpt (arXiv:2605.23904) borrows the deep-learning training loop. It is a
- strict single-lineage climber (parent is always the current best, like
- hill-climb) but adds three things from the DL analogy:
+ SkillOpt (arXiv:2605.23904, *Executive Strategy for Self-Evolving Agent Skills*)
+ organizes a hill-climb into **epochs × mini-batches** under a decaying integer
+ **edit budget**. The name is the paper's; the algorithm edits whatever the
+ selected capability owns — a prompt, a tool surface, a skill package — and never
+ assumes which.
- - a **textual learning rate** — an integer **edit budget** `L` per step, large
- early (explore broadly) and shrinking later (consolidate), on a
- `constant | linear | cosine` schedule;
- - a per-epoch **rejected-edit + failure-pattern buffer** injected into the next
- step's prompt (don't re-propose dead ends; these failures remain unsolved);
- - an **epoch-boundary slow / meta update**: re-evaluate the epoch-start skill vs
- the current best on a small train sample, categorize improved / regressed /
- persistent / stable, then run ONE extra **gated** step to fix regressions
- without breaking stable successes.
+ **Read the shared step first**, then this file. Parent materialization, the
+ optimizer call, the val evaluation, the significance gate, accept/reject,
+ snapshot/best, the memory and handover files: all of that is
+ `harness.run_step`, documented once in `algorithms/hill-climb/SKILL.md`
+ § "One iteration, end to end" and `algorithms/hill-climb/references/run-step.md`
+ (both land with PR #370; until then read `harness.run_step` itself). This file
+ states only what SkillOpt does differently.
- Every step calls the shared `run_step` (materialize → optimize → eval-VAL →
- significance gate → accept/reject → snapshot/best, with RejectedMemory/History);
- this skill only owns the schedule, the buffer, and the slow update. Gated on val;
- test stays sealed (that's `finalize`).
+ Know the bound before reaching for this algorithm: **`run_step` lets a caller
+ vary exactly two things — `parent_dir` and `instructions`.** SkillOpt pins
+ `parent_dir` to the current best, identical to hill-climb, so everything novel
+ lives in the `instructions` string plus the choice to run one extra step per
+ epoch. It is prompt shaping and step scheduling, not a different search.
- ## The DL analogy
+ ## What SkillOpt does differently
- | deep learning | SkillOpt |
- |---|---|
- | epoch over the dataset | epoch over the **train** ids (shuffled, seeded by epoch) |
- | mini-batch / gradient accumulation | `--batch-size` train tasks × `--accumulation` per step |
- | learning-rate schedule | **edit-budget** `L` schedule (`--lr-schedule`, decays `--edit-budget` → `--min-edit-budget`) |
- | gradient step | one `run_step` (a bounded edit, gated on val) |
- | momentum / replay | the per-epoch rejected-edit + failure-pattern buffer |
- | LR warm-restart / fine-tune | the epoch-boundary **gated** slow/meta update |
+ 1. **A decaying integer edit budget `L`.** `lr_schedule.build_schedule` emits one
+ integer per step over `constant | linear | cosine`, clamped to
+ `[--min-edit-budget, --edit-budget]` (`core/cap_evolve/lr_schedule.py:42-55`).
+ `L` is stated to the optimizer in prose — "at most L bounded edits" — and is
+ never mechanically enforced. See the next section before you tune it.
+ 2. **A per-epoch rejected-edit list.** Each reject appends its candidate id and
+ val Δ, and the next step's prompt asks the optimizer to avoid them
+ (`skillopt.py:120-125`, `:322-327`). It carries no description of *what* the
+ rejected edit changed, so treat it as a weak signal — the run-global
+ `LEDGER.md` that `run_step` already injects names the tasks each prior edit
+ broke and fixed, which is strictly more useful.
+ 3. **One extra gated step per epoch boundary** (from epoch 2). It compares the
+ epoch-start candidate against the current best, buckets tasks as
+ regressed / persistent-failure / stable-success, and asks for a consolidating
+ edit that fixes regressions without breaking the stable passes. It goes
+ through the same `run_step` and the same val gate — it is never
+ force-accepted (`skillopt.py:479-485`). Disable with `--no-slow-update`.
+ A fourth bucket, `improved`, is computed and logged but is *not* exclusive
+ with the others and never reaches the prompt (`skillopt.py:184-191`, `:139-166`).
- `L` is communicated to the optimizer in **natural language only** ("make at most
- L bounded edits") — the LLM is not mechanically clipped — so the loop logs
- *requested-vs-applied* edits to surface an optimizer that ignores its budget.
+ Epochs shuffle the train ids seeded by epoch number, so a rerun is reproducible.
+ `steps_per_epoch = ceil(len(train) / (batch_size × accumulation))`.
- ## When to use vs hill-climb / gepa
+ ## The textual learning rate, stated without the analogy
- | algorithm | parent | proposal unit | best when |
- |---|---|---|---|
- | `hill-climb` | current best | whole train set, one-shot each iter | broad gaps; you want the simplest loop |
- | `skillopt` | current best (single lineage) | mini-batch under a shrinking edit budget, epochs + gated slow update | you want disciplined annealing + per-epoch consolidation; a moderately sized train set |
- | `gepa` | sampled from a per-instance **Pareto frontier** | minibatch with a cheap local gate before full val | many specialists the mean hides; merge across lineages |
+ The knob is real: `L` decays, it is an integer, and the schedules are correct.
+ The *justification* is weaker than the ML vocabulary implies, and pretending
+ otherwise would mislead anyone tuning it.
- ## Inputs / outputs (manifest tokens)
- - **needs:** `scores` + `traces` (per-task val results to reflect on) and
- `candidate` (the parent to extend).
- - **provides:** `candidate` (the accepted best).
+ What plausibly holds: the gate accepts or rejects a whole candidate. A candidate
+ bundling six edits where five help and one hurts is rejected entirely, and you
+ learn nothing about which edit was the problem. Fewer edits per candidate means
+ an accepted candidate is more likely to contain only good edits and a rejected
+ one is cheaper to attribute and revert. That argues for small edits — it does not
+ by itself argue for *decay*.
- ## Standalone use
+ What does not transfer: in SGD, LR decay exists because nothing stops a large
+ step from overshooting near an optimum. Here the val significance gate already
+ rejects an overshooting edit before it can become the parent. The overshoot
+ protection is the gate, so the decay is not doing that job.
+ What is unmeasured: no ablation in this repo isolates the schedule's effect.
+ At realistic step counts the choice barely exists — over 12 steps from 4 down
+ to 2, `linear` and `cosine` differ at 2 of 12 positions. Prefer `constant` or
+ `linear` and spend your tuning budget on `--n-trials` and the gate instead. This
+ is a heuristic that has not been isolated; do not present it as a proven one.
+
+ ## Known gaps
+
+ These are shipped-behavior defects in `core/cap_evolve/skillopt.py`. The skill
+ describes what the code does today, not what it intends to do.
+
+ - **The mini-batch never reaches the optimizer** (issue #371). Mini-batch ids come
+ from **train** (`skillopt.py:230`) but filter the parent's **val** rows
+ (`skillopt.py:291` → `harness.py:2011-2012`; also `:315`, `:319`), and splits are
+ disjoint slices (`splits.py:117-119`). Every step renders `0 solid / 0 flaky /
+ 0 failing of 0 tasks` with no task ids, and `## Failure patterns still unsolved`
+ never appears — while the `(mini-batch of N train tasks, L=…)` label still
+ prints, which is why it looked healthy. Until #371 lands the per-step signal is
+ the label plus the `L` sentence, so the epoch/mini-batch structure is
+ bookkeeping rather than focus. PR #370 fixed the same defect in hill-climb's
+ `cyclic`/`hardest-first` modes.
+ - **The epoch-boundary re-evaluation scores the whole train split, not a sample.**
+ `skillopt.py:458-463` calls `evaluate_candidate(..., split="train")` twice with
+ no `ids=`, then discards everything outside the ~20 sampled ids
+ (`:464-466`). Budget it as `2 × len(train) × n_trials` rollouts per boundary.
+ - **When an epoch accepted nothing, the comparison is vacuous.** The re-eval is
+ guarded by `prev_epoch_best_id != run_dir.best_id` (`skillopt.py:455`); if
+ nothing moved, both sides stay `current_val` — the same list — so 0 regressed
+ and 0 improved are reported over **val** tasks while the log line claims a
+ train sample size (`:469-470`). The consolidation step still runs.
+ - **`requested_edits` vs `applied_changes` surfaces nothing.**
+ `_changed_components` (`skillopt.py:398-427`) counts files whose bytes differ,
+ not edits, and `applied_changes` is written at `:338`/`:345` and read nowhere —
+ no dashboard column, no check, no warning. A run at `L=4` logged
+ `applied_changes: 6` with no consequence.
+ - **"skill" leaks into the live prompt.** `skillopt.py:147` tells the optimizer to
+ compare "the skill" regardless of which capability is under optimization. An
+ algorithm must be capability-agnostic; this text is not.
+
+ ## Key flags
+
+ `--epochs`, `--batch-size`, `--accumulation` (mini-batches per step; multiplies
+ the effective batch), `--edit-budget` / `--min-edit-budget` / `--lr-schedule`,
+ `--slow-update-sample`, `--no-slow-update`. `--resume`, `--no-regression`,
+ `--n-trials`, `--workers`, `--gate-mode`, `--k-se`, `--protected-paths`,
+ `--store` behave as in hill-climb.
+
+ `--max-iterations` is accepted and **ignored** — the loop is epoch-driven, so
+ `cap-evolve run`'s iteration cap has no effect here (`scripts/run.py:42-43`, whose
+ help text now says so). Control the step count with `--epochs`/`--batch-size`.
+
```bash
python scripts/run.py --run-dir .capevolve/run_X --project .capevolve/project \
--optimizer 'python .../run-optimizer/scripts/run.py --name mock --workdir {workdir} --prompt {prompt}' \
- --epochs 4 --batch-size 8 --edit-budget 4 --lr-schedule cosine --min-edit-budget 2 \
- --n-trials 4
+ --epochs 4 --batch-size 8 --accumulation 1 \
+ --edit-budget 4 --lr-schedule linear --min-edit-budget 2 --n-trials 4
```
- Requires `baseline.json` first (like its sibling algorithms). `--resume`
- continues from the run's current best. `--no-slow-update` disables the
- epoch-boundary meta step; `--no-regression` adds a SWE-bench-style dual gate.
-
- ## Pitfall: small val sets
-
- The default gate is `significant`/`paired` (NOT naive strict-greater) so a tiny
- val set does not reject every edit on noise. With a small val set, raise
- `--n-trials` (real per-trial variance) or use a **graded** reward so the paired
- significance test has signal. Only the slow update is a "meta" step — it is still
- **gated on val**, never force-accepted, and its train sample is small + counted in
- budget (toggle with `--no-slow-update`).
-
- ## Agent-mode loop
- When `orchestration_mode: agent`, drive skillopt yourself: single-lineage climb with a textual learning rate over epochs/minibatches — each step propose an edit sized to the current LR, evaluate on **val** (minibatch then full as skillopt prescribes) via cap-evolve, gate Δ>k·SE, accept→snapshot / reject→revert & decay. Log steps to the run dir; between steps verify rollouts+results landed for the dashboard. Re-read `stop_condition`; stop on it/stall/budget. Seal once with `cap-evolve finalize`, then `report`.
+ Requires `baseline.json` first, like its sibling algorithms.
## References
- - `references/concepts.md` — the SkillOpt loop in detail, the textual-LR schedule,
- the buffer/slow-update mechanics, and citations (arXiv:2605.23904).
+ - `references/concepts.md` — the loop step by step, the schedule shapes with
+ worked values, and the buffer / consolidation mechanics. Load it when you need
+ to change the loop or reason about its rollout cost, not to run it.