v0.2.0 to v0.3.0

47 added, 5 removed. Audit A to A.

---

name: optimizing-skills

description: Disciplined, validation-gated revision of an EXISTING skill so each edit is a measured improvement rather than a guess. Use when editing, revising, or tuning a skill that already exists and there is evidence it underperforms (observed failures, drift, complaints) — invoke by name, or have versioning-skills / creating-skill defer to it before applying edits. Not for authoring a brand-new skill from scratch (use creating-skill) or one-off prose.

metadata:

- version: 0.2.0

+ version: 0.3.0

---

# Optimizing Skills

Treat the skill document as the parameter under optimization: change it only

when the change **demonstrably beats the version you already ship**. This is

the discipline distilled from SkillOpt (microsoft/SkillOpt, arXiv:2605.23904) —

its training apparatus dropped, its reproducibility discipline kept. The point

is to stop editing skills on intuition and start editing them on evidence.

## Core principle

A skill edit is only worth shipping if it strictly improves measured behavior

on a held-out check. Most edits that *feel* like improvements don't move the

needle, and some quietly regress. The gate below is what separates a real

improvement from a confident guess.

## The gate — run it every revision

1. **Assemble a held-out check set.** 3–8 representative tasks/prompts the skill

should handle well, and it **must include the failure(s) that prompted this

revision**. Keep the set fixed across the revision so before/after scores are

comparable.

2. **Hold two versions.** `best` = what you currently ship (never let it

silently degrade). `candidate` = `best` + your proposed edits.

3. **Score both on the check set.** "Run" here = dispatch each check task to the

**Agent tool** (`subagent_type=general-purpose`) with the skill version in

- context, or evaluate by hand for small sets. Score **per criterion**, not one

+ context, or evaluate by hand for small sets. Give the scoring agent the skill

+ version and the task, and nothing else. Never hand it the ledger, your

+ revision notes, or the diagnosis that motivated the edit: it will solve the

+ task from those instead of from the skill, and the score stops measuring the

+ skill. WikiSkill (arXiv:2608.27454) ablated exactly this and lost 2.8 points

+ of final quality, 7.8 on their hardest split, by letting the worker read the

+ improver's knowledge store. Score **per criterion**, not one

collapsed pass/fail. When a task carries several criteria, the criterion that

decides accept/reject is **the failure that prompted this revision**; the

others are regression guards that must not get worse. Collapsing criteria

masks the win: in the down-skilling-v1.2.0 retro, the edit drove architectural

hallucination 60%→0% while an unrelated length criterion stayed 0/5 in both

arms — a single combined pass/fail scored that as a 0–0 tie and would have

rejected a large, real improvement.

4. **Accept only if `candidate` strictly beats `best`** on the triggering-failure

criterion, with no regression guard worse. Ties → reject, keep `best`. An edit

that doesn't move the needle does not ship.

**When the skill's own output is compiled by an Agent** (down-skilling and

creating-skill produce a prompt an author writes from the SKILL), score **≥2

author samples per version**, or fix one author across both arms. A single

author sample per arm lets author capability dominate the edit effect: the same

down-skilling edit measured 95%→0% with one author pair and 60%→0% with another

— real either way, but n=1 cannot tell a real edit from a lucky author.

This two-tier `best`/`candidate` split is the heart of it: a working revision

can explore, but the shipped skill only ever ratchets upward.

## Bounded edits (the "textual learning rate")

Cap edits per revision — **default ~4** distinct add/replace/delete operations,

fewer as the skill matures. Large speculative rewrites drift and destroy your

ability to attribute a regression to a cause. If a revision wants more edits

than the budget, rank and keep the top ones (below) and let the rest wait.

## Reflect: failures first, then successes

Separate the evidence before proposing edits:

- **Failure reflection.** Across the failing cases, find the *common,

systematic* pattern — not a one-off edge case. Propose edits that fix the

pattern. **Failures take priority** in any merge.

- **Success reflection.** Across cases that already work, find generalizable

patterns worth encoding so they survive future edits. Reinforce; don't

duplicate.

For both: edits must **generalize** (never hardcode task-specific values), and

must **not duplicate** content already in the skill — patch genuine gaps only.

## Rank when over budget

When candidate edits exceed the budget, keep them in this priority order:

1. **Systematic impact** — fixes a recurring failure across many cases, not one.

2. **Complementarity** — fills a real gap rather than restating existing content.

3. **Generality** — phrased as a durable principle, not tied to one task/entity.

4. **Actionability** — concrete, followable guidance over vague advice.

Drop the rest. They can return next revision if still warranted.

## Protect the hard-won core

If a skill has a battle-tested core that routine edits keep eroding, fence it

off and treat it as off-limits to fast edits. Revisit it only on a **deliberate

longitudinal review**: compare the *same* check tasks across several versions to

catch slow drift and regressions that single-edit review misses. (SkillOpt

fences this region with HTML-comment markers and only rewrites it at epoch

boundaries — the same idea, manual cadence.)

+ ## Recording rejected proposals

+ **FIRST action of any revision, before you score anything:**

+ ```bash

+ python3 scripts/skill_ledger.py check --skill <name> --before best.md --after candidate.md

+ ```

+ Exit 1 means this exact edit was proposed before and failed the gate. Read the

+ recorded criterion and score, then propose something else. Do not re-score it.

+ **After the gate decides, record the outcome — accepted or rejected:**

+ ```bash

+ python3 scripts/skill_ledger.py record --skill <name> \

+ --before best.md --after candidate.md --outcome rejected \

+ --criterion "<the triggering failure>" --score-before 0.6 --score-after 0.6

+ ```

+ The ledger computes the diff itself and appends it. A rejected candidate is the

+ artifact worth keeping. The edit reverts and the record stays. The next

+ session cannot see this one, which is why the record has to live outside your

+ context rather than in it.

+ This is WikiSkill's `skill-impact.md` (arXiv:2608.27454), whose outer loop

+ keeps a never-rolled-back record of every proposal and its fate while the skill

+ itself rolls back. Their proposer is told in prose not to repeat rejected

+ approaches. `check` makes that mechanical instead, so it no longer depends on

+ anyone having read the ledger. The script lives in claude-workspace at

+ `scripts/skill_ledger.py`.

## Carry memory across revisions

- After a revision, record what you learned about editing **this** skill — which

- kinds of edits helped, which were brittle, redundant, or harmful — via

- `remember()` tagged with the skill name. Before the next revision, `recall()`

+ The ledger above holds the artifacts: which diffs were tried and how they

+ scored. `remember()` holds the judgment the diffs cannot carry. After a

+ revision, record what you learned about editing **this** skill — which kinds of

+ edits helped, which were brittle, redundant, or harmful — via `remember()`

+ tagged with the skill name. Before the next revision, `recall()`

it. This is the compounding part: each revision starts smarter than the last,

the way SkillOpt's optimizer-side meta-skill conditions its future edits.

## Edit mechanics

Edits are literal string operations (the `Edit` tool): the target text must

match **exactly** or the edit is a silent no-op. Keep targets unique and

verbatim. Prefer append / insert-after-heading / replace-exact / delete-exact,

and verify each edit landed before scoring.

## When NOT to use this

- Authoring a brand-new skill from scratch → **creating-skill**.

- Tracking/rolling back versions during development → **versioning-skills**.

- One-off prose with no reuse → just write it.

## Checklist

- [ ] Held-out check set assembled (includes the triggering failure), fixed for the revision

- [ ] Edits bounded (~4 max), each generalizable and non-duplicative

- [ ] Failure patterns addressed before success reinforcement

- [ ] Candidate scored per-criterion against `best`; accept decided by the triggering-failure criterion, others as regression guards; shipped only if strictly better

- [ ] For Agent-compiled artifacts (down-skilling, creating-skill): ≥2 author samples per version, or a fixed author across arms

- [ ] Hard-won core left untouched unless doing a deliberate longitudinal review

+ - [ ] Candidate run through `skill_ledger.py check` before scoring; a repeat was dropped, not re-scored

+ - [ ] Scoring agent given the skill version and the task only, never the ledger or the diagnosis

+ - [ ] Gate outcome recorded via `skill_ledger.py record`, rejections included

- [ ] Lesson about editing this skill recorded via `remember()`

For the deeper "dispatch reflection/scoring to the Agent tool" recipe and the

adapted reflection/ranking prompt templates, see

[`references/skillopt-provenance.md`](references/skillopt-provenance.md).