skill-package · diff
git:20260628.c434723 to git:20260629.a2d1871
78 added, 72 removed. Audit A to A.
---
name: skill-package
- description: Optimize an Agent Skill package itself — its SKILL.md (frontmatter + body), references, and bundled scripts. Use when the capability under optimization IS a skill, and you want the agent to use the skill more effectively, trigger it correctly, and follow it without wasted steps. Enforces the skill-creator authoring rules (progressive disclosure, valid frontmatter, body length, one-level references) so edits stay valid skills.
+ description: Optimize an Agent Skill package itself — its SKILL.md (frontmatter + body), references, and bundled scripts. Use when the capability under optimization IS a skill: you want the downstream agent to trigger it correctly and follow it without wasted steps. Enforces the skill-creator authoring rules (progressive disclosure, valid frontmatter, body budget, one-level references) so edits stay valid skills.
component: capability
argument-hint: "--path DIR"
allowed-tools: Read, Write, Edit, Bash
provides: [candidate]
needs: []
- sources: [skillgrad, trace2skill, skillopt]
+ sources: [agentskills, skillgrad, trace2skill, skillopt]
---
# Capability: skill package
The thing being optimized is a skill directory (a `SKILL.md` plus `references/`,
`scripts/`, `assets/`). This capability treats that whole package as the editable
- artifact and bakes in the **skill-creator** authoring rules so the optimizer
- improves the skill without breaking it.
+ artifact and bakes in the **skill-creator** authoring rules (sourced to first-party
+ Anthropic docs — see [`references/concepts.md`](references/concepts.md)) so the
+ optimizer improves the skill without breaking it.
- ## What you can change here
+ ## What you can change (highest leverage first)
Each lever is an edit class; pick the one that fixes the biggest failure cluster.
- (1-line generic examples; depth in [`references/concepts.md`](references/concepts.md).)
+ One-line examples here; depth in the referenced files.
- 1. **Edit the `description` / trigger** — the decision boundary that makes the skill
- fire. Sharpen *when* it applies for under-trigger; tighten the boundary + state
- near-miss cases for over-trigger. **Usually the highest-leverage edit.** *Ex:*
- "Formats data" → "Use when the user asks to export records to CSV."
- 2. **Edit the body** — improve clarity / altitude, remove dead weight, explain the
- *why* instead of rigid MUSTs, fix the step the agent keeps skipping. *Ex:* turn a
- vague "process the file" into a numbered procedure.
- 3. **Add / split references** — factor mutually-exclusive or rarely-co-used content
- into `references/*.md` (one level deep, TOC if long) as the body grows past ~500
- lines. *Ex:* split per-format guidance into `references/csv.md`, `references/json.md`.
- 4. **Add / edit bundled scripts** — when traces show the agent re-implementing the
- same helper, bundle it in `scripts/` and **state execute-vs-read intent**. *Ex:*
- add `scripts/validate.py` the skill runs instead of hand-writing the check.
+ 1. **The `description` / trigger** — the decision boundary that makes the skill
+ fire, and the single highest-leverage edit. Write it **third person**, state
+ **what** it does AND **when** to use it, and use the **keywords a user would
+ actually say**. Lean slightly pushy for under-trigger; tighten the boundary and
+ name near-miss cases for over-trigger. **Front-load the key use case** — the
+ listing truncates `description + when_to_use` at 1,536 chars. *Ex:* "Formats
+ data" → "Exports records to CSV. Use when the user asks to export or download a
+ table." Full playbook: [`references/description-optimization.md`](references/description-optimization.md).
+ 2. **The body** — improve clarity / altitude, remove dead weight, fix the step the
+ agent keeps skipping. The body is loaded on every trigger and **stays in context
+ all session — a recurring token cost**, so keep it **<500 lines AND ~5k tokens**
+ and **state what to do, don't narrate why at length**. Imperative voice; explain
+ the *why* of a rule briefly instead of piling on ALL-CAPS MUSTs.
+ 3. **References** — factor mutually-exclusive or rarely-co-used detail into
+ `references/*.md` as the body grows. Keep them **one level deep**, link each
+ **directly from SKILL.md with an explicit pointer saying what it contains and
+ when to load it**, and give long refs (>300 lines) a table of contents. Don't
+ nest (a ref pointing to another ref) — the agent may only partially read it.
+ 4. **Scripts** — when traces show the agent re-implementing the same helper, or a
+ step is deterministic/repeatable, **bundle it in `scripts/`** and **state
+ execute-vs-read intent**. A script runs via bash without its code entering
+ context (output-only token cost); prose is only *likely* and costs context.
+ Reserve prose for steps that need judgment.
- > **Keep edits valid skills:** valid frontmatter (`name`/`description`), body
- > < ~500 lines (progressive disclosure), references one level deep, no broken links.
- > Explain the WHY over rigid MUSTs, and don't introduce unaudited/exfiltrating
- > content — a skill body is executable context.
+ > **Keep edits valid skills:** valid frontmatter (`name` ≤64/[a-z0-9-]/no XML;
+ > `description` non-empty ≤1024/no XML, with a "use when" clause), body within
+ > budget, references one level deep, no broken links. Don't introduce
+ > unaudited/exfiltrating content — a skill body is executable context.
- ## What can be optimized
- - **The `description`** — the primary triggering signal (what + when). Sharpening
- it is usually the highest-leverage edit.
- - **The body** — clarity, the right altitude, removing dead weight, explaining the
- *why* instead of rigid MUSTs.
- - **References** — adding/splitting `references/*.md` as the body grows past ~500
- lines (progressive disclosure).
- - **Scripts** — factoring repeated work into `scripts/` the skill calls.
+ ## How agents use a skill (progressive disclosure)
+ Three loading levels — optimize for the cheapest that still works:
+ 1. **Metadata** (`name` + `description`) — always in context (~100 tokens). The
+ description is the *only* thing that decides whether the skill fires.
+ 2. **SKILL.md body** — loaded when the skill triggers (recurring session cost).
+ 3. **References / scripts** — loaded or executed only as needed.
- ## How agents use a skill
- Metadata (name+description) is always in context; the body loads when the skill
- triggers; references/scripts load only as needed. So a vague description → the
- skill never triggers; a bloated body → wasted context and worse behavior.
+ So a vague description → the skill never triggers; a bloated body → wasted context
+ and worse behavior; detail that belongs in a reference → paid for on every trigger.
## The description is the trigger — optimize it as a separable step
- The `description` is the decision boundary: it is what makes the skill fire. Most
- triggering failures are fixed by editing the *description*, not the body.
+ Most triggering failures are fixed by editing the *description*, not the body.
- - **Under-trigger** (the skill didn't fire when it should have) → make the
- description more explicit about *when* it applies: enumerate the phrasings and
- contexts that should fire it, including when the user doesn't name the skill.
- - **Over-trigger** (it fired when it shouldn't have) → tighten the boundary and
- state the near-miss cases it does NOT cover.
- - **Over-trigger caveat for newer models.** `CRITICAL`/`ALWAYS`/`MUST` in a
- description now causes over-triggering on current models — prefer plain
- "Use this when …". Reserve pushy phrasing for genuine under-triggering.
- - **Evaluate triggering on held-out prompts.** Build a small set of should-trigger
- and should-NOT-trigger prompts including **near-miss negatives** (prompts that
- look close but shouldn't fire), and select the description that scores best —
- don't overfit to the handful of iteration examples. (Note: trivial one-step
- tasks may not trigger any skill regardless of description.)
+ - **Under-trigger** → enumerate the phrasings and contexts that should fire it,
+ including when the user doesn't name the skill.
+ - **Over-trigger** → tighten the boundary and state the near-miss cases it does NOT
+ cover. `CRITICAL`/`ALWAYS`/`MUST` in a description over-triggers current models —
+ prefer plain "Use when …"; reserve pushy phrasing for genuine under-triggering.
+ - **Trivial single-step tasks** may not trigger any skill regardless of wording.
- ## Bundle a script when traces show repeated re-implementation
- If across runs the agent independently re-writes the same helper, or a step is
- deterministic and repetitive, **bundle it as a script in `scripts/`** and have the
- skill call it — code is consistent and repeatable where prose is only *likely*.
- **State the intent explicitly**: whether the agent should *execute* the script or
- *read it as reference*. (Scripts can run without their code entering context.)
+ ## Measure every edit against the objective
+ A skill edit is only an improvement if it raises the number we are optimizing.
+ - **The acceptance signal is the intake benchmark score** on the held-out **val**
+ split, via cap-evolve `evaluate` → `gate`. Keep only gated wins; reject edits
+ that don't clear the significance bar. **Never overfit the handful of iteration
+ examples** — a skill is used many times; fiddly task-specific rules hurt.
+ - **For triggering**, also track trigger-rate on a held-out set of should-trigger /
+ should-NOT-trigger prompts (with **near-miss negatives**), and pick the
+ description that scores best on the held-out set — the skill-creator loop's own
+ select-by-held-out discipline, which is exactly cap-evolve's train/val/test split.
+
## Handlers (scripts/abstract.py)
`materialize(dir)` → {SKILL.md, references/*} · `apply(dir, edits)` ·
- `validate(dir)` → checks frontmatter (`name` ≤64/[a-z0-9-]; `description` ≤1024
- with a "use when" clause), body ≤500 lines, references one level deep + TOC for
- long ones, and that linked files exist.
+ `validate(dir)` → frontmatter (`name` ≤64/[a-z0-9-]/no XML; `description` ≤1024/no
+ XML with a "use when" clause; POV + all-caps + 1,536-listing lints), body ≤500
+ lines / ~5k tokens, references one level deep + TOC for long ones, links exist.
## Optimizing it each iteration (analyze → ideate → edit)
- The optimizer should **analyze before editing**: from the traces + the current
- skill, identify (a) the recurring failures clustered by root cause (the step the
- agent skips, the wrong trigger, the part of SKILL.md it misreads) and (b) the good
- behavior seen only on some trials that should be made consistent; then make ONE
- targeted edit (description/body/reference) that fixes the biggest cluster and
- reinforces (b), staying within the skill-creator rules. Be economical: one good
- edit, then stop.
+ **Analyze before editing** (treat the skill like an evolving playbook you curate):
+ from the traces + the current skill, identify (a) recurring failures clustered by
+ root cause (the step the agent skips, the wrong trigger, the misread instruction)
+ and (b) good behavior seen only on some trials that should be made consistent. Then
+ make **ONE** targeted edit that fixes the biggest cluster and reinforces (b),
+ staying within the skill-creator rules. Be economical: one good edit, then stop.
## How to run
```
- python scripts/check.py
- python scripts/run.py --path <skill-package_dir> # candidate + validity report
+ python scripts/check.py # self-test (must pass)
+ python scripts/run.py --path <skill_dir> # candidate + validity report
+ python scripts/token_report.py --path <skill_dir> # progressive-disclosure budget
```
- ## Reference
- - `references/concepts.md` — grounded background for this capability.
-
## References
- - `references/concepts.md` — Agent Skills authoring + progressive disclosure (skill-creator).
+ - [`references/concepts.md`](references/concepts.md) — the authoring model and the
+ validity rules, with first-party sources. Load for grounding.
+ - [`references/description-optimization.md`](references/description-optimization.md)
+ — the trigger-tuning playbook. Load when fixing under/over-trigger.
+ - [`references/anti-patterns.md`](references/anti-patterns.md) — common bad smells
+ and the why. Load when a draft "feels off" or to review an edit.