skill-review · git:20261109.eacad26 · 2026-11-09 · sha256 c8bfceac53da5e4c

skill-review git:20261109.eacad26A

Immutable. This exact content never changes and is served at /api/v1/blob/c8bfceac53da5e4c.

No tool to install first
claude plugin marketplace add https://markdownregistry.com/mp/a/art_fafot2cdvzck74sv/c8bfceac53da5e4c08d57c5d845a6425b08775ca6057653462017b8fa46f7bbe.json

Then /plugin install skill-review@markdownregistry-fafot2cdvzck74sv in Claude Code. It installs this exact version, the one graded A here, pinned to the commit it was graded at, and leaves it there.

---
name: skill-review
description: Weekly check that Peter's skills still earn their lines. Turns the scout's open finding issues into one scored change and merges it when the replays of real past tasks say it helps, adds new replay cases from the week's friction, and reports skill read rates and merged rules that still gain events. Use for the scheduled weekly Skill review, or when asked to evaluate or compare versions of the skills. Not for finding new patterns in threads (skill-scout) or for cutting text (skill-gardener).
compatibility: Runs on the machine that holds the private eval cases (~/.local/share/skill-evals/orchestrate). Needs the claude and codex CLIs, gh and node.
---

# Skill review

Goal: every scout change reaches Peter with evidence. One run: add cases, score the filed findings, check adherence, report. This review measures and adds; cutting text is skill-gardener's job, every three days.

- Never commit cases, transcripts or run output to the skills repo. The repo is public; cases come from private work. They live in `~/.local/share/skill-evals/orchestrate/`.
- Budget: at most two eval comparisons per run, about $30 each (32 agent sessions plus judging). Run each in the background with a log (`runq` or a background command), and wait for it to finish before going on. Never end the turn while one runs.

The eval is `evals/orchestrate/` in `~/skills`; read its README first. It replays real past tasks up to the point of the first edit and grades the plan against what Peter later had to say. It doesn't measure execution after planning.

## 1. Add cases (up to 3)

Look at `~/.local/state/skill-scout/ledger.md` for events since the last review; its date is the last entry in `climb.md` in the eval folder. Pick threads where Peter's first message asked for a build and he stepped in at least twice afterwards. Write each one following the README's "Case format" and its rules for picking interventions. Make every third new case `"split": "holdout"`.

## 2. Score the filed findings

List them: `gh issue list -R peterpme/home --label finding --state open --json number,title,body --limit 100`. Skip findings whose body has no proposed text yet. If any remain:

1. Run `git -C ~/skills fetch origin`. Make a worktree at `~/skills/.claude/worktrees/skill-review-<date>` on branch `skill-review/<date>` from `origin/main`, and apply each finding's proposed text as its own commit, generic wording only.
2. Run `REPS=2 node evals/orchestrate/run.mjs origin/main .` from that worktree.
3. If the train mean rises by more than 0.1, run `SPLIT=holdout` once.
4. Verdict per batch: helps, within noise, or hurts (train mean down by more than 0.1, or any case down by more than its spread between reps).
5. **Helps or within noise:** run the leak check from skill-scout section 6, open the PR with aggregate numbers only, and merge it under the personal-repo rule in the global instructions. Then `npm run release` from `~/skills`, close each finding issue with the numbers, and mark the ledger `merged`.
6. **Hurts:** drop findings one at a time until the batch passes, merge what passes, and comment the numbers on each dropped finding, which stays open for the scout to gather more evidence or for Peter to overrule.

Case ids and judge notes name private work, so they go only in the T3 report and the finding issues, never in the skills PR.

## 3. Adherence

1. Run `python3 -I ~/skills/skill-scout/scripts/adherence.py --since 7d` and keep its table for the report. Exit 3 means the T3 database layout changed; report that instead of the table.
2. In the ledger table, take every row whose status starts with `merged` and compare its Events count with the `Merged rules:` line of the last `climb.md` entry. A count that rose means the merged rule is not holding. List each one in the report with old and new counts. It needs an open `finding` issue with the new events; if the scout has not reopened or filed one, do it under skill-scout section 6.
3. With no earlier `Merged rules:` line, record the counts and flag nothing.

## 4. Cuts and sizes

Not here. Skill-gardener cuts duplicated, stale and narrated lines and flags oversized skills every three days. A line that is wrong rather than surplus is a `finding` issue, scored in section 2.

## 5. Report

```
**Skill review, <date>:** <N findings merged as PR #n / M dropped; nothing to do>

| Check | Result |
|---|---|
| Findings PR #n | train <main> → <PR> (<cases>), holdout <…>, verdict, issues closed |
| New cases | N (ids) |
| Merged rules still gaining events | <candidate old → new, finding #n; none> |
| Spend | $N |
```

Then the adherence table, unchanged.

Then append the run to `climb.md`: the date, what was compared, the scores, what was kept, and one line `Merged rules: <candidate> <events>, …` for every `merged` ledger row.

## Set up

Set this up only on the machine with the eval cases, and only after this skill is merged: scheduled runs read `~/skills`, which tracks `main`. Call `schedule_task` with:

- `projectId`: the "No project" (scratch) project;
- `bindToCurrentThread: false`;
- `title: "Skill review"`;
- `schedule: {type: "fixed_time", timeOfDay: "08:00", weekdays: [1]}`;
- this prompt:

  > Run the skill-review skill: the weekly review on this machine. If `~/skills/skill-review/SKILL.md` or the eval cases are missing, reply in one line starting with "Needs your call:" saying which, and stop.

`docs/feedback.md` in the home repo holds Peter's own words about how agents should work, newest first. Read it before anything else; each entry outranks a pattern inferred from threads.