# CLAUDE.md This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. ## What this is A host-agnostic agent skill (`skill-evolution`) that gives the host agent a self-improvement feedback loop: it reads the agent's past session transcripts through the active `HostAdapter` (default Hermes → `~/.hermes/state.db`; Claude Code → `~/.claude/projects/*/*.jsonl`), compares them against the agent's installed skills, generates structured markdown proposals (`improve_existing`, `create_new`, `merge_skills`, `deprecate_skill`) for a human to review, and — if opted in — auto-applies high-confidence proposals through the host's mutation channel (Hermes: `skill_manage` instructions; Claude Code: direct file writes), gated on an evaluation framework that scores proposal content before anything mutates. There is no daemon and no server component. The "pipeline" is a set of standalone Python stdlib scripts that a scheduled job invokes and pipes together; the actual analysis step is an LLM call made by the host agent itself (not by any script here), following an analyzer prompt that the operator maintains outside this repo (see SKILL.md's "Scheduled Runs" section for what that prompt needs to do). The evaluation gate (`scripts/evaluate.py`), by contrast, *is* implemented in this repo — it's not delegated to the cron agent. ## Git tracking `.gitignore` used to use unanchored patterns that matched several directory names anywhere in the tree, silently excluding real pipeline code from version control while `git status` read clean. This was fixed by anchoring the patterns. `proposals/`, `reports/`, and `eval_history.jsonl` are gitignored and not shipped by this repo — `proposals/` in particular holds real analysis output derived from actual session transcripts, and free-text personal narrative a model can write into a proposal body isn't reliably caught by shape-based redaction, so these stay local-only rather than committed. This has no effect on runtime behavior: the pipeline still reads/writes these paths exactly as described below; only what `git` tracks is affected. `scripts/*.py` and `tests/*.py` remain fully tracked. `proposal.py`'s `get_proposals_dir()` defaults to `./proposals/` at the repo root (overridable via `SKILL_EVOLUTION_PROPOSALS_DIR`), and `evaluate.py`'s history file defaults to `./eval_history.jsonl` at the repo root (overridable via `SKILL_EVOLUTION_HISTORY_PATH`). The skill-quality cron wrapper writes timestamped markdown reports to `./reports/` (overridable via `SKILL_EVOLUTION_QUALITY_REPORT_DIR`). ## Commands ```bash # Run the pipeline manually, end to end (mirrors what the cron job does) python3 scripts/fetch_sessions.py --dry-run # preview sessions without marking them processed python3 scripts/fetch_sessions.py | python3 scripts/analyze.py # NDJSON sessions -> LLM-ready prompt text python3 scripts/skill_index.py # scan ~/.hermes/skills/ -> JSON index (name conflicts warned on stderr) # Proposal inspection/scaffolding python3 scripts/proposal.py --example # print an example proposal python3 scripts/proposal.py --list # list current proposals python3 scripts/proposal.py --show # show one proposal by id python3 scripts/proposal.py --validate ... # the diagnostic's rules on given files; exit 1 unless all valid python3 scripts/proposal_diagnostics.py # read-only queue diagnostic: state + reasons per pending proposal (--all, --format json) python3 scripts/review_labels.py --next # label pending proposals by hand (TTY); --stats, --export # Evaluation gate python3 scripts/evaluate.py --list-evaluators # show the resolved evaluator set python3 scripts/evaluate.py --retroactive --dry-run # preview batch re-evaluation of saved proposals python3 scripts/evaluate.py --retroactive # re-run evaluation against saved proposals, append history python3 scripts/evaluate.py --prune # prune eval history per SKILL_EVOLUTION_HISTORY_RETENTION python3 scripts/evaluate.py --backfill-transport-failures # report unflagged provider-fault entries (read-only) python3 scripts/evaluate.py --backfill-transport-failures --apply # mark them; keeps a .bak- copy first # One-off evaluation of any of the four targets (advisory by default — only `skill` and # `proposal` gate auto-apply; see SKILL_EVOLUTION_GATE_TARGETS below). # `deterministic` is meaningless for the non-skill targets, so opt out of it. SKILL_EVOLUTION_EVALUATORS=llm_judge,regression python3 scripts/evaluate.py --eval-target proposal --proposal-id SKILL_EVOLUTION_EVALUATORS=llm_judge,regression python3 scripts/evaluate.py --eval-target tool_calls --session-id SKILL_EVOLUTION_EVALUATORS=llm_judge,regression python3 scripts/evaluate.py --eval-target analyzer_prompt --session-id --proposal-id # Inspect processed-session state (read-only; state.py is a library, not a CLI) python3 -c "import sys; sys.path.insert(0,'scripts'); import state; print(len(state.load_processed()), 'sessions already processed')" python3 scripts/fetch_sessions.py --prune-state # prune state per SKILL_EVOLUTION_STATE_RETENTION (reports on stderr) # Optional GEPA optimizer (needs `pip install -e ".[optimizer]"`) SKILL_EVOLUTION_OPTIMIZER_ENABLED=true python3 scripts/optimize_skill.py --list-candidates # read-only report of low-scoring targets (skill: targets only) SKILL_EVOLUTION_OPTIMIZER_ENABLED=true python3 scripts/optimize_skill.py --list-candidates --target all # include proposal:/tool_calls:/analyzer_prompt: namespaces SKILL_EVOLUTION_OPTIMIZER_ENABLED=true python3 scripts/optimize_skill.py --skill [--iterations N] # run GEPA for one skill # Tests pip install -e ".[dev]" pytest ``` **Interpreter note.** The core pipeline is stdlib-only, so any `python3` runs it. The optimizer is not: `gepa` must be importable, so `optimize_skill.py --skill` needs the interpreter you installed the `[optimizer]` extra into. A project-local `.venv` on pyenv 3.12.2 is the setup these commands were verified against — use `.venv/bin/python scripts/optimize_skill.py ...` for optimizer commands, or the bare `python3` shown above for everything else. Running the optimizer under an interpreter without `gepa` fails fast with `_require_gepa()`'s actionable `RuntimeError` rather than misbehaving. `pytest` passes under either interpreter — the gepa-contract module skips when the extra is absent; see "Tests exist now" below for the currently-verified pass/skip counts rather than a number duplicated here (a stale duplicate is exactly how this line and that one drifted out of sync with each other before). No build step and no non-stdlib runtime dependencies for the core pipeline. `gepa` (the standalone PyPI package, not `dspy`) is an optional extra (`pip install -e ".[optimizer]"`) required only by `scripts/optimize_skill.py`; nothing else imports it. ## Architecture ``` fetch_sessions.py ──NDJSON──► analyze.py ──formatted prompt──┐ skill_index.py ──JSON index──────────────────────────────────┼──► [host agent + analyzer prompt] │ ▼ proposal.py (write proposal .md files, status: proposed) │ human review (status: approved) │ apply_proposal() ──► evaluate.evaluate_and_record() │ (deterministic + llm_judge + regression, │ appends to eval_history.jsonl) │ gate passed? ──No──► can_apply: False, proposal stays proposed │ Yes │ ▼ skill_manage (Hermes tool) optimize_skill.py (optional, SKILL_EVOLUTION_OPTIMIZER_ENABLED=true) --list-candidates: reads eval_history.jsonl, prints low-scoring targets (read-only, no proposal written) --skill : fetch_sessions.sessions_for_skill() ──► gepa.optimize_anything() ──► drafts improve_existing proposal from the winning candidate via proposal.save_proposal() (never calls apply_proposal() itself — optimizer-originated proposals go through the same gate as any other) ``` - **`scripts/fetch_sessions.py`** — reads `~/.hermes/state.db` (SQLite; `sessions`/`messages` tables), filters out already-processed sessions (via `state.py`'s state file) and anything older than `--lookback-hours`, keeps at most `resolve_max_sessions()` of what remains (`--max-sessions` > `SKILL_EVOLUTION_MAX_SESSIONS` > **200**, newest first, counted *after* processed sessions are skipped — the query used to `LIMIT 20` before skipping them, so a quiet day returned nothing while older pending sessions sat in the window, and a busy day's overflow was never analyzed; the overflow now stays unmarked for the next run, and `fetch_sessions_for_host()` applies the same cap to every other host, which previously had none), redacts messages via `contains_secret()` — which layers the fixed `SECRET_PATTERNS` substrings (known vendor prefixes, specific env-var names) with shape-based `SECRET_REGEXES`. The regex layer exists because the substring list provably leaked: scanning the real 26,794 message bodies found 27 `password=` assignments, 23 generic `*_TOKEN=`/`*_SECRET=` assignments, 6 generic `sk-` keys, 5 `Authorization: Bearer` headers and a credentialed DB URI that it missed, all of which would have been forwarded to a provider verbatim. Length floors (20+ chars for opaque keys, 8+ for assigned values) keep `MAX_TOKENS=1024` and `DEBUG=true` off the list; measured over-redaction is 0.229% of lines. This predicate is the single chokepoint for **both** redaction points (provider prompts and on-disk proposals), so err toward over-detection — applies `redact_pii()` (see below), truncates long messages, and emits one JSON object per session to stdout. Marks sessions processed as a side effect unless `--dry-run`. Its queries select `sessions.started_at`/`messages.timestamp` (matching the live schema); the per-session output dict's timestamp key is `started_at` (there is no `created_at` key, and `total_tokens` is not emitted — `analyze.py` defaults it). `sessions_for_skill()` (used by the optimizer) is a separate query on the same DB that filters sessions to those touching a given skill, capped at `max_sessions` most-recent-first. **The two functions handle an unreadable DB differently, on purpose:** `sessions_for_skill()` checks existence and the `sessions` table via `_session_db_is_readable()` *before* connecting and returns `[]` with a stderr warning, because the optimizer is interactive and "no history for this skill" is a normal outcome there; `fetch_sessions()` still raises, because it is the unattended cron path where an unreadable DB is a failure the operator needs surfaced rather than a run quietly reporting zero sessions every night. Checking before `sqlite3.connect()` also matters because `connect()` **creates** a file for a missing path — a typo'd `--db-path` previously crashed with a confusing "no such table: sessions" *and* left a stray 0-byte DB behind. - **Secrets and PII are handled differently, on purpose.** A detected **secret** drops the whole message (`_summarize_messages` skips it) — there is no version of a message worth sending once it carries a credential. Detected **PII** is *masked in place* by `redact_pii()`, so an email inside a paragraph of useful debugging evidence costs that email, not the paragraph, and the mask names the kind (`[PII:EMAIL]`) so a reviewer knows what was removed. It is wired at both redaction points: `fetch_sessions._summarize_messages()` (what reaches the analyzer) and `evaluate.redact_secrets()` (provider prompts and on-disk proposals). **Monetary amounts are deliberately NOT masked.** The scope is *identifiers*, not *amounts*: an email, phone, Luhn-valid card, IBAN or RUC/DNI identifies a person, while a bare amount identifies nobody and is exactly the evidence the analyzer needs to reason about finance/budgeting-style skills. Measured over a real message corpus, on what actually crosses the boundary: identifier patterns (email/phone/card/RUC) were fully masked while money amounts passed through untouched, confirming the redaction targets identifiers specifically rather than over-redacting numeric content generally. Card detection requires a leading digit of 3–6 *and* a Luhn check, which is what keeps hash-like 16-digit runs (`commit 1234567890123456`) out. - **`scripts/skill-evolution-fetch.sh`** — the bash wrapper a cron job invokes, so the cron entry stays one stable command. It resolves `fetch_sessions.py` **relative to its own location, following symlinks**, and passes arguments through. Two earlier strategies were broken and are worth not reinventing: probing a hardcoded skills-home path with a fallback that doesn't exist in this repo's layout (always failed), and `if git rev-parse --show-toplevel; then` to find the project root (that command's stdout isn't captured, so the repo path was printed to stdout and prepended a stray non-JSON line to the NDJSON stream `analyze.py` consumes). **Nothing but `fetch_sessions.py` may write to stdout.** Symlink-following matters whenever the deployed entry point is itself a symlink into this repo, which is a common way to install it. - **`scripts/state.py`** — small JSON-file-backed tracker (`~/.hermes/skill_evolution_state.json`) for which session IDs have already been analyzed. `fetch_sessions.py` duplicates this logic inline rather than importing it — if you change one, check the other (`tests/test_state_schema_compat.py` parametrizes over both modules to catch drift). **Two on-disk shapes exist and both must be read.** This repo and the deployed `SKILL.md` document `{"processed_sessions": [...], "last_analyzed_at": ..., "version": 1}`, but the file actually deployed is a flat `{session_id: iso_timestamp}` map (101 entries). Reading only the documented shape returned `[]` every run, which silently disabled dedup entirely — `--dry-run --lookback-hours 168` reported 20 unprocessed sessions that had all in fact been processed. `mark_processed()` now **preserves whichever layout the file already has** (writing the documented shape only for a fresh file) because rewriting the flat map would destroy timestamps another producer maintains. Don't "normalize" that file without confirming who else writes it. **Redirecting state:** `SKILL_EVOLUTION_STATE_FILE` overrides the path, resolved at call time by `get_state_file()` — present in *both* modules, since they duplicate deliberately. Its fallback is the module global rather than the literal path, so monkeypatching `fetch_sessions.STATE_FILE` (what `tests/test_state_schema_compat.py` does) still works; a blank value is ignored rather than redirecting to `""`. Note that once dedup works, `fetch_sessions.py --dry-run` legitimately returns **0 sessions** because every real session is already marked processed, which looks like a failure but is correct. **Host-prefixed since the host-agnostic read side (U2):** `load_processed()`/`mark_processed()`/`prune_processed()` all take a `host` argument (defaulting to `host.resolve_host()`, duplicated locally in both modules rather than imported — `host.py` imports `fetch_sessions.py`, so importing back would cycle). New entries on the flat-map shape are written `":"`; a legacy un-prefixed key is read as belonging to `"hermes"` only, which is what lets the 101 real pre-host entries keep working with zero migration. `prune_processed()` only ever touches the resolved host's own entries — a different host's rows, prefixed or legacy, pass through untouched, and `keep_ids` is matched against each entry's *bare* id rather than its on-disk key for the same reason. This is not yet expressed as a `HostAdapter` method (`scripts/host.py` doesn't have a fourth "processed state" method) — see that file's section above for the three methods it does have. **Pruning: `prune_processed(retention=None, keep_ids=None)`, also duplicated in both modules** (`tests/test_state_pruning.py` parametrizes over both, same drift-protection convention), gated on `SKILL_EVOLUTION_STATE_RETENTION` (same bare-int/`"Nd"`/`"Nmo"` syntax as `evaluate.py`'s retention parsing, duplicated locally rather than imported — `state.py`/`fetch_sessions.py` sit below `evaluate.py` in this repo's import direction and must not invert it; all three parsers warn on stderr and treat an invalid value as unbounded, because raising here — after `mark_processed()` — marked sessions processed and exited before emitting them, and in `evaluate_and_record()` turned every apply into "evaluation gate raised"; the post-mark housekeeping in both fetch paths is also wrapped by `_housekeep_after_marking()` so nothing in it can abort a run). **Real pruning applies only to the flat `{id: timestamp}` shape** — the documented shape carries no per-session timestamp, and its list order isn't chronological either (`fetch_sessions()` builds it from a Python `set` union, whose iteration order is hash-based, not insertion-based), so there is no temporal signal to prune by there without a schema change, which this deliberately does not attempt. Configuring retention against a documented-shape file is a no-op with a stderr warning on every call, not a silent gap. **`keep_ids` is structurally required, not a defensive nicety.** `mark_processed()` computes `now()` once per call and stamps every session in that batch with the identical value, so "keep the N most recent by timestamp" has no defined tiebreak among everything written in the same run — without a floor, a small retention count could drop most of the very batch a run just wrote, making those sessions look unprocessed again on the very next run. `fetch_sessions()` (the only real orchestrator of `prune_processed()` for the Hermes path) computes `just_processed = {s["session_id"] for s in results}` (distinct from `new_processed`, the full accumulated union) and calls `prune_processed(keep_ids=just_processed)` immediately after `mark_processed(new_processed)`, structurally after the `--dry-run` early return so a dry run never prunes either. **`keep_ids` closes the same-run hole, not the cross-run one.** If `SKILL_EVOLUTION_STATE_RETENTION` is configured tighter than the effective `--lookback-hours`, a session's state entry can be pruned while it's still inside a later run's query window — it no longer looks processed, gets re-fetched, re-marked, and re-analyzed, indefinitely, every time this recurs. There's no clean structural fix without coupling retention to lookback, so the mitigation is a warning, not a fix: `_warn_if_state_retention_below_lookback()` fires on stderr every run where age-based retention converts to fewer hours than `--lookback-hours` — count-based retention has no clean unit match to compare, and has its own related limitation (`keep_ids` overriding the limit means a busy day's batch can floor the file above the configured count, so age-based retention is the better fit for this file specifically). Manual escape hatch: `fetch_sessions.py --prune-state`, structurally like `evaluate.py --prune` (before/after count, early return) but reporting to **stderr, not stdout** — unlike `evaluate.py`, this script's stdout is the NDJSON channel other tooling parses. - **`scripts/skill_index.py`** — scans `~/.hermes/skills///SKILL.md`, hand-parses the YAML frontmatter (no PyYAML dependency — stdlib only) via `parse_name_description_frontmatter()` for `name`/`description`, and emits a JSON index the analyzer prompt uses as "what skills currently exist." `scan_skills(skills_dir, extra_roots=None)` also accepts **extra roots** (`~`-expanded, flat or category-nested per-directory), deduped by skill name with the primary root winning, so indexing `skills.external_dirs` entries (e.g. `~/.agents/skills`) only adds skills the Hermes tree doesn't already cover; the CLI exposes this as repeatable `--extra-root`. **Every root, the primary included, is walked per entry**: `//SKILL.md` is a flat skill (category = the root's basename) and each entry's children are still scanned as nested skills, because one directory can be both (`github/SKILL.md` alongside `github/github-issue-to-pr/SKILL.md`). The primary root used to be nested-only, which silently dropped 82 of the 130 flat skills in the live tree from the index. A flat entry whose `SKILL.md` resolves (via `realpath`) to a file a category already indexes is dropped in favor of the nested record — 46 live flat entries are symlinks to the same files as `general-skills/` — while two *distinct* files sharing a name both stay, so `read_skill_body()` reports ambiguity instead of silently picking one as the baseline (live case: `github` vs `software-development/github`). The CLI warns about such conflicts on stderr via `find_duplicate_names()`; `scan_skills()` itself stays silent because it runs on every skill-body lookup (~176 times per `skill_quality.py` pass). `HermesAdapter.iter_skills()` calls `scan_skills()` with no arguments when there are no extra roots, to keep zero-arg monkeypatches working. `evaluate.py`'s `DeterministicEvaluator` reuses this same parser to validate a proposal's frontmatter. `scan_skills()` **skips dot-prefixed category directories**: a live tree can hold `.archive/` (retired skills), `.curator_backups/` (timestamped snapshots) and `.hub/` (lockfiles, audit log, quarantine). Without that filter the index reported retired skills as currently existing, and an archived twin of a live skill (the same skill name present in both `.archive/` and a live category) made `optimize_skill.py --skill` reject the live skill as an ambiguous duplicate. - **`scripts/analyze.py`** — pure formatter: NDJSON sessions on stdin → markdown-ish text on stdout for LLM consumption. No analysis logic lives here; the actual judgment is made by whatever LLM/agent reads this text plus the analyzer prompt. Its only reusable function is `format_session(session) -> str` (one session at a time) — there is no batch formatter, so an in-process caller joins the per-session strings itself; `main()` is the stdin/stdout wrapper. Formatted session text plus the analyzer instructions plus the skill index can add up to a sizeable prompt, so provider context limits are a real design constraint on how many sessions one analysis pass can carry — `--lookback-hours`/`--max-sessions` (`SKILL_EVOLUTION_MAX_SESSIONS`, default 200 — a lot of context; lower it for a small-context analyzer) exist to bound that. - **`scripts/proposal.py`** — the schema and I/O layer: - `SkillEvolutionProposal` dataclass + `ProposalType`/`ProposalStatus` enums define the proposal shape. - `render()`/`load_proposal()` do round-trip markdown+YAML-frontmatter serialization using a **hand-rolled frontmatter parser** (`_parse_frontmatter`, no PyYAML) — it's whitespace-sensitive (2-space list items, 4-space nested change blocks). Be careful preserving exact indentation if you edit proposal rendering. - `get_proposals_dir()` defaults to `./proposals/` at the repo root (overridable via `SKILL_EVOLUTION_PROPOSALS_DIR`) — see "Git tracking" above. - `list_proposals()` skips files it cannot parse but **names each one and its reason on stderr**. It used to swallow `ValueError`/`IOError` entirely, and that fired for real: a model wrote `status: already_covered` (not in `ProposalStatus`), `load_proposal()` raised, and `--list`/`--retroactive` reported 7 of 8 files with no hint the eighth existed. Any model-authored field can do this, since the analyzer writes these files. The parse stays strict — the fix was making the loss audible, not accepting unknown values — because a proposal is a unit of work awaiting a *human decision*, and a file that errors gets looked at while a file that vanishes does not. - `save_proposal()` pipes the rendered markdown through `evaluate.redact_secrets()` before writing to disk, so a proposal can never persist a secret that leaked into session evidence or a proposed change. - **A proposal's file name is not necessarily its id.** `load_proposal()` records the file it read in `source_path` (never rendered), and `save_proposal()` writes back to that file when it lives in the target directory, falling back to `.md` only for new proposals. Always writing `.md` was a real hazard: with `20260823-001.md` and `20260823-001-new.md` both declaring one id, applying the copy wrote `status: applied` over its namesake and left the copy `proposed` — applicable a second time. `find_proposal_file()` (what `--show` uses) resolves an id the same way `proposal_diagnostics.py` counts claims — every readable file declaring it, plus `.md` when unreadable — and raises naming every file when several claim it rather than picking one. `load_proposal()`'s `ValueError` names the frontmatter problem (no opening `---`, or none closing it); a closing `---` on the last line without a trailing newline is accepted, since a frontmatter-only proposal ends exactly that way. - **Hand-written proposals use literal blocks for multi-line values.** The nightly job loads `SKILL.md` as its instructions and writes proposal files by hand; `SKILL.md`'s example used to carry only `field` + `description`, and 11 of 48 real files copied that shape with no `new_value` at all (12 of 39 pending proposals unappliable). The single-line form could not hold a multi-line anchor anyway. `_parse_frontmatter()` now reads `old_value: |-` / `new_value: |-` blocks (lines indented 6 spaces, `BLOCK_INDENT`; `|` keeps one trailing newline, `|-` none; block content is never b64-decoded), `render()` still writes the canonical quoted/`b64:` form, and `SKILL.md`'s "Writing a proposal file" section requires real values plus `proposal.py --validate ` before the job finishes. `--validate` runs `proposal_diagnostics.validate_paths()` — the same rules as the queue diagnostic, each file against its own directory — and exits 1 for any state but `valid`; possible-duplicate flags print as warnings only. `tests/test_proposal_block_scalars.py` loads `SKILL.md`'s example and fails if it stops being a complete proposal. - `apply_proposal()` validates that the status is pending — `proposed` or `approved` (`PENDING_STATUSES`; the documented review flow sets `approved` by hand, and accepting only `proposed` made that flow raise on every host) — and `confidence >= min_confidence`, resolves the active host adapter, and refuses (before the evaluation gate runs at all) if that host can't write skills or if the proposal can't be expressed as a mutation — a `create_new` with a missing/placeholder `body` is a drafting failure, not a valid mutation (a `create_new` proposal has, in practice, shipped a literal placeholder string instead of real content, which Hermes's `skill_manage` rejects for `create`). Then it runs the evaluation gate (`evaluate.evaluate_and_record()`) *before* mutating anything. If evaluation raises or the gate fails, it returns `can_apply: False` (with `evaluation_results` and, on a raised exception, `evaluation_error`) and leaves the proposal's status untouched. Only once the gate passes does it build a normalized mutation plan and hand it to `adapter.apply_skill_write(plan)`: the Hermes adapter returns `skill_manage` instruction dicts (the `create` instruction carries `name` + `body`, fixing the placeholder-body bug above) that the agent runs later, while the Claude Code adapter writes skill files directly and immediately. `apply_proposal()` flips `status` to `applied` only after the adapter reports `can_apply: True`, and the create_new history migration runs only then too — a refused write must never leave an orphaned migration. It does not call `skill_manage` itself — this script has no Hermes runtime dependency. - `proposal.py` imports `evaluate` at module scope; `evaluate.py` avoids importing `proposal` at module scope (it imports it locally inside `_select_retroactive_proposals`) specifically to break that cycle — keep that direction if you touch either file's imports. - **`scripts/proposal_diagnostics.py`** — read-only diagnostic of the proposal queue, so a reviewer (or a later prioritization step) knows which pending proposals can be reviewed at all. `diagnose_queue()` gives each proposal one primary state by precedence — `unreadable` > `duplicate_id` > `incomplete` > `ambiguous` > `archived` > `unresolved` > `stale` > `valid` — plus **every** reason that applies, because precedence alone hid that an incomplete proposal also targets an archived skill (completing it would not make it applicable); possible-duplicate flags come on top from four text rules (identical `body`/`description` change to the same target — a `create_new` keyed by the name it creates, since two new skills sharing only a `category` are not duplicates — shared `old_value` anchor with a pending or applied proposal, `new_value` already in the installed body, `create_new` reusing a live name). Every check reuses what the gate and apply path use (`_extract_evaluated_content`, `_splice_candidate`, `_build_write_plan`) and reads skills through the host adapter — one scan per run, each installed body read once — including `iter_archived_skills()` — "unresolved" is never equated with "nonexistent". It walks `*.md` itself rather than calling `list_proposals()` so unreadable files are reported, never calls a provider, and never writes; the persisted `status` stays the reviewer's decision. `duplicate_id` (several files claiming one proposal id, same rule as `proposal.find_proposal_file()`) ranks right after `unreadable` because nothing keyed on the id can tell such files apart. Locally (2026-10-17): 39 proposals, by reason 11 archived, 3 unresolved, 4 stale anchors, 12 incomplete (10 empty `new_value`, 2 script-path fields), 2 duplicate_id (the `20260823-001` pair), 0 unreadable. - **`scripts/review_labels.py`** — human labels for the proposal queue: the reference a decision-model pilot is measured against, since the gate's own verdicts cannot be (a pilot judged against them only measures agreement with the gate). `--next [N]` walks unlabeled pending proposals, `valid` first, showing the diagnostic, each change and the cited sessions, and asks for one of `LABELS` (`useful`, `harmful`, `duplicate`, `incomplete`, `obsolete`, `unsure`) plus a reason — optional only for `incomplete`; it needs a TTY. Rows go to `labels/review_labels.jsonl` (`SKILL_EVOLUTION_LABELS_PATH`, cwd-relative default like the history; gitignored and guarded by `tests/conftest.py`), latest per proposal wins, and never into the proposal itself — `status` stays the apply decision. Each row records the proposal file's digest (an edited proposal reads as outdated and is offered again) and a fixed `split` (`holdout` for ~30% of ids by hash, else `design`) so examples used to write a pilot's instructions are never the ones that check them. `--stats`, `--export`. - **`scripts/evaluate.py`** — the evaluation/gating framework (see "Evaluation framework" below). - **`scripts/optimize_skill.py`** — optional GEPA optimizer, gated behind `SKILL_EVOLUTION_OPTIMIZER_ENABLED=true` and the optional standalone `gepa` PyPI package (not `dspy`). `--list-candidates` prints `find_low_scoring_targets()`'s entries (history below `DEFAULT_LOW_SCORE_THRESHOLD` 0.6) read-only, with no optimization run. **A `skill:` target is judged only by entries that scored the installed text** (`evaluate.entry_subject() == "installed"`, i.e. `skill_quality.py`), never by the gate's entries for a proposal's candidate under the same key: on 2026-10-18 all 15 listed targets came from candidates — 9 proposals failing structure checks, 5 oversized proposed bodies — so the list measured proposals, not skills. A low installed score is also dropped when the skill no longer resolves (archived, ambiguous, gone) or its installed text no longer matches the entry's `content_sha256`; entries without a digest are kept. `--list-candidates` reports on stderr how many targets each rule dropped (`skipped=` dict); on the real history the list was empty with 18 targets scored only as candidates, because no valid installed-skill measurement existed yet. **An entry is judged by the LLM judge's own score** (`judge.score`) against the judge's pass threshold (`SKILL_EVOLUTION_LLM_JUDGE_THRESHOLD`, default 0.7) when it records one; older entries keep the combined score and `DEFAULT_LOW_SCORE_THRESHOLD` (0.6); an explicit `threshold` applies to both. The first full quality pass with a working judge (2026-10-18: 175 skills, 0 provider faults, 158 passed) showed why: the 8 skills the judge failed (0.57–0.68) averaged 0.78–0.84 combined once deterministic and regression were folded in, so none were listed, while the list held exactly 6 skills that failed only cumulative drift. `--skill [--iterations N]` runs `run_gepa_optimization()`: resolves the skill's installed `SKILL.md` via `skill_index.scan_skills()` as the seed candidate, fetches that skill's session history via `fetch_sessions.sessions_for_skill()` (capped at `DEFAULT_MAX_SESSIONS_FOR_SKILL`, most recent first; exits cleanly below `MIN_SESSIONS` without importing `gepa` at all), then calls `gepa.optimize_anything()` in Single-Task Search mode with an objective built by `_build_objective(seed_body, skill_name)` — which states the seed body's byte count and the allowed range, plus an instruction to preserve existing section headings. The range is the **intersection of all four limits the gate applies**, computed and intersected *before* any branching: the per-pass percentages (`SKILL_EVOLUTION_MAX_GROWTH_PCT`/`MAX_SHRINK_PCT`) against the body being replaced, the absolute per-pass deletion floor (`MAX_SHRINK_BYTES`), the absolute cap applied as the same ratchet the gate applies (`MAX_SKILL_SIZE_KB` — an oversized seed gets `upper = base`, i.e. it may not grow at all), and the cumulative percentages against `original_size_for_target()`. Each omission was the same bug in a new place: stating only the per-pass window over-advertised headroom for a skill partway toward its cumulative ceiling, and ignoring the cap advertised a ~124KB upper bound for the 103,656B skill that the gate rejects outright. An earlier version returned early from inside the cumulative block, so a window emptied by a *different* constraint went undetected — hence one intersection, then one branch. Notes explaining *why* the window is narrow are emitted in precedence order (ratchet → cumulative → byte floor); the byte-floor note is not cosmetic, since without it an oversized skill is handed a 2KB-wide window on a 100KB body with no stated reason. If the intersection is empty the objective says so, and **which advice it gives depends on which side is violated**: over the growth ceiling → reword within the current length (`base` is still admissible on the floor side); below the shrink floor → no revision of any length passes, so it says the target needs human attention rather than advising a same-length rewrite that would also be rejected. That second branch fixes advice that was silently wrong before. Without that budget the objective named only the task, so the reflection LM optimized purely for score and only met the size limits after the run: one real run spent its entire metric-call budget converging on a +121.8% candidate that was inadmissible from the first byte. It is a **soft** defence (a prompt the model may ignore) that does not replace the deterministic check — its value is not wasting budget on candidates born dead, scored by `score_candidate()` (built on `evaluate.py`'s provider/redaction layer, reusing `LLMJudgeEvaluator.parse_and_score()`) with `gepa`'s own reflective-mutation step also routed through that same provider layer via `_reflection_lm_adapter()` rather than `gepa`'s litellm/OpenAI default. `draft_proposal_from_gepa_result()` builds an `improve_existing` proposal from the winning candidate (rejecting structurally broken candidates before drafting), with a rationale reporting the full explored-candidate frontier. It never calls `apply_proposal()` — optimizer-drafted proposals re-enter the same human-review/auto-apply gate as any analyst-drafted one. The score-provider's judge `feedback` (which `gepa` embeds verbatim into its own internal reflection prompt) is produced under an explicit instruction not to quote session content verbatim and is wrapped in the same untrusted-content framing used elsewhere — narrowing, but not eliminating, a paraphrased-injection residual risk (`gepa`'s own internal prompt construction is out of reach). The three tuning constants `MIN_SESSIONS` (3), `DEFAULT_MAX_METRIC_CALLS` (**8**), and `DEFAULT_MAX_SESSIONS_FOR_SKILL` (20) are overridable via `SKILL_EVOLUTION_OPTIMIZER_MIN_SESSIONS`, `SKILL_EVOLUTION_OPTIMIZER_MAX_METRIC_CALLS`, and `SKILL_EVOLUTION_OPTIMIZER_MAX_SESSIONS_FOR_SKILL`; an explicit `--iterations` always overrides `DEFAULT_MAX_METRIC_CALLS` regardless. **`optimize_skill.py` has been run against the real `gepa==0.1.4` package and real session history**, which is what these defaults are tuned from: a real optimizer run against a skill with an explicit `--iterations 4` surfaced both a real score gain and a pathological +121.8%-growth failure inside those 4 calls, and took over two minutes wall-clock against a hosted provider (30s+/metric-call) — at the old default of 20 that's a 10+ minute unattended foreground call for what is, by design, a manual/occasional invocation (never cron-triggered). 8 keeps 2x headroom over what was empirically sufficient while bounding the worst case instead of doubling it on no further evidence. `MIN_SESSIONS` and `DEFAULT_MAX_SESSIONS_FOR_SKILL` were left unchanged: real runs never tested `MIN_SESSIONS`'s boundary, and both 8- and 20-session trainsets ran fine, so neither constant has evidence pointing anywhere in particular. - **`scripts/skill_quality.py`** — periodic skill quality tracking (P2-3, shipped 2026-08-01). Evaluates all installed skills against the same rubric as the evaluation gate (correctness, procedure-following, conciseness), records each evaluation in `eval_history.jsonl` under `skill:`, and generates a markdown or JSON report showing trends over time. Reuses `evaluate_skill_text()` with a synthetic proposal (field=body, same old/new value so size guards pass), `installed_skill_body()` for skill content, and the existing provider infrastructure. CLI: `--skill ` (evaluate one), `--output ` (write report), `--since Nd` (filter by recency), `--below 0.7` (filter by threshold), `--format json|markdown`. Cost: one LLM judge call per skill (~$0.01-0.05), so a full tree of 143 skills costs ~$1.50-7.00 per run — recommend weekly/monthly, not daily. Cron wrapper: `scripts/skill-quality-report.sh`. - **The analyzer prompt** (maintained by the operator outside this repo, not shipped here — see SKILL.md) is the "business logic" of the analysis half of the system: it defines the analysis criteria (skill coverage, cross-session patterns, specificity, confidence scoring 0.5–1.0), the output format matching `proposal.py`'s schema, and rules like skipping already-processed sessions and never calling the host's skill-mutation tool itself. The Python scripts in this repo are plumbing around that prompt, not a replacement for it. A structural gotcha worth knowing if you're writing your own: `create_new` proposals have no `target_skill`, so `target_key_for_proposal()` falls back to `proposal:`, giving each its own single-entry lineage unless the target key is derived from `proposed_changes` instead (which is what this repo's `RegressionEvaluator` does — see below) — otherwise regression tracking is effectively `improve_existing`-only. The evaluation half (`evaluate.py`) is separate: it scores whatever content a proposal already contains, regardless of which agent or prompt produced it. ### Evaluation framework (`scripts/evaluate.py`) **Local LM Studio provider:** `lmstudio` is a separate registered adapter using `SKILL_EVOLUTION_LMSTUDIO_BASE_URL` (default `http://127.0.0.1:1234/v1`) and the required `SKILL_EVOLUTION_LMSTUDIO_MODEL`. It reuses the OpenAI-compatible HTTP, redaction, retry and rubric-parsing layer without reading Ollama variables or requiring hosted credentials. This machine uses `opencode,lmstudio` with Phi-3.5 Mini Instruct; the host agent's analyzer model and embedding backend are unchanged. Introduced to gate auto-apply on measured quality rather than just the proposal's self-reported `confidence`. - **`EvalResult`** (`score`, `feedback`, `passed`, `evaluator_name`) is the shape every evaluator returns. **`Evaluator`** is the ABC; concrete evaluators register themselves into the module-level `REGISTRY` dict via `register_evaluator()` at import time — there's no dynamic discovery/plugin scanning. - **Which evaluators run** is controlled by `SKILL_EVOLUTION_EVALUATORS` (comma-separated; default `"deterministic,llm_judge,regression"`, resolved by `get_enabled_evaluators()`). An unknown name raises `ValueError` rather than silently skipping. - **Five evaluators**, all registered at import: - `DeterministicEvaluator` — binary size (`SKILL_EVOLUTION_MAX_SKILL_SIZE_KB`, default 15KB), growth-vs-baseline (`SKILL_EVOLUTION_MAX_GROWTH_PCT`, default 20%), **shrink-vs-baseline** (`SKILL_EVOLUTION_MAX_SHRINK_PCT`, default **15%** — deliberately tighter than the growth cap: an over-long skill is bounded by the absolute 15KB check and merely costs context, whereas deletion silently removes guidance, and 15% keeps one pass's bite to ~1.4KB of a median 9.7KB skill instead of ~1.9KB), and YAML-frontmatter-structure checks. The shrink floor is not symmetry for its own sake: the LLM judge's `conciseness` criterion rewards deletion, and a real GEPA run cut a skill body 10258B → 3041B (−70.4%), lost 17 of 29 headings including its `Red Flags`/`Common Rationalizations`/`Anti-Patterns` sections, and scored **higher** for it (0.85 → 0.90). A cap without a floor guards against bloat but not against content destruction. Both per-pass deltas are gated on `context["baseline_size"]` — see `evaluate_skill_text()` below. **Plus two cumulative limits** measured against `context["original_size"]` — where the target *started*, not the body it immediately replaces: `SKILL_EVOLUTION_MAX_CUMULATIVE_GROWTH_PCT` (default 50%) and `SKILL_EVOLUTION_MAX_CUMULATIVE_SHRINK_PCT` (default 30%). The per-pass limits reset their reference every pass, so they compound: at 20% per pass, four accepted passes halve a skill while each one looks compliant, and `RegressionEvaluator` can't catch it because every deletion *raises* the judge's `conciseness` score, so each pass legitimately outscores the last. The erosion is self-reinforcing. Verified: simulating repeated exact-floor cuts, the gate halts at pass 3 with 62% of the original retained instead of running to zero. Note the **cumulative** limit governs the final retained fraction, not the per-pass one — retuning the per-pass floor changes how many passes it takes to get there and how big each bite is, not where erosion stops. Cumulative allowances are deliberately wider than per-pass ones — one pass may move 20%, but total drift stays bounded. **The absolute cap is a ratchet, not a flat ceiling.** It fails only when the candidate is over the cap **and** larger than the baseline it replaces. A real installed-skill census found roughly 15% of skills already exceeding the 15KB cap (median size well under it, but with real outliers over 100KB) — a spot-check claiming far fewer was wrong; a full census is what should be trusted. Under a flat ceiling every body proposal for an over-cap skill failed on size *regardless of direction*, so a proposal shrinking one toward compliance was rejected with the identical message as one growing it: the gate could not tell improvement from worsening. `create_new` carries no baseline, so it keeps the strict cap and a skill is never born oversized — that exemption is a consequence of the rule ("a change that replaces existing text may not worsen; a change that creates text faces the hard cap"), not a proposal-type check to add later. Note this makes oversized skills *improvable*, not *downsizable*: at 2048B/pass, walking a 100KB+ skill to compliance would take dozens of approved passes, so a very large skill still needs a human to split it. **Plus an absolute per-pass deletion floor**, `SKILL_EVOLUTION_MAX_SHRINK_BYTES` (default **2048**, `0` disables), applied as a second sequential check after the percentage floor — the *stricter* of the two binds. A percentage scales with the skill, so it is weakest exactly where a deletion does most damage: 15% of the median skill is ~1.5KB, but 15% of the 103,656B one is 15,548B in a single pass. The default comes from the crossover, not from taste: `2048/0.15 = 13,653B`, just under the 15,360B cap, so the byte floor is inert for essentially every within-cap skill *by construction* and operative precisely on the oversized tail the ratchet unblocks (it binds 35 of 143). The two checks are sequential rather than one `min()` so the feedback names whichever limit actually bound. **These two changes are causally coupled and must not be separated:** the 15KB single-pass deletion was previously *masked* (the shrunk candidate was still over the cap and failed there first), so ratcheting the cap without the byte floor is a net regression in deletion safety. Growth needs no byte counterpart: per-pass absolute growth is bounded at `cap × g/(1+g)` = 2,560B for a within-cap skill, and at 0 for an oversized one under the ratchet. **That invariant scales with the cap** — raising `SKILL_EVOLUTION_MAX_SKILL_SIZE_KB` rescales it, so revisit if the cap moves. The frontmatter check only runs when `context["content_kind"] == "body"` — a bare description change or a merge/deprecate summary+rationale fallback is plain prose and is exempt. - `LLMJudgeEvaluator` — rubric-based (`correctness`, `procedure_following`, `conciseness`, each 0–1, threshold `SKILL_EVOLUTION_LLM_JUDGE_THRESHOLD` default 0.7). Frames evaluated content as untrusted data between a **per-call random hex boundary** (not a static tag) specifically so injected content can't forge the closing marker and escape the block; fails closed (`self._fail(...)`) on any provider error or malformed/out-of-range JSON response rather than guessing. The answer is located by `_find_rubric_object()`: the whole response if it is an object, else the *last* embedded object carrying every rubric key (raw_decode from each `{`) — a greedy `\{.*\}` used to span a reasoning model's `{frontmatter}` aside and its answer and fail as malformed JSON. - **`_post_json()` sets a descriptive `User-Agent`** (`USER_AGENT = "skill-evolution/0.1"`), overridable per call via the `headers` argument. This is load-bearing, not cosmetic: urllib's default `Python-urllib/3.12` is blanket-blocked by OpenCode Zen's gateway, which answers **403 for it and 200 for the byte-identical request** under any descriptive UA. If a provider starts returning 403 while `curl` succeeds, check the UA before suspecting the key. **Verified harmless against a real Ollama server** (`ollama==0.30.10`): its endpoint doesn't gate on User-Agent at all, so the fix changes nothing there — confirmed via a full round trip (`LLMJudgeEvaluator.evaluate()` → `_call_ollama()` → `_post_json()`) scoring a real installed skill. That pass also surfaced a real, previously unknown friction point unrelated to the UA itself: a reasoning-capable local model (`gemma4`) used 54–56s of the 60s default `call_provider` timeout on one judge call, and a heavier one (`qwen3.5`) timed out outright — local models burn a large share of their output on chain-of-thought before the JSON, in a way a hosted API typically doesn't. That friction is now tunable via `SKILL_EVOLUTION_PROVIDER_TIMEOUT` (see the provider adapter layer below). Still unverified against Claude (no `ANTHROPIC_API_KEY` in this environment). - **OpenCode Zen gotcha**: two catalogues exist and they are *not* interchangeable. Zen (`https://opencode.ai/zen/v1`, the `_call_opencode` default) carries `big-pickle` and the `*-free` variants (`deepseek-v4-flash-free`, `mimo-v2.5-free`, …); the Go subscription tier (`https://opencode.ai/zen/go/v1`) is a different, smaller list **without** `big-pickle` and whose DeepSeek id has no `-free` suffix. Point `SKILL_EVOLUTION_OPENCODE_BASE_URL` at the tier matching the model id you set. Both are OpenAI-compatible (`/chat/completions`, bearer auth), so the caller mirrors the Ollama one; unlike Ollama it reaches a third party, so a missing `OPENCODE_API_KEY` fails closed with an actionable message instead of sending an unauthenticated request. Model lists are public — `curl -s https://opencode.ai/zen/v1/models` needs no key. **Verified 2026-10-18, two more constraints:** Zen's free models (`big-pickle`, the `*-free` ids) answer **403 `FreeTierError`** to any client but OpenCode itself, so they cannot judge from this repo even with a valid key; and Go answers **400 `MissingSessionID`** without an `x-opencode-session` header, which `_call_opencode_go()` now sends (one id per process, `_OPENCODE_SESSION_ID`). Go's default model is `deepseek-v4.1-flash` — `deepseek-v4-flash` left its catalogue. Go's docs allow third-party clients that identify themselves with their own user agent, which `_post_json()` does. - `RegressionEvaluator` — compares a target's new aggregate score against the *last entry that actually passed* in that target's history (not just the last entry) so a failed attempt can never lower the bar a later one is compared against. No prior passing entry → auto-passes ("no baseline yet"). Works for both `improve_existing` and `create_new` proposals: `target_key_for_proposal()` extracts the skill name from `proposed_changes` for `create_new`, and `migrate_proposal_history()` rewrites `proposal:` entries to `skill:` after creation, connecting the skill's full evaluation lineage. - `HumanReviewEvaluator` — opt-in interactive gate (P2-4). Not in `DEFAULT_EVALUATORS`; enable via `SKILL_EVOLUTION_EVALUATORS=...,human_review`. Requires a TTY (stdin/stdout `isatty()`); fails closed without one. Binary prompt: y/yes → approve, n/no → reject, empty/junk re-prompts up to 3 times then fail-closed; EOF/KeyboardInterrupt fail-closed. Runs **last** in `run_evaluators()` after automatic + regression, receives `prior_results` context. Approve/reject return `new_score` (the automatic aggregate), mirroring `RegressionEvaluator` — a hardcoded 1.0 would inflate `eval_history.jsonl` gate entries that `RegressionEvaluator` baselines against. Fail-closed returns 0.0. Optional rejection reason captured (`"no reason given"` if blank). Feedback: `"human approved: no note"` / `"human rejected: "`. `scripts/skill_quality.py` strips `human_review` from the env var at startup (warns on stderr) to prevent per-skill interactive prompts in the cron wrapper. - `EmbeddingSimilarityEvaluator` — opt-in semantic similarity checks (P2-5). Not in `DEFAULT_EVALUATORS`; enable via `SKILL_EVOLUTION_EVALUATORS=...,embedding_similarity`. Requires the `embeddings` extra (`pip install -e ".[embeddings]"`), which installs `fastembed` (~50MB, ONNX-based). Three modes selected automatically by context: **duplicate_detection** (flags content too similar to existing skills, similarity > threshold), **drift_detection** (flags content that drifted too far from baseline, similarity < threshold), **grounding_check** (flags a proposal's new text that the sessions it cites do not support, similarity < threshold). Thresholds configurable via `SKILL_EVOLUTION_EMBEDDING_DUPLICATE_THRESHOLD` (0.90), `SKILL_EVOLUTION_EMBEDDING_DRIFT_THRESHOLD` (0.90), `SKILL_EVOLUTION_EMBEDDING_GROUNDING_THRESHOLD` (0.74, calibrated — see below). Backend architecture is extensible via `SKILL_EVOLUTION_EMBEDDING_BACKEND` (default `fastembed`); other backends (`ollama`, `openai`, `llama_cpp`) are stubbed with implementation examples in their docstrings. **Registration, loading and abstention.** `evaluate.py` imports `embedding_similarity` at the bottom of the module (cycle-safe: the three names it imports back are defined by then; guarded, so a broken optional evaluator warns on stderr instead of breaking every import of the gate), so the evaluator is available to library callers — `apply_proposal()`, `skill_quality.py` — and not only to the CLI, as it was when it registered under `__main__` alone. The backend is built on first comparison, not in `__init__`: `get_enabled_evaluators()` instantiates evaluators per `run_evaluators()` call, and a missing `fastembed` now fails this evaluator closed inside `_run_one()` instead of raising out of the gate. Without a comparison set (or with an empty one) it **abstains** as `n/a` rather than reporting the 1.0 pass it used to. Duplicate detection takes `(name, body)` pairs and scores higher-is-better — 1.0 when it passes, `1 - similarity` when it blocks, naming the closest skill — because nearest-neighbour similarity between distinct installed skills already runs ~0.67–0.97 with the default model. **The default model (`BAAI/bge-small-en-v1.5`) truncates every input at 512 tokens (~2 KB)** while skills run ~9 KB at the median, so whole-body drift embeddings compare only each body's opening and come out identical for most changes; drift therefore splits both texts into windows of at most `DRIFT_WINDOW_CHARS` (1500) at headings, then line breaks, matches each new window to its closest old one, and reports the length-weighted mean of those matches — not the minimum, which on installed skills scored an added section (0.64–0.75) no better than a wholesale rewrite (0.54–0.66). Both default thresholds (0.90) come from measuring the default model on the 176 installed skills: duplicates at 0.85 flagged 31% of existing skills against a neighbour (0.90: 11%; real sibling pairs sit at 0.94–0.97), and weighted drift scored real applied changes and added sections at 0.98–1.00 against 0.64–0.73 for rewrites. A passing drift check scores 1.0 (the similarity goes in the feedback), like a passing duplicate check: every real change measures below 1.0, and fed into the regression mean that made a judge-improved change fail regression for being a change at all. Identical content abstains without embedding anything. **Every applicable check runs in one call** (it used to be one mode per call, by `elif`): duplicates (`create_new`) or drift (a body with a baseline on disk) — never both — plus grounding whenever the proposal cites sessions; the result fails if any check fails, scores the lowest, and names each check in its feedback. **Grounding** reads the cited sessions through `HostAdapter.read_sessions(ids)` (Hermes: `fetch_sessions.sessions_by_id()`, which checks the DB before connecting; Claude Code: `projects/*/.jsonl`, refusing ids that could leave `projects/`), already redacted, with each message's full truncated text as `content` (`_summarize_messages(full_content=True)`) rather than the 500-char preview. It compares only the proposal's new text (`_proposed_new_text()`: body/description `new_value`s) — the rest of the skill predates the proposal — windowed like drift, each new-text window matched to its closest session window, length-weighted mean; a pass scores 1.0. Sessions are read only when `embedding_similarity` is enabled. The 0.74 default comes from the 17 real proposals with new text and cited sessions: against their cited sessions they scored 0.71–0.85 (median 0.80), against 3 random draws of as many non-cited sessions 0.58–0.80 (median 0.69), AUC 0.96; at 0.74, 16/17 pass and 12% of non-cited draws do. The previous 0.60 passed every non-cited draw. A sample of 17 is small: re-measure once labels exist. Duplicate detection still embeds whole bodies, i.e. their openings. - **Gate combination**: `run_evaluators()` runs in three phases — (1) automatic evaluators (everything except `regression` and `human_review`), (2) `regression` fed the mean of automatic scores as `new_score` (human score never enters the regression baseline), (3) `human_review` last with `prior_results` context (automatic + regression verdicts). `combine_gate(results, strictness)` then combines per `SKILL_EVOLUTION_GATE_STRICTNESS` (or a per-proposal-type override, `SKILL_EVOLUTION_GATE_STRICTNESS_`, or a per-target override `SKILL_EVOLUTION_GATE_STRICTNESS_`, resolved by `resolve_gate_strictness()` — target beats type beats global): `"strict"` (default) requires every evaluator to pass; `"majority"` requires more than half. Zero configured evaluators never blocks. **An evaluator may abstain** (`EvalResult.not_applicable`, built with `Evaluator._not_applicable(reason)`) when it has nothing to compare for this content: the result is kept and shown as `=n/a` in gate feedback, but `applicable()` drops it before `combine_gate()`, the regression `new_score` mean, `_record_target`'s and `_cli_gate_result()`'s scores, and `skill_quality.py`'s mean and verdict — so an abstention can neither block nor tip a majority, and a target where everything abstained behaves like one with zero evaluators. Abstaining is not failing: an evaluator that tries and errors still fails closed (R21). `append_history()` writes the key only when true, like `transport_failure`. - **Multi-target gate (P2-1)**: `evaluate_and_record()` runs every target in `resolve_gate_targets()` (`SKILL_EVOLUTION_GATE_TARGETS`, default `skill,proposal`; unknown names raise `ValueError`; empty falls back to default) through the same registry and appends one combined `gate` entry per gating target — `skill:`, `proposal:`, and per-session `tool_calls:`/`analyzer_prompt:` — each combined under its own strictness, then ANDs the per-target decisions. Targets with no data don't block: a proposal without `session_ids` skips the session-based targets. A provider fault is flagged per-target on the entry it affected. The `kind` history field tags each entry's lineage (`skill_text`/`proposal`/`tool_calls`/`analyzer_prompt`); `migrate_proposal_history()` uses it to leave proposal-document entries under `proposal:` instead of folding them into the skill lineage. - **Provider adapter layer** (`call_provider()`) is stdlib-only `urllib` HTTP to Claude (`ANTHROPIC_API_KEY` + `SKILL_EVOLUTION_CLAUDE_MODEL`), Ollama/llama.cpp (`SKILL_EVOLUTION_OLLAMA_BASE_URL`/`SKILL_EVOLUTION_OLLAMA_MODEL`), OpenCode Zen (`OPENCODE_API_KEY` + `SKILL_EVOLUTION_OPENCODE_BASE_URL`/`SKILL_EVOLUTION_OPENCODE_MODEL`), OpenAI (`OPENAI_API_KEY` + `SKILL_EVOLUTION_OPENAI_BASE_URL`/`SKILL_EVOLUTION_OPENAI_MODEL`, default `https://api.openai.com/v1` / `gpt-4o`), or Gemini (`GEMINI_API_KEY` + `SKILL_EVOLUTION_GEMINI_BASE_URL`/`SKILL_EVOLUTION_GEMINI_MODEL`, default `https://generativelanguage.googleapis.com` / `gemini-2.0-flash`), selected by `SKILL_EVOLUTION_PROVIDER` (default `claude`) with an optional per-evaluator override `SKILL_EVOLUTION__PROVIDER`. **`SKILL_EVOLUTION_PROVIDER` may be a comma-separated fallback chain** (e.g. `claude,opencode,ollama`): `resolve_provider_chain()` splits on commas and `call_provider()` tries each in order — a *permanent* error on one provider (auth failure 401/403, missing API key, unknown provider name, malformed response shape) skips to the next, and a *transient* error (timeout, 5xx, rate-limit 429, connection-level URLError) retries within that provider per the backoff rules below before the chain moves on. Only when every provider in the chain has been exhausted does it raise `ProviderError` (fail-closed). A single value (no commas) preserves the original single-provider behaviour, so no existing configuration breaks. OpenAI and Gemini follow the same `_post_json()` pattern as the other three branches; Gemini is the one shape outlier — it hits `v1beta/models/{model}:generateContent` with a `contents[].parts[].text` request body and `candidates[].content.parts[].text` response, and authenticates via the `x-goog-api-key` header rather than a bearer token, keeping the key out of URLs. Both fail closed on a missing key or a malformed response shape, matching OpenCode's posture. `GEMINI_API_KEY` is also in `fetch_sessions.SECRET_PATTERNS`, so it's caught by the redaction layer below like `OPENAI_API_KEY`/`ANTHROPIC_API_KEY` already were. The HTTP timeout is **one generic knob for every provider** — `SKILL_EVOLUTION_PROVIDER_TIMEOUT` (seconds, default 60), resolved by `resolve_provider_timeout()` inside `call_provider()`; an explicit `timeout=` argument always wins. It is deliberately not per-provider (no `SKILL_EVOLUTION_OLLAMA_TIMEOUT`): the latency it exists for is a property of the model, not the adapter — local reasoning models spend most of their output on chain-of-thought before the JSON (a real `gemma4` judge call took 54–56s of the 60s default, and `qwen3.5` timed out outright). **Transient transport failures retry with bounded exponential backoff** — `SKILL_EVOLUTION_PROVIDER_RETRIES` (default `2`, resolved by `resolve_provider_retries()`) controls retries *after* the first attempt; `SKILL_EVOLUTION_PROVIDER_RETRY_BASE_SECONDS` (default `1.0`) sets the base delay (`base × 2^attempt`, so two retries cost ~1s + 2s), and a 429 `Retry-After` header overrides the schedule. `_is_retryable_transport_error()` gates the retry: HTTP 408/429/5xx, `TimeoutError`, and connection-level `URLError` retry; HTTP 4xx other than 408/429 (auth/bad-request) and malformed-output `ValueError` do not, because they won't fix themselves. `_is_permanent_provider_error()` serves the fallback chain: auth 401/403, a missing API key, an unknown provider name, or a response shape mismatch is *permanent* and skips to the next provider instead of burning backoff sleeps on a request that will never succeed. `0` for `RETRIES` disables retry and restores strict fail-fast. Every prompt is run through `redact_secrets()` (reusing `fetch_sessions.py`'s `contains_secret()`/`SECRET_PATTERNS`, whole-line replacement) before it ever leaves the machine, independent of the redaction `save_proposal()` already does before writing to disk — R18/R19 in the plan doc treat "sent to a provider" and "written to disk" as two separate redaction points, not one. - **History**: entries now also carry `content_size` and `baseline_size` (utf-8 bytes), which is what makes cumulative drift detectable — `original_size_for_target()` reads the *earliest* `baseline_size` for a target (falling back to the earliest `content_size`) so a candidate can be compared against where the skill began. Both fields are optional, so pre-existing history written before size recording simply yields `None` and leaves the cumulative check inert. `evaluate_and_record()` and `evaluate_skill_text()` both derive the evaluated text via the shared `_extract_evaluated_content()` helper, so the sizes recorded are measured over exactly the text that was scored. `append_history()`/`read_history()` read/write one shared append-only JSONL file (`get_history_path()`, default `./eval_history.jsonl` at the repo root, overridable via `SKILL_EVOLUTION_HISTORY_PATH` — see "Git tracking" above), keyed per-target (`target_key_for_proposal()` → `"skill:"` or `"proposal:"` fallback; for `create_new` proposals it extracts the skill name from `proposed_changes` before falling back, and `migrate_proposal_history()` rewrites history from `proposal:` to `skill:` after the skill is created). `prune_history()` enforces `SKILL_EVOLUTION_HISTORY_RETENTION` (a bare int = max versions, `"90d"`/`"6mo"` = age-based), and now **archives rather than deletes**: dropped entries are appended (not overwritten) to a sibling file before the primary is rewritten — `get_history_archive_path()`, default derived from the primary path's basename (`eval_history.jsonl` → `eval_history.archive.jsonl`), overridable independently via `SKILL_EVOLUTION_HISTORY_ARCHIVE_PATH`. Writing the archive *before* rewriting the primary matters: a crash mid-prune then loses nothing, where the reverse order could lose data permanently. R22 in the eval plan permits either "archived or compacted" — archiving was chosen because it already satisfies R22's actual goal (a bounded primary file) without a lossy "compacted summary" record needing to independently satisfy three different readers. **The survivor rule per target is up to five always-kept anchors, not one** (the three below plus the earliest entries carrying `baseline_size`/`content_size`, which is what `original_size_for_target()` actually reads — the earliest entry is often a sizeless refused attempt, and protecting only it let a prune move the cumulative baseline forward; both functions share `_earliest_with()`/`_ORIGINAL_SIZE_KEYS`) — the most recent entry (needed by `optimize_skill.find_low_scoring_targets()`), the most recent **passing** entry, and the **earliest** entry, deduplicated when they coincide. The retained count per target can therefore exceed the configured limit by up to two entries; that's intentional. Two real correctness holes motivated the extra two anchors: `RegressionEvaluator` baselines against the last entry with `passed == True`, not just the last entry — the old "keep only the most recent" rule could prune a passing baseline while newer failing retries survived, silently falling back to "no baseline yet" for exactly the targets that most need regression coverage; and `original_size_for_target()` reads the *earliest* entry as its cumulative-drift baseline — without protecting it, trimming a target's oldest entries would silently shift that baseline forward, forgiving prior drift. Both are closed by always retaining the anchor entry itself, rather than giving those two functions a fallback read into the archive file — same safety guarantee, no second data-access path to build and maintain. **Pruning now runs automatically, not just via `--prune`.** `evaluate_and_record()` takes `auto_prune: bool = True` and calls `prune_history()` right after its `append_history()` call — a no-op by default, since `prune_history()` itself early-returns unless `SKILL_EVOLUTION_HISTORY_RETENTION` is configured. The same env var now controls both *whether* pruning happens and that it happens on every real evaluation from then on; `apply_proposal()` calls `evaluate_and_record(proposal)` with no extra kwargs, so it gets this for free. `retroactive_reevaluate()`'s loop passes `auto_prune=False` and calls `prune_history()` once after the whole batch instead — `prune_history()` rewrites the *shared* history file (every target, not just the one being re-evaluated), so pruning after every append in an N-proposal batch would mean N full-file passes for a result identical to pruning once. `--prune` remains available for one-off backfill of pre-existing bloat, and its report now includes the archived count. **Skill-text entries say what they scored, and every judged entry says who judged.** `subject` is `installed` (`skill_quality.py`, the installed `SKILL.md`) or `candidate` (the gate, what a proposal would produce); both used to be indistinguishable `kind: skill_text` entries under `skill:`. `content_sha256` (`content_digest()`, 16 hex) lets a reader tell whether the installed skill still matches what was scored. `entry_subject()` infers the subject of older entries — `skill_quality.py` wrote no session ids and equal content/baseline sizes, which no gate entry does — and on the real history that splits the 426 skill-text entries into its 401 `skill_quality` outages and 25 proposal evaluations, with no rewrite. `judge` (`{"provider", "model", "rubric_version"}`) is set by `LLMJudgeEvaluator` from the provider that actually answered — `call_provider()` still returns plain text (two dozen test stubs depend on that signature) and records a `ProviderReply` in the `_LAST_PROVIDER_REPLY` ContextVar; `provider_model(name)` is the single place each provider's model id is resolved, used by every `_call_*` too. With a chain like `opencode,lmstudio`, consecutive scores can come from `big-pickle` and from a 3.8B local model. `JUDGE_RUBRIC_VERSION` is pinned by `tests/test_judge_provenance.py`, which hashes the judge prompt and fails until the version is bumped. `judge_provenance(results)` carries it onto the combined entry on all three write paths, like `transport_failure`, and `RegressionEvaluator` notes in its feedback when its baseline was judged by another model or rubric — the verdict is unchanged. On success the judge dict also carries the judge's own `score`, apart from the combined gate score (see the optimizer above). **A measurement of the installed skill is not a change.** `_evaluate_candidate()` sets `context["subject"]`: `installed` when the candidate equals the installed body read from disk (`skill_quality.py`'s synthetic proposal), else `candidate`. For an installed measurement it skips `original_size` — cumulative drift bounds what proposals may do, and on the first real pass it failed 6 skills on edits made outside the pipeline (one grew from 295B to 1.5KB). `RegressionEvaluator` compares like with like on `skill:` targets — installed against earlier installed measurements, candidates against earlier candidates — since mixing them made 3 quality measurements "regress" against proposal scores and would make every gate decision depend on the latest quality pass; gate decisions on existing history are unchanged (every earlier passing entry was a candidate). `skill_quality.get_previous_score()` likewise reads installed measurements only. **Provider faults are flagged on every gate-entry write path.** `evaluate_and_record()`, `skill_quality.py` and the single-target CLI (`_cli_gate_result()`) all set `transport_failure` when any evaluator failed on a provider fault; `find_low_scoring_targets()` skips flagged entries. History written before the last two paths carried the flag recorded outages as quality failures — locally 401 of 453 entries, all HTTP 401s from `skill_quality.py`, which put 174 skills on the optimizer's candidate list. `--backfill-transport-failures` marks those, matching `_PROVIDER_FAULT_FEEDBACK` (the provider-error shapes after `llm_judge failed closed: `) and **never the bare prefix**, which judge parse errors share. Report-only unless `--apply`; applying copies the file to `.bak-` and rewrites atomically, line by line, so a corrupt line that `_read_all_entries()` would silently drop survives verbatim. Entries whose feedback is only `/llm_judge=fail` (written by `_record_target`) carry no cause and cannot be classified after the fact. - **`evaluate_skill_text()`** scores the `Candidate` that `_build_candidate()` derives — the single derivation the gate, the history sizes recorded by `evaluate_and_record()` and `proposal_diagnostics.py` share — and builds the evaluator context. **Every change counts, not one per field.** `_extract_evaluated_content()` used to keep a `{field: change}` dict, so a proposal with several body changes had only its *last* one spliced and scored while apply wrote all of them: on both hosts a 70-byte skill grew to 52 KB with "all deterministic checks passed", an earlier empty `new_value` deleted its anchor on Claude Code, and an ambiguous earlier anchor went unexamined (8 of the 39 local proposals have several body changes). Now `_structural_problem()` rejects an empty `new_value` in *any* change, several description changes, or a whole-body replacement mixed with other body changes (naming the change), and `compose_body_changes()` applies every body change in order to the installed skill — each anchor matched exactly once against the text the earlier changes produced, as both adapters apply them — then folds a description change into the composed frontmatter via `skill_index.rewrite_frontmatter_description()` (frontmatter-scoped; shared with the Claude Code write). Several changes with no installed skill to apply them to fail closed; a single change keeps its old best-effort fallback. `_build_write_plan()` refuses a structurally unusable `improve_existing` before the gate. It passes `content_kind` **and `baseline_size`**, measured the way `DeterministicEvaluator` measures the candidate. That wiring is load-bearing: the growth-vs-baseline guard is `if baseline_size:`-gated, so omitting the key silently disables it — a proposal could balloon a skill +343% and still report "all deterministic checks passed." `create_new` has no baseline, so the key stays absent and the guard stays inert rather than dividing by a zero baseline. **`baseline_size` comes from the installed `SKILL.md`, not from the change's `old_value`** (`_resolve_baseline()` → `installed_skill_body()`, body changes only). `old_value` is written by the analyzer LLM — the analyzer prompt asks it to emit `old_value: ` and `proposal.py` persists it verbatim — so it is proposal-supplied input, not an observation. That became load-bearing when the cap turned into a ratchet: the cap now asks "is this larger than what it replaces?", so an inflated `old_value` would raise the very ceiling it is checked against, and a proposal claiming a 951KB baseline could ship a 950KB body past a 15KB cap while every percentage check reported a rounding-error delta. This is not mainly an adversarial concern — a duplicated or truncated transcription is an ordinary LLM failure, and `--retroactive` re-scores proposals whose `old_value` may no longer describe anything on disk. `optimize_skill._resolve_baseline_size_bytes()` already applied this distrust at draft time; this is the same discipline at the gate. Body changes only (for a description change the installed *file* is not what it replaces — substituting it would read as a ~−98% shrink), best-effort for a **missing** skill (fall back to `old_value` rather than fail a gate decision on a lookup miss) — but an **ambiguous** target, two distinct installed files declaring the name, fails closed: `evaluate_skill_text()` returns a `structure` failure naming both files (`ambiguous_skill_paths()`, real-path based so a symlink twin is one skill) before any baseline is read, and `evaluate_and_record()` records that entry without sizes. Falling back there measured the ratchet against the analyzer's own text, and apply could not tell which file to change; live case `github` vs `software-development/github`, which `skill_quality.py` now reports as ambiguous rather than "not found", and **shared with `evaluate_and_record()`** so the size recorded in history is the size that was actually judged. `_resolve_baseline()` also returns whether the baseline came from disk: only such a baseline is handed to `embedding_similarity` as `context["baseline"]` (comparing a proposal with its own `old_value` observes nothing), and a `create_new` body gets `context["existing_skills"]` — `(name, body)` for every installed skill via `_installed_skills_for_comparison()`, through the host adapter. Both are built only when `embedding_similarity` is enabled, since the duplicate set reads every installed skill; other targets get neither key and the evaluator abstains. `skill_quality.py`'s synthetic proposals splice to the installed body itself, so drift abstains there instead of adding a 1.0 that measured nothing. - **Three fast-follow evaluation targets** (R5) exist as sibling functions to `evaluate_skill_text()`, reusing the same registry and gate without modifying it or `apply_proposal()`'s call site: - `evaluate_proposal(proposal)` — scores the proposal as a document (summary, rationale, proposed_changes). Target key: `proposal:` (always, regardless of whether the proposal has a `target_skill`). No baseline — proposals are evaluated on their own merits. - `evaluate_tool_calls(session_id_or_messages)` — scores tool-call quality from session messages. Input can be a session_id (queries `state.db` via `_fetch_session_messages()`) or a list of message dicts (for testing). Target key: `tool_calls:`. Content is JSON-serialized tool_calls with result snippets capped at 200 chars each, total cap 5KB. - `evaluate_analyzer_prompt(session_id, proposal_id)` — scores whether the analyzer's generation step produced a grounded proposal. Reads session messages (truncated to 300 chars each, total cap 10KB) and the saved proposal markdown. Target key: `analyzer_prompt:`. **Wiring posture (P2-1)**: `proposal` gates auto-apply alongside `skill` by default; `tool_calls`/`analyzer_prompt` gate only if added to `SKILL_EVOLUTION_GATE_TARGETS`. The gate decision is the AND of every gating target, each combined under its own strictness. `deterministic` is opt-in only** for these targets (via `SKILL_EVOLUTION_EVALUATORS`) — its size/growth checks are meaningless for non-skill-text content. The default set (`llm_judge,regression`) is the right one for proposal/tool-call/analyzer-prompt evaluation. - **`evaluate_and_record()`** is the single call site both `apply_proposal()` and the retroactive path share — it runs each gating target (default `skill,proposal`), combines each under its own strictness, ANDs the per-target decisions, and appends one combined `EvalResult` (`evaluator_name="gate"`) **per gating target** to history, so both paths produce identical history semantics. - **Retroactive/batch mode** (`--retroactive`, `retroactive_reevaluate()`) only re-evaluates already-saved pending proposals (`proposed` or `approved`) by default (`_select_retroactive_proposals`) — rejected/applied proposals are excluded because they're not live decisions and re-scoring them would inject entries into the same history stream the live gate regresses against; pass `include_all_statuses=True` (CLI: `--all-statuses`) to opt in. `--target`/`--since` filter further. - **Module-identity gotcha**: running `python3 scripts/evaluate.py` directly loads it as `__main__`; since `proposal.py` does `import evaluate` at module scope, the file registers itself under `sys.modules["evaluate"]` at the top (`if __name__ == "__main__": sys.modules.setdefault(...)`) so both names resolve to the *same* module instance (same `REGISTRY`, same classes) instead of `proposal.py` re-executing and re-registering everything as a second, distinct module. ### Proposal types and the write side (`apply_proposal()` → `adapter.apply_skill_write(plan)`) | `type` | What it means | Hermes emits (`applied_by: agent`) | Claude Code does (`applied_by: direct`) | |---|---|---|---| | `improve_existing` | Patch a skill's description/body | one `patch` instruction per `proposed_changes` entry | rewrites the installed `SKILL.md` in place (each `old_value` must occur exactly once, else refuse) | | `create_new` | New skill for a recurring uncovered pattern | one `create` instruction carrying `name` + full `body` (fixed to require real content, not a placeholder) + description/category | writes a real `skills//SKILL.md` (refuses existing names) | | `merge_skills` | Combine overlapping skills into `target_skill` | one `delete` instruction per `source_*`-prefixed change, each tagged `absorbed_into` | archives each source into `skills/.archive//` after validating the umbrella exists | | `deprecate_skill` | Remove a stale/unused skill | one `delete` instruction | archives the skill into `skills/.archive//` | All four types pass through the same evaluation gate (above) before the adapter is asked to mutate anything — the gate runs once per `apply_proposal()` call regardless of `type`, and `apply_proposal()` only flips `status` to `applied` after the adapter reports `can_apply: True`. ## Conventions specific to this codebase - **Prefer host-agnostic design.** Multi-agent support (Codex, Claude Code, Claude, future providers) is an explicit project goal, and the owner's stated principle is that the more agnostic the design, the better — so treat host-neutrality as the default for new work instead of a later retrofit. The Hermes coupling used to be four hardcoded points scattered one-per-module; the **read side of that is now a real abstraction, not just an isolation convention.** `scripts/host.py` defines `HostAdapter` (an ABC with three read methods — `iter_sessions(since=None)`, `iter_skills()`, `read_skill_body(skill_name)` — plus the write side: `supports_write: bool = False` and a fail-closed concrete `apply_skill_write(plan)` that returns `{"can_apply": False, "reason": ...}` unless an adapter overrides it), a plain-dict `HOST_ADAPTERS` registry populated by `register_adapter()` at import time (instances, not classes — a host adapter carries no per-call state, so nothing is gained by re-instantiating per lookup, unlike `evaluate.REGISTRY`), and `resolve_host()`/`get_adapter()` reading `SKILL_EVOLUTION_HOST` (default `hermes`) in the same explicit-arg-beats-env-var-beats-default order `evaluate.resolve_provider()` already uses — an unknown host name raises rather than silently reading the wrong tree. `HermesAdapter` reproduces today's behavior byte-for-byte by delegating to `fetch_sessions.fetch_sessions()`/`skill_index.scan_skills()`. `ClaudeCodeAdapter` (root: `SKILL_EVOLUTION_CLAUDE_CODE_HOME` > `CLAUDE_CONFIG_DIR` — Claude Code's own config-dir override, which it ignored, so a profile kept there was invisible — > `~/.claude`) is the second adapter, reading skills from the flat `skills/*/SKILL.md` tree (no category level — `category` is reported as the constant `"user"`) and sessions from `projects/*/*.jsonl`, one JSONL-per-session with an undocumented, versioned record format handled defensively (allowlist role mapping, best-effort timestamp/field extraction, never raising on a malformed line or record) — it routes through the same `fetch_sessions.contains_secret()`/`redact_pii()`/`_summarize_messages()` redaction chokepoint rather than reimplementing it. `evaluate.py` (`installed_skill_body()`) and `optimize_skill.py` now read skills through `host.get_adapter()` instead of calling `skill_index.scan_skills()` directly, so both are host-neutral as of this abstraction landing. **The two host-neutral contracts a new adapter must produce** are the session dict shape (`session_id`, `started_at`, `title`, `model`, `source`, `message_count`, `user_messages`, `assistant_messages`, `messages` — the last already redacted/truncated) and the skill dict shape (`name`, `category`, `description`, `path`, `size`); `read_skill_body()`'s contract is narrower still — best-effort text or `None` on no match, an ambiguous match, or a read failure, *never* raise, since callers like the evaluation gate's baseline-size lookup must degrade to whatever baseline they already had rather than fail a gate decision on an unrelated lookup. `iter_archived_skills()` (concrete, default `[]`) lists retired skills in the same record shape — Hermes indexes `/.archive`, Claude Code `/skills/.archive`, both via `scan_skills()` since it reads flat and grouped layouts — so a caller can tell "archived" from "unresolved" without probing one host's paths; a host with no archive reports nothing and its callers say "unresolved". `read_sessions(session_ids)` (concrete, default `[]`, never raises) returns cited sessions by id in the session dict shape, each message also carrying its full redacted text as `content`, with no processed-state or lookback filter — it serves evidence lookups (the grounding check), not the analyzer's feed. **The write side shipped with the same abstraction (2026-07-31).** `proposal.py`'s `apply_proposal()` remains the sole mutation path, but it now builds a host-neutral mutation plan and hands it to `adapter.apply_skill_write(plan)`, and it refuses (before the gate runs) on any host that can't write or any proposal that can't be expressed as a mutation — never spending a provider call on a proposal that can never apply. `HermesAdapter.supports_write = True` and its `apply_skill_write()` re-emits the legacy `skill_manage` instruction dicts byte-for-byte (returned `applied_by: agent` for the cron agent to execute later). `ClaudeCodeAdapter.supports_write = True` and its `apply_skill_write()` performs the writes directly and immediately (returned `applied_by: direct`): real `skills//SKILL.md` files via atomic temp-file + `os.replace` (`_atomic_write_text()`, which keeps a replaced file's permission bits and gives a new file the umask default — `mkstemp()`'s 0600 used to leak through, and the history backfill shares the helper), deprecate/merge archive sources into `skills/.archive//`, and **symlinked skill directories are refuse-to-write** (38/41 installed Claude Code skills are symlinks into `~/.agents/skills/` — the adapter writes around them instead of following or replacing them). The Hermes adapter is instruction-emitting and the Claude Code adapter is file-writing, so they cannot share a code path; what they share is the gate, the plan shape, and the fail-closed posture. **The processed-session state seam shipped (2026-08-01).** `HostAdapter` now has three concrete state methods — `iter_processed()`, `mark_processed(session_ids)`, `prune_processed(retention=None, keep_ids=None)` — plus a `_state_file()` hook for path resolution. Each adapter owns its own state file: `HermesAdapter` inherits the default (`~/.hermes/skill_evolution_state.json`), while `ClaudeCodeAdapter` overrides `_state_file()` to return `/skill_evolution_state.json`. `SKILL_EVOLUTION_STATE_FILE` remains the universal override — when set, it redirects whichever host is active. `fetch_sessions_for_host()` and `main() --prune-state` now route through the adapter's state methods instead of calling module-level functions with `host=`. `state.py` is now imported by `host.py` at module scope (no cycle — `state.py` imports nothing local), making it a real library rather than a test-only module. The host-prefix scheme (`:` keys) stays for legacy-entry handling and shared-override safety. `fetch_sessions()` (the Hermes cron path) stays byte-for-byte — it's the critical deployed path and already calls the state functions with `host="hermes"` and the default path. That includes prose in prompts: the GEPA `objective` deliberately says "this agent skill", not "this Hermes Agent skill", both to survive generalization and to avoid biasing the reflection LM toward one host's conventions. The remaining write-side follow-up is the project-level `.claude/skills/` tree, which is deliberately out of scope — the read side walks one flat root (`SKILL_EVOLUTION_CLAUDE_CODE_HOME`) and "which project" is undefined at the adapter level. - Every script is a standalone stdlib-only CLI with its own `argparse` in `main()`, importable for its core function (`fetch_sessions()`, `scan_skills()`, `apply_proposal()`, `run_evaluators()`, etc.) — keep new scripts consistent with this pattern rather than introducing a shared framework/package layout. - No PyYAML anywhere by design (`SKILL.md` promises "no pip packages required") — frontmatter is parsed and rendered by hand in both `skill_index.py` and `proposal.py`. If a change needs richer YAML (nested structures beyond flat keys + one-level lists), it likely needs a matching parser update in `_parse_frontmatter`, not just a `render()` change. - Auto-apply is opt-in via env vars (`SKILL_EVOLUTION_AUTO_APPLY`, `SKILL_EVOLUTION_MIN_CONFIDENCE`), read by the cron agent/prompt, not by the Python scripts themselves. The evaluation gate inside `apply_proposal()`, by contrast, always runs (it's not behind an opt-in flag) — `SKILL_EVOLUTION_EVALUATORS`/`SKILL_EVOLUTION_GATE_STRICTNESS` only change *which* evaluators run and how strictly, not whether the gate runs at all. - `gepa` is imported lazily (`_require_gepa()` inside `optimize_skill.py`, called from `run_gepa_optimization()`, not at module import time) so the rest of the pipeline works with the dependency absent — don't move that import to module scope. **`_require_gepa()` returns the `gepa.optimize_anything` *submodule*, not the top-level `gepa` package** — that submodule is the namespace holding the `optimize_anything` function plus `GEPAConfig`/`EngineConfig`/`ReflectionConfig`. The package deliberately binds the name `optimize_anything` to the submodule (see `gepa/__init__.py`: "expose submodule; use `from gepa.optimize_anything import optimize_anything` for the function"), so reading those names off the package yields a non-callable module and three `AttributeError`s. Verified against `gepa==0.1.4`; the parameter names the call site passes (`seed_candidate`/`evaluator`/`objective`/`config`, `max_metric_calls`, `reflection_lm`) are all correct for that version. - **Stubbed-dependency tests can't catch API drift** — every test in `tests/test_optimize_skill.py` monkeypatches `_require_gepa` to return a hand-built fake, so it asserts the code against its own assumptions. Pair that with a contract test against the real package (`tests/test_optimize_skill_gepa_contract.py`, guarded by `pytest.importorskip("gepa")` so it skips without the optional extra). The same rule applies to seams between modules: the isolated `DeterministicEvaluator` growth tests pass `baseline_size` in by hand, which is why nothing caught `evaluate_skill_text()` never supplying it — assert wiring end-to-end, not just each side of the seam. - **Test isolation is structural (`tests/conftest.py`).** Per test, every inherited `SKILL_EVOLUTION_*` variable and `CLAUDE_CONFIG_DIR` is dropped and each runtime path (history, proposals, reports, session DB, Claude Code home; the state file via the `state.STATE_FILE`/`fetch_sessions.STATE_FILE` module globals, the convention the state tests already use) points at a temp dir. Per session, the real state files, history, `proposals/`, `reports/` and skill trees are fingerprinted and any change fails the run naming the path. Both exist because a conformance fixture patched `fetch_sessions.STATE_FILE` while `HermesAdapter` resolves through `state.get_state_file()` — so every `pytest` wrote to, then ran a 30d prune over, the real `~/.hermes/skill_evolution_state.json`, and its 101 real entries were lost. A test that checks a *default* path must `delenv` the variable itself. `tests/host_sandbox.py` builds a real Hermes or Claude Code tree per test; behaviour that exists on both hosts is tested on both by parametrizing over `HOSTS`. - Tests exist now (`tests/test_*.py`, one file per evaluator/feature area, run via `pytest`; last verified 2026-10-18: **1143 passed + 1 skipped** under a `python3` that has `fastembed` but not `gepa` — the skip is the gepa-contract module, guarded by `pytest.importorskip("gepa")`. Under a `.venv` with `gepa` but not `fastembed` it's **1140 passed + 7 skipped**, all 7 skips being `tests/test_embedding_backends.py` cases guarded by `pytest.importorskip("fastembed")` — so the exact pass/skip split depends on which optional extras are installed, not just `gepa` alone. Treat any specific number here as a point-in-time sanity check, not a contract — the suite changes; re-run `pytest -q -rs` for the current count rather than trusting this line) — this replaces an earlier state where `pyproject.toml` configured `pytest` but no `tests/` directory existed. New evaluators or gate behavior should get a matching `tests/test_evaluate_*.py`.