aeon-skill-evals · diff
git:20260512.9addfe3 to git:20260512.2331923
88 added, 93 removed. Audit A to A.
---
name: aeon-skill-evals
description: |
- Validate skill outputs against assertion manifests, diff vs prior runs to flag regressions, file
- issues for new failures, queue concrete fixes. Production-grade quality net for the skill catalog.
- Use for continuous quality monitoring, pre-deploy gates on new skill variations, or triage source
- for systemic catalog issues.
- Triggers: "evaluate my skills", "check skill output quality", "regression scan", "bootstrap evals
- for skill X", "which skills are failing assertions".
+ Validate the output of any installed skill against an assertion manifest — word counts, required
+ patterns, forbidden phrases, output shape, source citation. Detects regressions before users do.
+ Bootstrap mode generates a starter manifest from a skill's recent successful runs. Use as a
+ pre-deploy gate on skill variations, continuous quality monitoring, or to catch hallucinated
+ output before it goes downstream.
+ Triggers: "evaluate this skill's output", "check skill X for regressions", "bootstrap evals
+ for Y", "did this skill output pass quality gates", "is the output of X regressing".
---
# aeon-skill-evals
- Quality net for the skill catalog. Each skill can declare an assertion manifest — pattern matches, word-count floors, forbidden phrases, required structure. Every run's output is checked against the manifest; failing assertions produce regression alerts and queued fixes routed to `aeon-skill-repair`.
+ Quality net for installed skills. Each skill can declare an assertion manifest; every output run is checked against the manifest; failing assertions surface regressions and route concrete fixes.
- The job isn't to prevent bad output. It's to catch it the moment it ships and route the fix before a human notices.
+ The job isn't to prevent every bad output. It's to catch the regression the moment it happens and route it to repair before a human notices.
## Manifest format
+ A YAML or JSON file (e.g. `evals.yaml`) keyed by skill name:
+
```yaml
- # evals.json (or YAML)
token-movers:
min_words: 200
required_patterns:
- "Top movers"
- "24h"
forbidden_patterns:
- "I cannot"
- "as an AI"
- output_pattern: "memory/logs/*.md"
- min_distinct_tokens: 5
+ - "as a language model"
must_cite_source: true
+ min_distinct_items: 5
narrative-tracker:
min_words: 400
- required_patterns:
+ required_sections:
- "TRANSITIONS"
+ - "POSITIONS"
- "MAP"
- required_sections:
- - "transitions"
- - "positions"
forbidden_patterns:
- "exciting"
- "consider"
must_have_position_call: true
- ```
-
- Assertion types: minimum/maximum word counts, required substrings, forbidden phrases, output-shape regex, source-citation requirements, deduplication minimums, per-section presence checks, position-call requirements.
-
- ## Regression detection
-
- Compares the current run's assertion-pass set against the prior run's:
+ must_name_drivers: true
- | State | Meaning | Action |
- |---|---|---|
- | **NEW_FAIL** | Passing last run, failing now. | File an issue; severity scales with prior pass streak. |
- | **NEW_PASS** | Failing last run, passing now. | Close the open issue, log the win. |
- | **CHRONIC** | Failing > 3 consecutive runs. | Escalate severity, recommend `enabled: false` until repair. |
- | **STABLE_FAIL** | Always failing. | Don't re-file; flag as known issue needing redesign. |
+ paper-pick:
+ required_sections:
+ - "central claim"
+ - "why it's worth"
+ - "where it might be wrong"
+ - "read order"
+ must_cite_paper: true
+ ```
- ## Issue filing
+ Assertion types:
+ - `min_words` / `max_words` — word count bounds.
+ - `required_patterns` / `forbidden_patterns` — substring matches.
+ - `required_sections` — markdown header presence.
+ - `must_cite_source` / `must_cite_paper` — at least one URL or arXiv link.
+ - `min_distinct_items` — list-output dedup floor.
+ - `output_pattern` — regex over the full output.
+ - Custom binary checks (`must_have_position_call`, etc.) — defined per skill family.
- For each `NEW_FAIL`, files a structured issue:
+ ## Operations
- ```yaml
- ---
- id: ISS-104
- title: "token-movers: forbidden 'as an AI' matched in output"
- status: open
- severity: medium
- category: quality-regression
- detected_by: aeon-skill-evals
- detected_at: 2026-05-12
- affected_skills: [token-movers]
- ---
+ | Operation | Description |
+ |---|---|
+ | `eval` | Run every manifest-defined skill against its latest output. |
+ | `eval --skill=NAME` | One skill. |
+ | `bootstrap --skill=NAME` | Generate a starter manifest from recent successful runs. |
- ## Failing assertion
- forbidden_patterns: "as an AI"
+ ## Regression detection
- ## Where it appeared
- memory/logs/2026-05-12.md, line 42 of token-movers output
+ Compares the current run's assertion-pass set against the prior run:
- ## Pass streak before failure
- 17 consecutive runs
+ | State | Meaning |
+ |---|---|
+ | **NEW_FAIL** | Passing last run, failing now. Severity scales with prior pass streak. |
+ | **NEW_PASS** | Failing last run, passing now. Logged as a win. |
+ | **CHRONIC** | Failing > 3 consecutive runs. Recommendation: disable until human review. |
+ | **STABLE_FAIL** | Always failing. Indicates a manifest assertion that doesn't match the skill's actual output — flag for manifest review. |
- ## Recommended next
- aeon-skill-repair --target=token-movers (category: prompt-bug)
- ```
+ Stored in a local `evals-state.json`.
## Bootstrap mode
- Pointed at a skill with no manifest:
+ Pointed at a skill with no manifest, the skill:
- | Field | How bootstrapped |
- |---|---|
- | Word-count floor | p25 of historical successful runs. |
- | Required patterns | Most common section headers in recent successful outputs. |
- | Forbidden patterns | Default list (refusals, hedging filler, hallucination markers). |
- | Output pattern | The skill's actual write target. |
+ 1. Samples the last 5 successful run outputs.
+ 2. Computes:
+ - `min_words` at p25 of historical runs (so a slightly-shorter run still passes).
+ - Required patterns from the most common section headers / structural elements.
+ - Forbidden patterns from a default list (refusals, hedging filler, hallucination markers).
+ - Custom binary checks where the skill family has them.
+ 3. Emits the proposed manifest for review.
- Proposed manifest is presented as a PR for review, never auto-committed.
+ The operator reviews and saves the manifest. The skill never auto-commits a bootstrapped manifest — assertions are observations, and observations need a human signoff.
## Output
```
*Skill Evals — 2026-05-12*
- 47 skills checked
- 42 PASS
- 3 NEW_FAIL — issues filed
- 2 NEW_PASS — issues closed
- 5 CHRONIC — operator review recommended
- 1 coverage gap (no manifest)
+ Checked 14 skills against their manifests
- NEW_FAIL details
- token-movers — forbidden "as an AI" matched (3-line refusal in output)
+ PASS (11)
+ FAIL (2 NEW)
+ token-movers — forbidden "as an AI" matched in output
Pass streak before failure: 17 runs
Severity: medium (regression, not always-broken)
- Filed: ISS-104, recommended next: aeon-skill-repair --target=token-movers
+ Recommended: aeon-skill-repair --target=token-movers
- daily-routine — required pattern "Top tokens" absent (output truncated)
+ paper-pick — required section "where it might be wrong" missing
Pass streak before failure: 8 runs
Severity: medium
- Filed: ISS-105, recommended next: aeon-skill-repair --target=daily-routine
-
- narrative-tracker — min_words 400 not met (actual 220 — empty signals day)
- This may be expected on a quiet day; flagged as intermittent.
+ Recommended: aeon-skill-repair --target=paper-pick
- NEW_PASS
- github-trending — back to passing after autoresearch evolution last week. ISS-098 closed.
- paper-pick — back to passing after repair PR #412 landed. ISS-101 closed.
+ NEW_PASS (1)
+ github-trending — back to passing after a fix landed. Closing prior fail record.
- CHRONIC — recommend operator review
- ai-framework-watch (4 runs failing): required pattern "watchlist" absent
- ...
+ CHRONIC (1)
+ some-skill — failing 4 consecutive runs on "must_cite_source"
+ Recommendation: operator review. Either fix the skill or relax the assertion.
- Coverage
- 46/47 skills have manifests; bootstrap recommended for: spawn-instance
+ COVERAGE
+ 3 installed skills have no manifest: huggingface-trending, hn-digest, rss-digest
+ Run: bankr-run skill-evals bootstrap --skill=<name>
```
- ## Pipeline integration
+ ## Pipeline position
- - **Downstream of:** any skill that writes output.
- - **Upstream of:**
- - `aeon-skill-repair` (uses failing assertions as fix targets).
- - `aeon-skill-health` (consumes pass-rate over time).
- - `aeon-operator-scorecard` (weekly synthesis input).
+ - **Downstream of**: any skill that writes output you care about.
+ - **Upstream of**:
+ - `aeon-skill-repair` (failing assertions → fix targets).
+ - Operator dashboards (pass rate / regression cadence).
## Guidelines
- - Assertions are observations, not specifications. Bootstrap from real output, don't write speculatively.
- - Forbidden patterns catch hallucination markers and refusals — keep the list tight.
- - Chronic failures get escalated, not re-filed.
- - Manifest changes go through PR, never inline.
- - Coverage gaps (skills without manifests) are surfaced separately — they're a different problem from failures.
+ - Assertions are observations from real output, not specifications. Bootstrap before writing speculatively.
+ - Forbidden patterns catch hallucination markers and refusals. Keep the list tight; don't lint stylistic choices.
+ - Chronic failures get a recommendation, not a re-file. Same issue, same response.
+ - Manifest changes are reviewed; never auto-edited by this skill.
+ - Coverage gaps (skills without manifests) are surfaced separately — a known unknown is different from a failure.
+ ## Required keys
+
+ None — local file evaluation only.
+
## Pairs with
- - `aeon-skill-repair` (downstream — fixes the failures).
- - `aeon-autoresearch` (when failures are quality-regression, not deterministic).
+ - `aeon-skill-repair` downstream (consumes failing assertions as fix targets).
+ - `aeon-autoresearch` when failures are quality-regression rather than deterministic.
+ - `aeon-skill-security-scan` as a sibling input-safety net.