evaluation-methodology · diff
git:20260325.b4afcc4 to git:20260326.c3db094
465 added, 27 removed. Audit A to A.
---
name: evaluation-methodology
- description: "PluginEval quality methodology — dimensions, rubrics, statistical methods. Use when understanding how plugin quality is measured or when interpreting evaluation results."
+ description: "PluginEval quality methodology — dimensions, rubrics, statistical methods, and scoring formulas. Use this skill when understanding how plugin quality is measured, interpreting evaluation results, calibrating scoring thresholds, or explaining quality badges to stakeholders."
---
# Evaluation Methodology
- ## Quality Dimensions (Ranked by Weight)
+ This document is the authoritative reference for how PluginEval measures plugin and skill quality.
+ It covers the three evaluation layers, all ten scoring dimensions, the composite formula, badge
+ thresholds, anti-pattern flags, Elo ranking, and actionable improvement tips.
- 1. **Triggering accuracy (0.25)** — Does the skill fire when it should?
- 2. **Orchestration fitness (0.20)** — Works as a worker in agent→skill hierarchy?
- 3. **Output quality (0.15)** — Consistent, correct results?
- 4. **Scope calibration (0.12)** — Right depth and breadth?
- 5. **Progressive disclosure (0.10)** — SKILL.md lean, depth in refs?
- 6. **Token efficiency (0.06)** — Minimal context waste?
- 7. **Robustness (0.05)** — Handles edge cases?
- 8. **Structural completeness (0.03)** — Right sections present?
- 9. **Code template quality (0.02)** — Working examples?
- 10. **Ecosystem coherence (0.02)** — Cross-references, no duplication?
+ Related: [Full rubric anchors](references/rubrics.md)
- ## Three Evaluation Layers
+ ---
- - **Layer 1 (Static):** Deterministic analysis, <2 seconds, no LLM
- - **Layer 2 (LLM Judge):** G-Eval style rubric assessment via Agent SDK
- - **Layer 3 (Monte Carlo):** N simulated invocations with statistical CIs
+ ## The Three Evaluation Layers
- ## Statistical Methods
+ PluginEval stacks three complementary layers. Each layer produces a score between 0.0 and 1.0 for
+ each applicable dimension, and later layers override or blend with earlier ones according to
+ per-dimension blend weights.
- - **Wilson score interval:** Triggering activation rate CIs
- - **Bootstrap resampling (1000x):** Output quality CIs
- - **Clopper-Pearson exact:** Failure rate CIs
- - **Cohen's kappa:** Inter-judge agreement
- - **Elo/Bradley-Terry:** Pairwise ranking against gold corpus
+ ### Layer 1 — Static Analysis
+ **Speed:** < 2 seconds. No LLM calls. Deterministic.
+
+ The static analyzer (`layers/static.py`) runs six sub-checks directly against the parsed SKILL.md:
+
+ | Sub-check | What it measures |
+ |---|---|
+ | `frontmatter_quality` | Name presence, description length, trigger-phrase quality |
+ | `orchestration_wiring` | Output/input documentation, code block count, orchestrator anti-pattern |
+ | `progressive_disclosure` | Line count vs. sweet-spot (200–600 lines), references/ and assets/ bonuses |
+ | `structural_completeness` | Heading density, code blocks, examples section, troubleshooting section |
+ | `token_efficiency` | MUST/NEVER/ALWAYS density, duplicate-line repetition ratio |
+ | `ecosystem_coherence` | Cross-references to other skills/agents, "related"/"see also" mentions |
+
+ These six sub-checks feed directly into six of the ten final dimensions (via `STATIC_TO_DIMENSION`
+ mapping). The remaining four dimensions — `output_quality`, `scope_calibration`,
+ `robustness`, and part of `triggering_accuracy` — receive no static contribution and rely
+ entirely on Layer 2 and/or Layer 3.
+
+ **Anti-pattern penalty** is applied multiplicatively to the Layer 1 score:
+
+ ```
+ penalty = max(0.5, 1.0 − 0.05 × anti_pattern_count)
+ ```
+
+ Each additional detected anti-pattern reduces the score by 5%, flooring at 50%.
+
+ ### Layer 2 — LLM Judge
+
+ **Speed:** 30–90 seconds. One or more LLM calls (Sonnet by default). Non-deterministic.
+
+ The `eval-judge` agent reads the SKILL.md and any `references/` files, then scores four
+ dimensions using anchored rubrics (see [references/rubrics.md](references/rubrics.md)):
+
+ 1. **Triggering accuracy** — F1 score derived from 10 mental test prompts
+ 2. **Orchestration fitness** — Worker purity assessment (0–1 rubric)
+ 3. **Output quality** — Simulates 3 realistic tasks; assesses instruction quality
+ 4. **Scope calibration** — Judges depth and breadth relative to the skill's category
+
+ The judge returns a structured JSON object (no markdown fences) that the eval engine merges
+ into the composite. When `judges > 1`, scores are averaged and Cohen's kappa is reported as
+ an inter-judge agreement metric.
+
+ ### Layer 3 — Monte Carlo Simulation
+
+ **Speed:** 5–20 minutes. N=50 simulated Agent SDK invocations (default). Statistical.
+
+ Monte Carlo runs `N` real prompts through the skill and records:
+
+ - **Activation rate** — Fraction of prompts that triggered the skill
+ - **Output consistency** — Coefficient of variation (CV) across quality scores
+ - **Failure rate** — Error/crash fraction with Clopper-Pearson exact CIs
+ - **Token efficiency** — Median token count, IQR, outlier count
+
+ The Layer 3 composite formula:
+
+ ```
+ mc_score = 0.40 × activation_rate
+ + 0.30 × (1 − min(1.0, CV))
+ + 0.20 × (1 − failure_rate)
+ + 0.10 × efficiency_norm
+ ```
+
+ where `efficiency_norm = max(0, 1 − median_tokens / 8000)`.
+
+ ---
+
+ ## Composite Scoring Formula
+
+ The final score is a weighted blend across all three layers for each dimension, then summed:
+
+ ```
+ composite = Σ(dimension_weight × blended_dimension_score) × 100 × anti_pattern_penalty
+ ```
+
+ ### Dimension Weights
+
+ | Dimension | Weight | Why it matters |
+ |---|---|---|
+ | `triggering_accuracy` | 0.25 | A skill that never fires — or fires incorrectly — has no value |
+ | `orchestration_fitness` | 0.20 | Skills must be pure workers; supervisor logic belongs in agents |
+ | `output_quality` | 0.15 | Correct, complete output is the primary deliverable |
+ | `scope_calibration` | 0.12 | Neither a stub nor a bloated monster |
+ | `progressive_disclosure` | 0.10 | SKILL.md is lean; detail lives in references/ |
+ | `token_efficiency` | 0.06 | Minimal context waste per invocation |
+ | `robustness` | 0.05 | Handles edge cases without crashing |
+ | `structural_completeness` | 0.03 | Correct sections in the right order |
+ | `code_template_quality` | 0.02 | Working, copy-paste-ready examples |
+ | `ecosystem_coherence` | 0.02 | Cross-references; no duplication with siblings |
+
+ ### Layer Blend Weights
+
+ Each dimension draws from different layers at different ratios. With all three layers active
+ (`--depth deep` or `certify`):
+
+ | Dimension | Static | Judge | Monte Carlo |
+ |---|---|---|---|
+ | `triggering_accuracy` | 0.15 | 0.25 | 0.60 |
+ | `orchestration_fitness` | 0.10 | 0.70 | 0.20 |
+ | `output_quality` | 0.00 | 0.40 | 0.60 |
+ | `scope_calibration` | 0.30 | 0.55 | 0.15 |
+ | `progressive_disclosure` | 0.80 | 0.20 | 0.00 |
+ | `token_efficiency` | 0.40 | 0.10 | 0.50 |
+ | `robustness` | 0.00 | 0.20 | 0.80 |
+ | `structural_completeness` | 0.90 | 0.10 | 0.00 |
+ | `code_template_quality` | 0.30 | 0.70 | 0.00 |
+ | `ecosystem_coherence` | 0.85 | 0.15 | 0.00 |
+
+ At `--depth standard` (static + judge only), blends are renormalized to drop the Monte Carlo
+ column. At `--depth quick` (static only), all weight falls on Layer 1.
+
+ ### Blended Score Calculation
+
+ For a given depth, the blended score for dimension `d` is:
+
+ ```
+ blended[d] = Σ( layer_weight[d][layer] × layer_score[d][layer] )
+ ─────────────────────────────────────────────────────
+ Σ( layer_weight[d][layer] for available layers )
+ ```
+
+ This normalization ensures that skipping Monte Carlo at standard depth doesn't artificially
+ deflate scores.
+
+ ---
+
+ ## Interpreting Dimension Scores
+
+ Each dimension score is a float in `[0.0, 1.0]`. The CLI converts it to a letter grade:
+
+ | Grade | Score range | Meaning |
+ |---|---|---|
+ | A | 0.90 – 1.00 | Excellent — no meaningful improvement needed |
+ | B | 0.80 – 0.89 | Good — minor gaps only |
+ | C | 0.70 – 0.79 | Adequate — one or two clear improvement areas |
+ | D | 0.60 – 0.69 | Marginal — needs targeted work |
+ | F | < 0.60 | Failing — significant remediation required |
+
+ When reading a report, focus first on the lowest-graded dimension that has the highest weight.
+ A D in `triggering_accuracy` (weight 0.25) costs far more than a D in `ecosystem_coherence`
+ (weight 0.02).
+
+ **Confidence intervals** appear in the report when Layer 2 or Layer 3 ran. Narrow CIs (± < 5
+ points) indicate stable scores. Wide CIs suggest inconsistency — often caused by an ambiguous
+ description or instructions that work for some prompt styles but not others.
+
+ ---
+
## Quality Badges
+ Badges require both a composite score threshold AND an Elo threshold (when Elo is available).
+ The `Badge.from_scores()` logic checks composite first, then Elo if provided:
+
| Badge | Composite | Elo | Meaning |
- |-------|-----------|-----|---------|
- | Platinum | 90+ | 1600+ | Reference quality |
- | Gold | 80+ | 1500+ | Production ready |
- | Silver | 70+ | 1400+ | Functional, needs improvement |
- | Bronze | 60+ | 1300+ | Minimum viable |
+ |---|---|---|---|
+ | Platinum ★★★★★ | ≥ 90 | ≥ 1600 | Reference quality — suitable for gold corpus |
+ | Gold ★★★★ | ≥ 80 | ≥ 1500 | Production ready |
+ | Silver ★★★ | ≥ 70 | ≥ 1400 | Functional, has improvement opportunities |
+ | Bronze ★★ | ≥ 60 | ≥ 1300 | Minimum viable — not yet recommended for users |
+ | — | < 60 | any | Does not meet minimum bar |
+
+ The Elo threshold is skipped when Elo has not been computed (i.e., at quick or standard depth
+ without `certify`). A skill can earn a badge on composite score alone in those cases.
+
+ ---
+
+ ## Anti-Pattern Flags
+
+ The static analyzer detects five anti-patterns. Each carries a severity multiplier that feeds
+ into the penalty formula.
+
+ ### OVER_CONSTRAINED
+
+ **Trigger:** More than 15 occurrences of MUST, ALWAYS, or NEVER in the SKILL.md.
+
+ **Problem:** Overly prescriptive instructions reduce model flexibility, increase token overhead,
+ and signal that the author is trying to micromanage every output rather than providing
+ principled guidance.
+
+ **Fix:** Audit every MUST/ALWAYS/NEVER. Replace directive language with explanatory framing
+ where possible. Reserve hard constraints for genuine safety or correctness requirements. Target
+ fewer than 10 such directives per 100 lines.
+
+ ### EMPTY_DESCRIPTION
+
+ **Trigger:** The frontmatter `description` field is fewer than 20 characters after stripping.
+
+ **Problem:** Without a meaningful description, the Claude Code plugin system cannot determine
+ when to invoke the skill. The skill becomes invisible to autonomous invocation.
+
+ **Fix:** Write a description of at least 60–120 characters that includes:
+ - A "Use this skill when..." or "Use when..." trigger clause
+ - Two or more concrete contexts separated by commas or "or"
+
+ ### MISSING_TRIGGER
+
+ **Trigger:** The description does not contain "use when", "use this skill when",
+ "use proactively", or "trigger when" (case-insensitive).
+
+ **Problem:** Even a long description is useless for autonomous invocation if it doesn't
+ include a clear trigger signal. The system's routing model needs an explicit cue.
+
+ **Fix:** Prepend "Use this skill when..." to the description, followed by specific scenarios.
+ Example: "Use this skill when measuring plugin quality, interpreting score reports, or
+ explaining badge thresholds to a team."
+
+ ### BLOATED_SKILL
+
+ **Trigger:** SKILL.md exceeds 800 lines AND the skill has no `references/` directory.
+
+ **Problem:** A monolithic SKILL.md forces the entire document into context on every invocation,
+ wasting tokens on content only needed in edge cases.
+
+ **Fix:** Create a `references/` directory and move supporting material there:
+ - Detailed rubrics → `references/rubrics.md`
+ - Extended examples → `references/examples.md`
+ - Configuration reference → `references/config.md`
+
+ The SKILL.md should link to these files with `[text](references/filename.md)` so the model
+ can fetch them on demand.
+
+ ### ORPHAN_REFERENCE
+
+ **Trigger:** SKILL.md contains a markdown link `[text](references/filename)` where
+ `filename` does not exist in the `references/` directory.
+
+ **Problem:** Dead links waste tokens on context that will never resolve and confuse the model.
+
+ **Fix:** Either create the missing reference file or remove the dead link.
+
+ ### DEAD_CROSS_REF
+
+ **Trigger:** SKILL.md references another skill or agent by relative path and that path
+ cannot be resolved from the skills/ directory.
+
+ **Problem:** Broken ecosystem links undermine the plugin's coherence score and may cause
+ the model to attempt navigation to non-existent files.
+
+ **Fix:** Verify the referenced skill exists. Update the path or remove the reference.
+
+ ---
+
+ ## Elo Ranking
+
+ PluginEval uses an Elo/Bradley-Terry rating system to rank a skill against the gold corpus.
+
+ **Starting rating:** 1500 (the corpus median by convention).
+
+ **K-factor:** 32 (standard for moderate-stakes ratings).
+
+ **Expected score formula** (standard Elo):
+
+ ```
+ E(A vs B) = 1 / (1 + 10^((B_rating − A_rating) / 400))
+ ```
+
+ **Rating update after each matchup:**
+
+ ```
+ new_rating = old_rating + 32 × (actual_score − expected_score)
+ ```
+
+ where `actual_score` is 1.0 for a win, 0.5 for a draw, 0.0 for a loss.
+
+ **Confidence intervals** on the Elo rating are computed via bootstrap resampling (500
+ samples) of the matchup results, reported as a 95% CI.
+
+ **Corpus percentile** reflects where the skill ranks within the indexed gold corpus.
+ A skill at the 80th percentile beat 80% of corpus entries across pairwise comparisons.
+
+ **Position bias check:** The judge evaluates each pair in both orders (A vs B, then B vs A)
+ and flags disagreements to detect order-dependent bias.
+
+ The `plugin-eval init` command builds the corpus index from a plugins directory:
+
+ ```bash
+ plugin-eval init ./plugins --corpus-dir ~/.plugineval/corpus
+ ```
+
+ ---
+
+ ## CLI Reference
+
+ ### Score a skill (quick static analysis only)
+
+ ```bash
+ plugin-eval score ./path/to/skill --depth quick
+ ```
+
+ Returns Layer 1 results in < 2 seconds. Useful for fast feedback during authoring.
+
+ ### Score with LLM judge (default)
+
+ ```bash
+ plugin-eval score ./path/to/skill
+ ```
+
+ Runs static + LLM judge (standard depth). Takes 30–90 seconds.
+
+ ### Score with full output as JSON
+
+ ```bash
+ plugin-eval score ./path/to/skill --output json
+ ```
+
+ Emits structured JSON including `composite.score`, `composite.dimensions`, and
+ `layers[0].anti_patterns`. Suitable for CI integration:
+
+ ```bash
+ plugin-eval score ./path/to/skill --depth quick --output json --threshold 70
+ # exits with code 1 if score < 70
+ ```
+
+ ### Full certification (all three layers + Elo)
+
+ ```bash
+ plugin-eval certify ./path/to/skill
+ ```
+
+ Runs static + LLM judge + Monte Carlo (50 simulations) + Elo ranking. Takes 15–20 minutes.
+ Assigns a quality badge. Use before publishing a skill to the marketplace.
+
+ ### Head-to-head comparison
+
+ ```bash
+ plugin-eval compare ./skill-a ./skill-b
+ ```
+
+ Evaluates both skills at quick depth and prints a dimension-by-dimension comparison table.
+ Useful for deciding between two implementations or measuring improvement before/after a
+ rewrite.
+
+ ### Initialize corpus for Elo
+
+ ```bash
+ plugin-eval init ./plugins
+ ```
+
+ Builds the local corpus index at `~/.plugineval/corpus`. Required before Elo ranking works.
+
+ ---
+
+ ## Tips for Improving a Skill's Score
+
+ Work through dimensions in weight order. The largest gains come from fixing the top-weighted
+ dimensions first.
+
+ ### Triggering Accuracy (weight 0.25)
+
+ - Include "Use this skill when..." in the description, followed by 3–4 specific contexts.
+ - Add the word "proactively" if the skill should auto-activate without explicit user request.
+ - Test mentally: generate 5 prompts that should trigger the skill and 5 that should not.
+ Does your description discriminate correctly?
+ - Avoid descriptions that only name the skill or describe what it does — they must describe
+ *when* to use it.
+
+ ### Orchestration Fitness (weight 0.20)
+
+ - The SKILL.md should document what the skill *receives* (inputs) and what it *returns*
+ (outputs), not what it orchestrates or coordinates.
+ - Avoid words like "orchestrate", "coordinate", "dispatch", "manage workflow" in SKILL.md.
+ - Include at least one explicit "Output format" section showing what the skill returns.
+ - Provide 2+ code blocks demonstrating concrete worker behavior.
+
+ ### Output Quality (weight 0.15)
+
+ - Give specific, actionable instructions — not just goals.
+ - Cover at least one edge case explicitly (empty input, malformed data, etc.).
+ - Include an examples section showing representative inputs and expected outputs.
+ - The more concrete the instructions, the higher the judge will score this dimension.
+
+ ### Scope Calibration (weight 0.12)
+
+ - Target 200–600 lines for SKILL.md. Below 100 is a stub; above 800 without references/ is bloat.
+ - Every section in SKILL.md should be necessary for a worker executing the skill. Move
+ background reading, extended examples, and reference tables to `references/`.
+ - If the skill is very narrow, consider merging it with a sibling. If it's very broad,
+ consider splitting it.
+
+ ### Progressive Disclosure (weight 0.10)
+
+ - Add a `references/` directory for supporting material. This earns a 0.15–0.25 bonus on
+ the progressive disclosure sub-score.
+ - Keep the SKILL.md itself focused on the execution path — what the worker needs to do.
+ - An `assets/` directory (diagrams, templates) adds another bonus.
+
+ ### Token Efficiency (weight 0.06)
+
+ - Audit MUST/ALWAYS/NEVER count. Target < 1 per 10 lines.
+ - Search for repeated paragraphs or near-duplicate bullet points and consolidate.
+ - If you have tables with the same structure repeated in multiple sections, combine them.
+
+ ### Robustness (weight 0.05)
+
+ - Add a "Troubleshooting" or "Edge Cases" section.
+ - Cover at least 3 failure modes and how the skill should handle them.
+ - Mention what to return or report when the skill cannot complete its task.
+
+ ### Structural Completeness (weight 0.03)
+
+ - Ensure at least 4 headings (H2 or H3) are present.
+ - Include at least 3 code blocks.
+ - Add an explicit "## Examples" section and a "## Troubleshooting" section.
+
+ ### Code Template Quality (weight 0.02)
+
+ - All code blocks should be syntactically valid and copy-paste ready.
+ - Include language tags on fenced code blocks (` ```bash `, ` ```python `, etc.).
+ - Show realistic inputs and outputs, not placeholder pseudocode.
+
+ ### Ecosystem Coherence (weight 0.02)
+
+ - Add a "## Related" or "## See Also" section listing sibling skills or agents.
+ - Use relative paths when cross-referencing: `../other-skill/` not absolute paths.
+ - Avoid duplicating content that already exists in another skill — link to it instead.
+
+ ---
+
+ ## Troubleshooting
+
+ ### "Score is much lower than expected after adding content"
+
+ The anti-pattern penalty compounds. Run with `--output json` and inspect
+ `layers[0].anti_patterns`. If you have 5+ anti-patterns, the multiplier can reduce your
+ score to 75% of its raw value regardless of how good the content is. Fix the flags first.
+
+ ### "triggering_accuracy is low despite a detailed description"
+
+ The `_description_pushiness` scorer looks for specific syntactic patterns, not just length.
+ Verify your description contains the phrase "Use this skill when" or "Use when" (exact
+ phrasing matters — it's a regex match). Also check that you have multiple use cases separated
+ by commas or "or" to earn the specificity bonus.
+
+ ### "LLM judge scores vary significantly between runs"
+
+ This is expected for ambiguous skills. The judge generates 10 mental test prompts
+ non-deterministically. Improve score stability by tightening the description and adding
+ concrete examples. When `judges > 1`, averaged scores will be more stable. Use
+ `--depth deep` with `certify` which runs Monte Carlo to get statistically-bounded scores.
+
+ ### "progressive_disclosure score is low even though the file is the right length"
+
+ Check whether the file is in the 200–600 line sweet spot. Files shorter than 100 lines
+ score only 0.20 on this sub-check. Also confirm that `references/` files are not empty —
+ the scorer checks for non-empty reference files, not just the directory.
+
+ ### "compare shows my rewrite scores lower than the original"
+
+ Quick depth (`--depth quick`) only runs static analysis. If the rewrite moved content to
+ `references/` and shortened SKILL.md significantly, static scores for structural completeness
+ may drop even though overall quality improved. Run `--depth standard` for a fairer comparison
+ that includes the LLM judge's assessment of content quality.
+
+ ---
+
+ ## References
+
+ - [Full Rubric Anchors — all 4 judge dimensions](references/rubrics.md)