git:20260615.b734c35 to git:20260616.b2b0cde

---

name: benchmark-archaeology

description: Evaluation Methodology Archaeology Campaign — 5 strategies for systematic

analysis of AI/ML benchmarks, metrics, and leaderboards. Reveals construct validity

issues, saturation, data contamination, and evaluation protocol inconsistencies.

execution: campaign

dependencies:

strategies:

- benchmark-audit

- coverage-mapping

- protocol-forensics

- saturation-analysis

- validity-probing

sops:

- context-checkpoint

- context-init

---

# Benchmark Archaeology

Systematic excavation and critical analysis of AI/ML evaluation methodology. Treats benchmarks as historical artifacts requiring forensic examination — uncovering hidden assumptions, methodological drift, validity decay, and coverage gaps that accumulate over time.

## Strategy Routing

| Signal | Route To |

|--------|----------|

| "audit this benchmark", "benchmark quality", "BetterBench" | benchmark-audit |

| "saturation", "ceiling", "score plateau", "when will X be solved" | saturation-analysis |

| "does it actually measure", "construct validity", "what does score mean" | validity-probing |

| "what's not tested", "coverage gaps", "missing capabilities" | coverage-mapping |

| "different papers get different scores", "protocol differences" | protocol-forensics |

## Manifest

### Strategies (5)

| Strategy | Purpose |

|----------|---------|

| benchmark-audit | Systematic quality assessment using BetterBench 46-criterion framework |

| saturation-analysis | Track score trajectories, detect saturation and failure points |

| validity-probing | Challenge construct validity — does benchmark measure claimed capability? |

| coverage-mapping | Map evaluation coverage, identify untested capability dimensions |

| protocol-forensics | Analyze evaluation protocol differences across papers for same benchmark |

### Tactics (3)

| Tactic | Purpose |

|--------|---------|

| score-trajectory-analysis | Collect historical scores, fit saturation curves, detect inflection points |

| artifact-detection | Detect annotation artifacts and shortcuts in benchmarks |

| evaluation-protocol-comparison | Compare implementation differences of same benchmark across papers |

### Subagent SOPs (9 + 1 shared)

| SOP | Purpose |

|-----|---------|

| benchmark-inventory | Identify and catalog all relevant benchmarks in target domain |

| metric-decomposition | Decompose composite metrics into constituent signals |

| contamination-audit | Detect train-test data leakage and memorization artifacts |

| construct-validity-assessment | Evaluate whether benchmark measures its claimed capability |

| documentation-audit | Assess documentation completeness against BetterBench/Datasheets standards |

| capability-taxonomy-mapping | Build capability taxonomy, map existing benchmark coverage |

| leaderboard-dynamics-analysis | Analyze leaderboard score distributions, compression, selective reporting |

| protocol-element-extraction | Extract evaluation protocol parameters from papers |

| benchmark-synthesis | Produce final structured audit report |

| saturation-detection (shared) | Detect saturation signals in score trajectories (from literature-survey) |

## Budget Table

| Strategy | Benchmarks | Papers | Web Searches |

|----------|-----------|--------|--------------|

| benchmark-audit | 5 | 30 | 40 |

| saturation-analysis | 15 | 50 | 60 |

| validity-probing | 3 | 40 | 30 |

| coverage-mapping | 20 | 30 | 50 |

| protocol-forensics | 5 | 60 | 30 |

| **Total** | **48** | **210** | **210** |

## MCP Tools

| MCP Server | Tools |

|------------|-------|

| brave-search | brave_web_search, brave_llm_context |

| apify | rag-web-browser, google-scholar-scraper |

| alphaxiv | get_paper_content, answer_pdf_queries |

| semantic-scholar | ss_paper, ss_relevance_search, ss_citations, ss_references |

## Context Management

All outputs write to `context/benchmark-archaeology/`:

```

context/benchmark-archaeology/

audit/ # benchmark-audit outputs

saturation/ # saturation-analysis outputs

validity/ # validity-probing outputs

coverage/ # coverage-mapping outputs

forensics/ # protocol-forensics outputs

synthesis/ # Final cross-strategy synthesis

```

Each strategy maintains its own state ledger within its output directory.

<!-- BEGIN available-tables (generated) -->

## Available Strategies

| --- | --- |

| benchmark-audit | Systematic quality assessment using BetterBench 46-criterion framework — 5 benchmarks, 30 papers, 40 web searches |

| coverage-mapping | Map evaluation coverage, identify untested capability dimensions — 20 benchmarks, 30 papers, 50 web searches |

| protocol-forensics | Analyze evaluation protocol differences across papers for same benchmark — 5 benchmarks, 60 papers, 30 web searches |

| saturation-analysis | Track score trajectories, detect saturation/failure points — 15 benchmarks, 50 papers, 60 web searches |

| validity-probing | Challenge construct validity — does benchmark measure claimed capability? — 3 benchmarks, 40 papers, 30 web searches |

## Available SOPs

| --- | --- |

| context-checkpoint | Append research process and results to the current Phase's context file. Each append MUST contain >=500 lines of markdown covering both process and results. Use this skill at plan-designated checkpoint points — typically after each strategy completes or at key decision nodes within a research Phase. |

| context-init | Create a new context file for a research Phase. Called once at Phase start to initialize the file that subsequent context-checkpoint calls will append to. Use this skill whenever a new research Phase begins and a fresh context file is needed. |

<!-- END available-tables (generated) -->