git:20260614.c3aa45a to git:20260614.6cb90a2

---

name: benchmark-verify

description: "Normalized, MANDATORY end-to-end verification for any benchmark IC after it has been driven from Design Documents → generated RTL → silicon through the full Vibe-IC flow. Produces ONE BENCHMARK_VERIFICATION_REPORT.md per IC covering all six pillars with hard gates: (1) Functional Verification Coverage == 100% (closed-loop until met), (2) 56-step Output Comparison vs the open-source reference (every applicable step PASS), (3) Code Coverage line >= 90%, (4) FPGA digital verification (test patterns on-board/BFM), (5) Analog closed-loop verification (or N/A for pure-digital), (6) Design-for-ECO readiness — spare-cell coverage PASS + spare preservation intact (or N/A if the IC has no place-and-route). Use when: 'verify the benchmark', 'benchmark verification report', 'is this IC production-ready', 'cross-check vs open source', after /vibe-ic-all on a benchmark IC, or for any IC in benchmark_clean/."

---

# Benchmark Verify — normalized doc→production-ready verification

We run benchmark ICs to validate the **whole Vibe-IC flow**: that starting from a

complete set of **Design Documents** we can reach a **production-ready** result.

A benchmark run is **not finished** when the flow returns a verdict — it is finished

when this skill's five pillars all pass and the single report is written.

This skill is **chip-AGNOSTIC** and **MANDATORY** for every benchmark IC. Apply it

to each IC under `benchmark_clean/<ic>/` (and any future benchmark set).

## ⛔ Prerequisite — honesty + the corrected protocol

Read `benchmark_clean/METHODOLOGY.md` first. Non-negotiable:

- **Input = Design Documents ONLY** (incl. explicit PDK); nothing produced by a later

phase is fed back as input.

- **Phase-2 RTL must be GENERATED**; IP reuse is allowed but **every reused source MUST be

tagged `REUSED-IP` in `SOURCE_MANIFEST.md`** (vs `GENERATED`). Production-readiness credit

applies only to GENERATED content; reused content is reported separately.

- **No vacuous result counts as PASS** (e.g. a Magic `gds read` that dropped geometry and

reports "0 DRC" is INCONCLUSIVE, not clean). A missing input is **PENDING**, never a silent PASS.

*Enforced by `programs/drc_vacuous_pass_check.py`* — flips a 0-violation DRC verdict to

INCONCLUSIVE unless the log proves geometry was loaded; SKIPs (never PASSes) when no DRC log exists.

- The upstream open-source IP is the **golden oracle for cross-check only** — never a Phase-1/2 input.

## The five pillars (== report sections == hard gates)

### Pillar 1 — Functional Verification Coverage ▸ gate: **== 100%**

**LLM judgment (irreducible):** walk L1–L23 (functional spec, interface, register map,

command/protocol, timing, behavioral sequences, test cases) and enumerate **every requirement** —

a program cannot reliably know that a prose timing sentence in L8 is a distinct *testable*

requirement (vs a restatement) nor author the directed test that covers it. For each requirement,

bind a verification item (directed test, assertion, golden-vector check, or formal property) and

confirm it PASSES against the generated RTL.

- `Functional Coverage = verified_requirements / total_requirements` → **must be 100%**

(*the == 100% gate is enforced by `programs/benchmark_verify_report.py`*).

- Emit `reports/functional_coverage.json`: `{"requirements":[{"id","source","desc","status":"PASS|FAIL|PENDING"}]}`.

- **Closed-loop:** if < 100%, write the missing tests and/or fix the RTL (use `rtl-repair`)

and re-verify until 100%. Do not waive a requirement; if a doc requirement is genuinely

untestable, that is a spec defect to record, not a pass.

> **Spec-first coverage attribution (ORGANIC #697, BINDING — program-first):** before claiming

> Pillar-1 PASS, run `programs/spec_coverage_check.py` to make our self-verification as complete as

> the hidden scorer. Both derive from the SAME spec — where "spec" is the WHOLE input chain

> (prompt → fact graph → L1-L23):

> ```bash

> python3 programs/spec_coverage_check.py --prompt input/prompt.md \

> [--fact-graph input/fact_graph.json] --ldocs generated_docs/ \

> --rtl <rtl> --tb <self_tb> --strict --json reports/gates/spec_coverage.json

> ```

> Every spec-derived checklist item (ports, reset, latency, table rows, worked examples, **every

> ENUMERATED SET + its outside-the-set/default boundary**, signed-ness, byte order, overflow,

> handshake) must be COVERED by the self-TB. A `--strict` BLOCK means our TB is weaker than the

> hidden one — close the gap (write the missing assertion) before PASS. On any FAIL, run with

> `--failure "<behavior>"`: a `coverage-gap` ⇒ enhance our TB; an `extraction-gap` ⇒ the program

> names the `route_to:` station (pm-agent / ic-expert-agent / spec-to-rtl) that dropped the

> requirement; only a `spec-absent` (nowhere in the chain, with the searched stations cited) is a

> genuine floor and never a Pillar-1 fail. (The structural extraction + per-station routing is

> deterministic; deciding whether a prose sentence is a distinct *testable* requirement is the LLM

> step above.)

### Pillar 2 — 56-step Output Comparison vs open-source reference ▸ gate: **every applicable step PASS**

For each of the 56 canonical flow steps (`flow/phase1_phase2_phase3.yaml`, which now includes

the Design-for-ECO spare-cell insertion step), compare OUR step output against the open-source

reference's corresponding output. Comparison is **step-appropriate, not byte-identical**.

*The step→class→comparison-method dispatch table and the legal verdict-token enum

(MATCH / EQUIVALENT / IN-RANGE / BOTH-CLEAN / DIFFERENT-BUT-OK / N/A / GAP / NO-TOOL / FAIL) and

which tokens count as PASS are enforced by `programs/benchmark_verify_report.py`* (its

`STEP_METHOD`, `VERDICT_TOKENS`, `PASS_TOKENS`). The step classes are:

**equivalence** (RTL/sim/formal/LEC/post-layout — LEC/co-sim/proof), **metric** (synth/SDC/DEF/

CTS/STA/SPEF/power — magnitude in range), **clean** (lint/CDC/DRC/LVS/IR/EM/antenna/SI — both

independently clean), **layout-endpoint** (GDS — both DRC/LVS/STA-clean + functional-equiv), **N/A**

(analog A1–A9 / mixed-signal M1–M4 for a pure-digital IC; manufacturing for no-silicon).

**LLM judgment (irreducible):** for a *metric* step, deciding whether two magnitudes are in a

*sensible* range (vs a hard threshold) requires understanding WHY they diverge; for a

*layout-endpoint* step, deciding that a different micro-architecture (our carry-save vs the ref's

array) is a *benign* divergence rather than a failure is design judgment, not a fixed compare.

Produce one `cross_check/**/step_<id>.md` per step (what ran, OUR result, REF result, verdict

token from the enum above).

**Close runnable GAPs** (e.g. run SPEF/IR/EM/antenna/power if the flow skipped them). Genuine

NO-TOOL items (e.g. open-source full-chip fast-SPICE) are recorded honestly — and noted if the

reference shares the same limitation. No step may be left unresolved.

### Pillar 3 — Test Cases & Code Coverage ▸ gate: **line coverage >= 90%**

Author unit-test-style test cases (cocotb / SV testbench, via `eda_cocotb` / `eda_simulate`)

for the generated RTL. *The coverage measurement + the >= 90% line-coverage gate are enforced by

`programs/coverage_metric_check.py` / `coverage_closure.py` / `verilator_coverage_measure.py`.*

- **Line coverage >= 90%** (report branch + toggle too). Emit `reports/code_coverage.json`:

`{"line_pct","branch_pct","toggle_pct"}`.

- **Closed-loop:** below 90% → add test cases targeting the uncovered lines until >= 90%.

### Pillar 4 — FPGA digital verification ▸ gate: **PASS** (or documented N/A)

Generate **test patterns** and run the digital portion on FPGA (`fpga_test_harness_gen.py`,

`eda_fpga_compile`, `eda_fpga_program`, on-board or BFM). Record `reports/hw_test.json`

`{"verdict":"PASS|FAIL","patterns":N}`. A core with no board-pin contract may run a BFM/

gate-level equivalent — state which. N/A only with explicit justification.

### Pillar 5 — Analog verification ▸ gate: **converged** (or N/A for pure-digital)

If the IC has analog blocks (`analog/analog_block_list.json` or `phase3/analog/*`), run the

analog **closed-loop**: `analog-sizing-loop` → corner sweep (`ams-sim` / `eda_spice_corner`) →

post-layout resim (`analog-extraction-resim`) → per-block PV (`analog-output-verify`) →

HIL if a bench is available (`analog-hw-tuning-loop`). Pure-digital ICs mark this N/A.

### Pillar 6 — Design-for-ECO readiness ▸ gate: **spare-cell coverage PASS + spares preserved** (or N/A)

A production IC must be ready for a cheap **metal-only ECO** instead of a full base-layer

respin. That requires a distributed, tied-off pool of spare std cells/gates

(inverter/nand2/nor2/dff/mux2/aoi/oai) + reserved ECO pads — all `dont_touch`/`keep`, placed

after placement and before CTS at ~1-5% density. The methodology is the **`design-for-eco`**

skill. This pillar reads the two deterministic checker outputs:

- **Coverage / readiness** — `reports/spare_cell_coverage.json` from

`spare_cell_coverage_check.py`: density / distribution / tie-off targets met → `status: PASS`.

- **Preservation** — `reports/spare_preservation.json` from `spare_cell_preservation_check.py`:

NO spare/ECO cell/gate/pad was optimized away by any later step → `all_keep_attr_intact: true`

and `removed: 0`. (Every optimization skill — synth-doctor, rtl-repair, hold-fix, eco-plan,

ppa-predict, drc-fix — carries the hard rule that it must never strip a keep-marked spare.)

**Applicability:** this gate applies to any **digital place-and-route IC** (a DEF/GDS exists

under `phase3/`). It is **N/A** only for an IC that genuinely never reached place-and-route

(e.g. analog-only). A **missing spare report is PENDING**, never a silent PASS.

**Closed-loop:** if PENDING/FAIL, run the stage3 Design-for-ECO step + its two checkers (or

restore any spare a prior optimization dropped) until coverage PASS + preservation intact.

## Procedure (run for each benchmark IC)

1. Confirm the IC reached the flow end (RTL generated + phase3 attempted). Read `SOURCE_MANIFEST.md`.

2. **Pillar 2:** dispatch the per-step cross-check (split phase1/2 and phase3 to keep tool

runs bounded), writing `cross_check/<half>/step_*.md`. Run REAL tools in `iic-eda`; close gaps.

3. **Pillar 1:** build `reports/functional_coverage.json` from the L-doc requirements; close-loop to 100%.

4. **Pillar 3:** author tests, measure coverage → `reports/code_coverage.json`; close-loop to >= 90%.

5. **Pillar 4:** test patterns + FPGA/BFM run → `reports/hw_test.json`.

6. **Pillar 5:** analog closed-loop if applicable.

6b. **Pillar 6 (digital PnR ICs):** confirm the stage3 Design-for-ECO step ran, then run the two

checkers → `reports/spare_cell_coverage.json` (readiness PASS) + `reports/spare_preservation.json`

(`all_keep_attr_intact` + `removed:0`). N/A only if the IC never reached place-and-route.

7. **Aggregate + gate** — generate the single report:

```bash

python3 ${CLAUDE_PLUGIN_ROOT}/programs/benchmark_verify_report.py <project_dir> \

--ref <open_source_reference_dir>

```

It emits `<project>/BENCHMARK_VERIFICATION_REPORT.md` with the 6-pillar gate summary +

the full 56-step comparison table, and exits non-zero until ALL gates pass.

8. **Close the loop** on any failing/pending gate, then re-aggregate. The benchmark IC is

**DONE only when the report's OVERALL is PRODUCTION-READY** (functional 100% · 56/56 applicable

PASS · code line >= 90% · FPGA PASS · analog converged-or-N/A · Design-for-ECO ready-or-N/A) —

with honest, evidence-backed results and source provenance tagged.

## Acceptance

A benchmark IC passes `benchmark-verify` iff its `BENCHMARK_VERIFICATION_REPORT.md` shows:

- Functional Coverage **100%**, and

- Every applicable step of the 56 = PASS (no GAP/PENDING/FAIL/unexplained NO-TOOL), and

- Code line coverage **>= 90%**, and

- FPGA digital verification **PASS** (or justified N/A), and

- Analog **converged** (or N/A), and

- **Design-for-ECO ready** — `reports/spare_cell_coverage.json` readiness PASS **and**

`reports/spare_preservation.json` `all_keep_attr_intact` + `removed:0` (or N/A if the IC has

no place-and-route); verified by `spare_cell_coverage_check.py` + `spare_cell_preservation_check.py`,

per the `design-for-eco` skill, and

- `SOURCE_MANIFEST.md` present with GENERATED/REUSED-IP tagged.

Anything short of that is **NOT complete** — keep working (closed-loop), do not claim done.

> **Plugin-test hard rule.** If the verification run modified any plugin code (e.g. a chip-agnostic fix to `benchmark_verify_report.py` or a checker), re-run the FULL suite before claiming done: `cd <plugin> && pytest -q` (pytest.ini collects BOTH `programs/tests/` AND `tests/`). Never validate with only `programs/tests/` — the integration/regression gates (INDEX freshness, every-skill-has-compliance, orchestrator branch tests) live in `tests/`. See `_shared/TESTING_STRATEGY.md`.

> *Enforced by `programs/plugin_change_pytest_gate.py`* — detects changed plugin `*.py` (git or an explicit `--changed-files` list) and FAILs DONE unless `--pytest-log` attests a clean run that collected BOTH test trees; ERRORs (never silent-PASSes) when the change state can't be determined.

## Worked example

`benchmark_clean/spm/` — first IC verified under this skill: RTL 100% GENERATED, functional

equivalence vs golden + upstream (10,013 vectors), 56-step cross-check under

`benchmark_clean/spm/cross_check/`, signed-off DRC/LVS/STA. Use it as the reference shape for

a complete report.

## Compliance gate (mandatory)

After producing your output, save it to a file and run:

```bash

python3 plugins/vibe-ic/_shared/skill_compliance_check.py \

--requirements plugins/vibe-ic/skills/benchmark-verify/compliance.yaml \

<your_output_file>

```

Exit 0 = PASS, exit 1 = FAIL with specific missing elements listed.

`compliance.yaml` in the corresponding skill directory enumerates

every required element of your output: section headers, metadata fields,

handoff lines, tool invocations.

**Your task is not complete until the audit returns PASS.** Missing

elements are the single largest source of skill-execution non-determinism

across different agents.