git:20260720.22d020c to git:20260724.d4db756

---

name: ai-spec-coverage-mapper

description: "Build-an-X workflow that uses an LLM to map existing tests to spec sections - given a spec doc + the test suite, the LLM identifies which tests cover which sections, surfaces uncovered sections (gap), and recommends specific tests to add. Output is a coverage matrix per spec ID. Scope is mapping tests that already exist and naming the gaps, not authoring tests for new acceptance criteria. Use when a spec doc and a test suite both exist but nobody can say which requirements are actually covered - before a release sign-off, an audit, or a decision about where to spend the next round of test effort."

---

# ai-spec-coverage-mapper

## Overview

Coverage tools (per `lcov-analysis` in the qa-test-reporting plugin)

report which **lines** are tested. They don't report which **spec

sections** are tested.

A spec section like "AC-1.4: Already-applied promo shows 'Already

applied'" might be:

- Covered by a test (good).

- Partially covered (the AC says "Already applied"; the test

asserts "Already used" - semantic drift).

- Not covered.

This skill uses an LLM to map tests ↔ spec sections semantically.

## When to use

- The team has a spec doc + a test suite and wants spec-coverage

visibility.

- An audit / compliance review needs spec-test traceability.

- After AC drift - tests written months ago for ACs that have

since changed.

## Step 1 - Inputs

```yaml

spec_path: "docs/specs/checkout.md"

test_globs:

- "tests/checkout/**/*.spec.ts"

- "features/checkout/*.feature"

ac_extraction:

pattern: "AC-(\\d+\\.\\d+):" # AC IDs in the spec

```

The LLM reads both the spec and the tests; outputs the mapping.

## Step 2 - Run the mapper

```python

# scripts/ai-coverage.py

import openai

spec_text = read(spec_path)

ac_list = extract_acs(spec_text)

test_files = read_all(test_globs)

system_prompt = """

You map AC IDs to tests. For each AC ID, identify:

- which test files cover it

- which test names within those files cover it

- coverage tier: full | partial | none

If partial, explain what aspect is missing.

"""

response = openai.chat.completions.create(

model='gpt-4',

messages=[

{'role': 'system', 'content': system_prompt},

{'role': 'user', 'content': f"Spec:\n{spec_text}\n\nTests:\n{test_files}"},

],

)

print(response.choices[0].message.content)

```

## Step 3 - Output

```markdown

## Spec → test coverage map

**Spec:** `docs/specs/checkout.md`

**ACs:** 12

**Tests inventoried:** 47

### Coverage matrix

| AC ID | Description | Coverage | Tests |

|----------|------------------------------------------|----------|-----------------------------------------------------|

| AC-1.1 | Valid promo applies discount | ✅ full | `promo.spec.ts > "applies WELCOME10"` |

| AC-1.2 | Expired promo shows error | ✅ full | `promo.spec.ts > "shows error for EXPIRED50"` |

| AC-1.3 | Invalid format shows "Code not found" | ✅ full | `promo.spec.ts > "rejects NOTREAL"` |

| AC-1.4 | Already-applied promo shows "Already applied" | ⚠ partial | `promo.spec.ts > "rejects duplicate"`. ⚠ Test asserts "Already used" - message drift from AC. |

| AC-1.5 | Promo applies before tax | ❌ none | |

| AC-2.1 | Stripe webhook delivery retried | ✅ full | `webhook.spec.ts > "retries on 500"` |

| AC-2.2 | Stripe webhook delivery DLQ after 3 fails | ❌ none | |

| ...

### Action items

| AC ID | Action |

|---------|-------------------------------------------------|

| AC-1.4 | Update test message assertion to match AC ("Already applied"). |

| AC-1.5 | Add test asserting promo applies before tax. |

| AC-2.2 | Add test for DLQ-after-3-fails behavior. |

### Coverage trend

(Compare with prior run)

- AC count: 12 (was 10)

- Full coverage: 9/12 = 75% (was 8/10 = 80%)

- 2 new ACs added in this PR; both uncovered.

```

## Step 4 - Continuous use

Schedule weekly:

```yaml

on:

schedule:

- cron: '0 4 * * MON'

jobs:

spec-coverage:

runs-on: ubuntu-latest

steps:

- uses: actions/checkout@v5

- run: python scripts/ai-coverage.py

- uses: peter-evans/create-issue-from-file@v5

with:

title: 'Spec coverage report - week of ${{ github.event.repository.updated_at }}'

content-filepath: spec-coverage-report.md

```

## Step 5 - Confidence + LLM hallucination

LLMs may claim a test "covers" an AC when it doesn't. Verification:

- Spot-check the highest-priority ACs manually.

- Cross-reference with `acceptance-test-from-criteria` (in the qa-bdd plugin)

if the team uses `@AC-X.Y` tags - those are the ground truth.

- Compare LLM's claim vs. test code via human review.

## Anti-patterns

| Anti-pattern | Why it fails | Fix |

|-----------------------------------------------------------------------|---------------------------------------------------------------------------|-----|

| Trusting LLM's "full coverage" claim without verification | Hallucination; tests don't actually cover the AC. | Spot-check + cross-reference (Step 5). |

| Running on the entire codebase repeatedly | Cost + slow. | Filter to changed ACs / tests since last run. |

| One-shot mapping; never updated | Drift; mapping stale. | Weekly cadence (Step 4). |

| No action items per gap | Coverage gaps surface but nothing happens. | Per-gap action item (Step 3 example). |

## Limitations

- **LLM quality varies.** Same caveats as

- **Spec format matters.** Well-structured ACs (numbered, clearly

scoped) map cleanly; vague specs produce vague mappings.

- **No code-execution verification.** The LLM reads code; doesn't

run it. A test that imports correctly but throws at runtime

may be classified "covered."

- **Test naming influences mapping quality.** Per-test names like

"test_1" don't help the LLM identify what's covered.

## References

skill: generates tests for the gaps this skill identifies.

- `acceptance-test-from-criteria` (in the qa-bdd plugin) - for tag-based AC traceability without LLM.

- `coverage-debt-tracker` (in the qa-test-impact-analysis plugin) - line-coverage debt; complementary to spec coverage.