coverage-analysis · v1.0 · 2026-05-10 · sha256 3515f1a7de40c884
coverage-analysis v1.0A
Immutable. This exact content never changes and is served at /api/v1/blob/3515f1a7de40c884.
---
name: coverage-analysis
description: >-
Measure and improve test coverage meaningfully. Covers Istanbul/V8/c8/coverage.py
configuration, coverage gap analysis methodology, coverage-as-ratchet in CI (never
let it decrease), PR coverage diff checks, and distinguishing meaningful from vanity
coverage. Use when: "code coverage," "coverage gap," "Istanbul," "coverage threshold,"
"coverage report," "branch coverage."
Related: unit-testing, ci-cd-integration, qa-metrics, ai-qa-review.
license: MIT
metadata:
author: kindlmann
version: "1.0"
category: metrics
---
<objective>
Measure coverage to find gaps, not to chase numbers.
</objective>
---
## Discovery Questions
1. **What coverage tooling is configured?** Check for `jest.config.*` (coverageProvider), `vitest.config.*` (coverage), `.nycrc`, `c8` in scripts, or `[tool.coverage]` in `pyproject.toml`.
2. **What is the current coverage level?** Run the existing coverage command and note line, branch, and function percentages. This is the baseline for the ratchet.
3. **Is coverage gated in CI?** Check GitHub Actions or GitLab CI for `--coverage`, `coverageThreshold`, `fail_under`, or `--cov-fail-under` flags.
4. **What is the target, and who set it?** A target without rationale ("the VP said 80%") leads to gaming. Targets should reflect risk tolerance and codebase maturity.
5. **Check `.agents/qa-project-context.md` first.** Respect existing coverage conventions and thresholds.
---
## Core Principles
**1. Coverage measures breadth, not depth.** A line of code being executed does not mean it is tested correctly. `expect(true).toBe(true)` executes the function but asserts nothing meaningful. Coverage tells you what code ran, not whether the tests would catch a bug.
**2. Branch coverage matters more than line coverage.** A ternary `condition ? a : b` on one line counts as "covered" in line coverage even if only one branch executes. Branch coverage exposes untested paths.
```typescript
// Line coverage: 100% (the line executed)
// Branch coverage: 50% (only the 'true' branch ran)
function discount(price: number, isPremium: boolean): number {
return isPremium ? price * 0.8 : price;
}
// Test only covers isPremium = true
expect(discount(100, true)).toBe(80);
// Missing: expect(discount(100, false)).toBe(100);
```
**3. Ratchet pattern: never decrease, only increase.** Record the current coverage as the minimum threshold. Every PR must meet or exceed it. Coverage goes up over time without forcing artificial targets.
**4. Focus on gaps, not numbers.** A project at 85% coverage is not "better" than one at 75%. What matters is whether the untested 15% or 25% contains critical business logic. Analyze gaps by risk, not by percentage.
**5. New code has a higher bar than legacy code.** Require 90%+ coverage on new code in PRs even if the overall project is at 65%. This prevents coverage decay without demanding a rewrite of legacy code.
---
## Coverage Tools
> **Node baseline:** `c8` v11 and `nyc` v18 (both released 2026-02-22) require **Node 20 || >= 22**. If your project still runs on Node 18, pin `c8@^10` / `nyc@^17` until you can upgrade. New projects should standardize on Node 20+ and the current majors.
### V8 / c8 (Node.js Built-in)
V8's built-in code coverage is faster than Istanbul because it does not instrument source code. Use `c8` as the CLI wrapper.
```bash
npm i -D c8@^11 # Node 20+ baseline
```
```json
// package.json
{
"scripts": {
"test": "vitest run",
"test:coverage": "vitest run --coverage"
}
}
```
```typescript
// vitest.config.ts
import { defineConfig } from "vitest/config";
export default defineConfig({
test: {
coverage: {
provider: "v8",
reporter: ["text", "html", "lcov", "json-summary"],
reportsDirectory: "./coverage",
include: ["src/**/*.ts"],
exclude: [
"src/**/*.test.ts",
"src/**/*.spec.ts",
"src/**/*.d.ts",
"src/**/index.ts", // Barrel exports
"src/**/types.ts", // Type-only files
"src/**/*.stories.ts", // Storybook
"src/generated/**", // Generated code
],
thresholds: {
lines: 80,
branches: 80,
functions: 80,
statements: 80,
},
},
},
});
```
### Istanbul / nyc (JavaScript/TypeScript)
Istanbul instruments source code for coverage tracking. Slower than V8 but more widely compatible.
```bash
npm i -D nyc@^18 # Node 20+ baseline
```
```json
// .nycrc.json
{
"all": true,
"include": ["src/**/*.ts"],
"exclude": [
"src/**/*.test.ts",
"src/**/*.spec.ts",
"src/**/*.d.ts",
"src/**/index.ts",
"src/generated/**"
],
"reporter": ["text", "html", "lcov", "json-summary"],
"report-dir": "./coverage",
"check-coverage": true,
"branches": 80,
"lines": 80,
"functions": 80,
"statements": 80,
"watermarks": {
"lines": [70, 90],
"functions": [70, 90],
"branches": [70, 90],
"statements": [70, 90]
}
}
```
```json
// package.json (with Jest)
{
"scripts": {
"test:coverage": "jest --coverage"
},
"jest": {
"coverageProvider": "v8",
"collectCoverageFrom": [
"src/**/*.ts",
"!src/**/*.{test,spec,d}.ts",
"!src/**/index.ts",
"!src/generated/**"
],
"coverageThreshold": {
"global": {
"branches": 80,
"functions": 80,
"lines": 80,
"statements": 80
}
}
}
}
```
### coverage.py (Python)
```bash
pip install pytest-cov # current coverage.py 7.13.x
```
> coverage.py 7.13.0 added `.coveragerc.toml` as a TOML-first standalone config — prefer it for new Python projects when you want config separated from `pyproject.toml`. 7.12.0 split statements vs branches totals in HTML and JSON reports.
```toml
# pyproject.toml (or .coveragerc.toml in 7.13+)
[tool.coverage.run]
source = ["src"]
branch = true
omit = [
"src/**/test_*.py",
"src/**/conftest.py",
"src/**/__init__.py",
"src/generated/*",
]
[tool.coverage.report]
fail_under = 80
show_missing = true
skip_covered = true
precision = 1
exclude_lines = [
"pragma: no cover",
"if TYPE_CHECKING:",
"if __name__ == .__main__.",
"raise NotImplementedError",
"@overload",
"\\.\\.\\.", # Ellipsis in abstract methods
]
[tool.coverage.html]
directory = "coverage/html"
[tool.coverage.xml]
output = "coverage/coverage.xml"
```
```bash
# Run tests with coverage
pytest --cov=src --cov-report=term-missing --cov-report=html --cov-report=xml
# Fail if coverage drops below threshold
pytest --cov=src --cov-fail-under=80
```
### Coverage Report Types
| Reporter | Output | Use Case |
|----------|--------|----------|
| `text` | Terminal table | Quick local check |
| `html` | Interactive HTML | Detailed local analysis, clicking through files |
| `lcov` | `lcov.info` file | SonarQube, Codecov, Coveralls integration |
| `json-summary` | `coverage-summary.json` | CI scripts, PR comments, dashboard metrics |
| `cobertura` | `cobertura-coverage.xml` | GitLab CI coverage visualization |
---
## Gap Analysis
### Identify Untested Code Paths
Coverage reports show which lines and branches are not executed. But not all gaps are equal. Prioritize by risk.
**Step 1: Generate the coverage report.**
```bash
npm run test:coverage
# Open coverage/index.html in a browser
```
**Step 2: Sort files by uncovered lines.** Parse `coverage-summary.json`, sort files by `(total - covered)` descending, and focus on the top 20. A script that reads the JSON summary and outputs a table of file, line%, branch%, and uncovered count makes this repeatable.
**Step 3: Map gaps to risk areas.**
| Gap Location | Risk Level | Action |
|-------------|-----------|--------|
| Payment processing | Critical | Write tests immediately |
| Auth/permissions | Critical | Write tests immediately |
| Data validation | High | Add to next sprint |
| Error handling paths | High | Add to next sprint |
| Utility functions | Medium | Cover when modifying |
| UI formatting | Low | Skip unless regression-prone |
| Generated code | None | Exclude from coverage |
### Coverage Diff on PRs
Show coverage change per PR so reviewers see the impact of each change.
```yaml
# GitHub Actions: coverage diff comment
- name: Run tests with coverage
run: npm run test:coverage
- name: Coverage Report
uses: davelosert/vitest-coverage-report-action@v2
if: github.event_name == 'pull_request'
with:
json-summary-path: coverage/coverage-summary.json
json-final-path: coverage/coverage-final.json
vite-config-path: vitest.config.ts
```
Alternative: use `marocchino/sticky-pull-request-comment@v2` with a script that reads `coverage-summary.json`, filters to changed files (via `git diff --name-only origin/main...HEAD`), and renders a markdown table showing per-file line and branch coverage.
---
## Coverage as CI Gate
### Threshold Configuration
**Global threshold (minimum for the entire project):**
```typescript
// vitest.config.ts
coverage: {
thresholds: {
lines: 80,
branches: 80,
functions: 80,
statements: 80,
},
}
```
**Per-directory thresholds (stricter for critical code):**
```typescript
// vitest.config.ts
coverage: {
thresholds: {
// Global baseline
lines: 75,
branches: 75,
// Stricter for critical paths
"src/payments/**": { lines: 95, branches: 90 },
"src/auth/**": { lines: 90, branches: 85 },
"src/utils/**": { lines: 80, branches: 80 },
},
}
```
**Jest per-file thresholds:**
```javascript
// jest.config.js
module.exports = {
coverageThreshold: {
global: { branches: 75, functions: 75, lines: 75, statements: 75 },
"./src/payments/": { branches: 90, functions: 90, lines: 95, statements: 95 },
"./src/auth/": { branches: 85, functions: 85, lines: 90, statements: 90 },
},
};
```
### Ratchet Pattern
Never let coverage decrease. Record the current level as the minimum and update it upward when coverage improves.
```typescript
// scripts/coverage-ratchet.ts
import fs from "fs";
import coverageSummary from "../coverage/coverage-summary.json";
const RATCHET_FILE = ".coverage-ratchet.json";
interface Ratchet {
lines: number;
branches: number;
functions: number;
statements: number;
updatedAt: string;
}
// Read current ratchet or initialize
let ratchet: Ratchet;
try {
ratchet = JSON.parse(fs.readFileSync(RATCHET_FILE, "utf-8"));
} catch {
ratchet = { lines: 0, branches: 0, functions: 0, statements: 0, updatedAt: "" };
}
const current = (coverageSummary as any).total;
const metrics = ["lines", "branches", "functions", "statements"] as const;
let failed = false;
for (const metric of metrics) {
const currentPct = current[metric].pct;
const ratchetPct = ratchet[metric];
if (currentPct < ratchetPct) {
console.error(
`FAIL: ${metric} coverage dropped from ${ratchetPct}% to ${currentPct}% (delta: ${(currentPct - ratchetPct).toFixed(1)}%)`
);
failed = true;
} else if (currentPct > ratchetPct) {
console.log(`IMPROVED: ${metric} coverage increased from ${ratchetPct}% to ${currentPct}%`);
ratchet[metric] = Math.floor(currentPct); // Floor to avoid ratcheting on decimals
} else {
console.log(`STABLE: ${metric} coverage at ${currentPct}%`);
}
}
if (failed) {
console.error("\nCoverage ratchet failed. Coverage must not decrease.");
console.error("If this is intentional (e.g., removing dead code), update .coverage-ratchet.json manually.");
process.exit(1);
}
// Update ratchet file
ratchet.updatedAt = new Date().toISOString();
fs.writeFileSync(RATCHET_FILE, JSON.stringify(ratchet, null, 2) + "\n");
console.log(`\nRatchet updated: ${JSON.stringify(ratchet)}`);
```
Commit `.coverage-ratchet.json` to the repo (e.g., `{ "lines": 82, "branches": 78, ... }`). In CI, run the ratchet script after tests. On main branch merges, auto-commit the updated ratchet file if coverage improved.
### PR Diff Coverage Gate
Require that new code in a PR meets a higher threshold (e.g., 90%) than the project baseline. In CI, use `git diff --numstat origin/main...HEAD` to identify changed files, then check their coverage from `coverage-summary.json`. Fail the pipeline if the average coverage of changed files falls below the threshold. This prevents coverage decay without demanding a rewrite of legacy code.
**Hosted alternatives:** **Codecov**, **Coveralls**, and **Trunk Coverage** all ship first-class differential PR coverage with merge-blocking gates and inline annotations. Most teams prefer these over hand-rolled diff scripts — pick one if you don't already have a coverage host. Codecov + GitHub: `codecov/codecov-action@v5` reads `lcov.info` and posts a PR comment with diff coverage automatically.
---
## Mutation Testing
Mutation testing measures *assertion quality*, not just code execution. With Stryker JS v9.6+ and Vitest 4.1+, the cost is low enough to make mutation score realistic on PR-changed files.
### Stryker (JS/TS)
```json
// stryker.config.json
{
"$schema": "./node_modules/@stryker-mutator/core/schema/stryker-schema.json",
"testRunner": "vitest",
"coverageAnalysis": "perTest",
"incremental": true,
"incrementalFile": ".stryker-tmp/incremental.json",
"mutate": ["src/**/*.ts"],
"thresholds": { "high": 80, "low": 60, "break": 50 }
}
```
Run on PR-changed files only with `--mutate '$(git diff --name-only origin/main...HEAD | grep "src/.*\.ts")'`. Pair `incremental: true` with the JSON cache so consecutive runs only re-mutate touched files.
### mutmut (Python)
```bash
mutmut run --paths-to-mutate=src/
mutmut results
```
mutmut 3.x is a from-scratch rewrite (still active 2026); pin via `pip install mutmut==3.*`.
### Targeting
Mutation testing is expensive on whole codebases. Restrict to:
- Pure business logic (validators, calculators, transformers)
- Critical paths (payment, auth, data integrity)
- Code with low confidence in test quality (look at branch-coverage > 90% but few assertion variants)
Skip on UI rendering code, glue code, and generated code.
### Reading the score
A mutation score of 80% means 80% of mutated bugs were caught by your tests. Lower than your coverage % is normal — many mutations land in untested branches the coverage report already flagged. The interesting gap is **high coverage + low mutation score**: code is executed but assertions don't constrain it.
---
## Meaningful vs Vanity Coverage
### Why 100% Coverage Is Usually Wrong
100% coverage requires testing every branch of every line, including:
- Error handling for impossible states
- Default cases in exhaustive switches
- Framework lifecycle methods that are never called directly
- Defensive checks against corrupted data
Tests written to hit 100% are often trivial, brittle, and do not catch real bugs.
### Diminishing Returns
| Coverage Range | Value | Effort |
|---------------|-------|--------|
| 0% to 60% | High -- covers main paths, catches obvious regressions | Low |
| 60% to 80% | Medium -- covers error paths, edge cases | Medium |
| 80% to 90% | Lower -- covers unusual combinations, defensive code | High |
| 90% to 100% | Minimal -- covers unreachable code, framework internals | Very high |
The sweet spot is 75-85% for most projects. Critical paths (payments, auth) should aim higher (90%+).
### What NOT to Cover
Exclude these from coverage calculations. They inflate the denominator without adding value.
```typescript
// vitest.config.ts or jest.config.js -- exclude patterns
exclude: [
"**/*.d.ts", // Type definitions
"**/index.ts", // Barrel exports (re-exports only)
"**/*.stories.{ts,tsx}", // Storybook stories
"**/generated/**", // Auto-generated code (GraphQL, Prisma)
"**/migrations/**", // Database migrations
"**/__mocks__/**", // Test mocks
"**/types/**", // Type-only modules
]
```
### Quality Indicators Beyond Percentage
Coverage percentage alone is insufficient. Combine it with:
| Indicator | What It Measures | How to Get It |
|-----------|-----------------|---------------|
| **Mutation score** | Would tests catch a real bug? | Stryker / mutmut |
| **Branch coverage** | Are all conditional paths tested? | V8/Istanbul with branch reporting |
| **Critical path coverage** | Are payment/auth/data flows fully covered? | Per-directory thresholds |
| **Defect escape rate** | Do production bugs occur in tested code? | Post-incident analysis |
| **Coverage delta** | Is coverage improving or declining? | Ratchet pattern tracking |
---
## Anti-Patterns
**Treating coverage as proof of quality.** "We have 90% coverage so we are well-tested" is a dangerous statement. Coverage says code executed, not that behavior was verified. A test with no assertions contributes to coverage but catches zero bugs.
**Excluding files to inflate numbers.** Adding files to the exclude list because they are hard to test (e.g., error handlers, integration modules) hides the most important gaps. Only exclude genuinely untestable code: generated files, type definitions, and barrel exports.
**Writing trivial tests to hit targets.** Tests like `it("should exist", () => { expect(MyClass).toBeDefined(); })` add coverage without value. Every test should verify a behavior that, if broken, would affect users.
**Global threshold without per-module analysis.** An 80% global threshold can pass even if the payment module is at 30% coverage -- as long as utility functions inflate the average. Use per-directory thresholds for critical paths.
**Coverage threshold set once, never adjusted.** The ratchet should increase over time. A team that has been at 78% for 6 months is not improving. Review the ratchet quarterly and investigate why it stagnates.
**Ignoring branch coverage.** Line coverage reports 100% on `const result = condition ? a : b` even if only one branch runs. Always report and gate on branch coverage alongside line coverage. Branch coverage is the more honest metric.
**Coverage in E2E tests only.** E2E tests execute many lines but cover broad, shallow paths. A single E2E test might touch 60% of the codebase without testing any edge case. Unit tests provide targeted, deep coverage of logic branches. Measure coverage from unit and integration tests separately from E2E.
---
## Done When
- Coverage is instrumented and running automatically in CI on every push (no manual steps to generate the report).
- Coverage threshold is enforced in CI configuration (build fails when line or branch coverage drops below the defined minimum).
- Coverage report is published as a CI artifact (HTML report and `json-summary`) and linked from PR comments showing per-PR coverage delta.
- Meaningful vs. vanity coverage distinction is documented for the codebase: excluded paths are listed and justified (generated code, barrel exports, type definitions), and per-directory thresholds are set higher for critical paths such as payments and auth.
- Coverage-as-ratchet is configured so the threshold only goes up: `.coverage-ratchet.json` is committed, the ratchet script runs in CI, and the build fails if coverage regresses from the recorded baseline.
## Related Skills
- **unit-testing** -- Test writing patterns, mocking strategies, framework-specific coverage configuration, and Vitest 4 `coverage.changed` for changed-files-only coverage in CI.
- **ci-cd-integration** -- Pipeline configuration for coverage gates, artifact storage, and PR comments.
- **qa-metrics** -- Quality KPIs including coverage trends, mutation scores, defect escape rates, and Test Impact Analysis as a coverage-driven CI lever.
- **ai-qa-review** -- AI-assisted identification of coverage gaps and undertested code paths; Vitest browser mode for component coverage parity.