cve-hunt-benchmark-harness git:20260913.e13b6b8A

Immutable. This exact content is served forever at /api/v1/blob/d1d19dae2b47ace2.

---
name: cve-hunt-benchmark-harness
description: "Ground-truth vulnerability-hunting benchmarks: challenge definitions with known-vulnerable targets, ground-truth vulnerability records with match keywords and point values, precision/recall/F1 scoring of discovered findings against ground truth, time-to-finding metrics, decoy clean samples to measure false-positive rates, and severity breakdown reporting. Use when evaluating how well an agent or tool finds real vulnerabilities, building a vulnerability-hunting eval set, or scoring a security-assessment run against known answers. Derived from the T3MP3ST platform's benchmark framework (Apache-2.0)."
category: security
triggers:
  - "vulnerability benchmark"
  - "cve hunt"
  - "ground truth eval"
  - "precision recall security"
  - "hunting evaluation"
  - "false positive rate"
tools:
  - file_read
  - file_write
  - file_grep
  - file_glob
  - shell_execute
  - web_fetch
---

# CVE-Hunt Benchmark Harness

Derived from T3MP3ST's `src/benchmark/index.ts` and `bench/` corpora
(Apache-2.0). The point of a ground-truth benchmark: measure finding quality —
precision AND recall — instead of counting scanner output.

## 1. Challenge definition

A benchmark challenge bundles everything needed to score a hunt:

```
id, name, description, difficulty(easy|medium|hard|expert),
target{address, type, zone},
tasks[{name, description, phase, operatorType, priority}],
groundTruth[{id, title, severity, category, matchKeywords[], points}],
timeLimitSec
```

- `groundTruth` is the answer key: every KNOWN vulnerability in the target with
  its severity, category, `matchKeywords` (what a correct finding should
  mention), and a `points` value.
- `tasks` defines the hunting workflow phases so runs are comparable.
- `timeLimitSec` bounds the run — time-to-finding is a scored metric.

## 2. Scoring — precision, recall, F1

For a run that produced `findings[]` against `groundTruth[]`:

- A finding matches a ground-truth vuln when it hits the vuln's `matchKeywords`
  (keyword containment over the finding's title/description/category).
- `truePositives` = findings that matched ≥1 ground-truth vuln.
- `falsePositives` = findings that matched nothing.
- `precision = TP / (TP + FP)` — how trustworthy the tool's output is.
- `recall = matched_ground_truth / total_ground_truth` — how much was found.
- `f1 = 2 * precision * recall / (precision + recall)`.
- `pointsScored` sums matched vulns' `points`; report `scorePercent` against
  `maxPoints`.
- `timeToFirstFinding` = seconds until the first valid finding; also report
  `findingsPerMinute` and `totalDurationSec`.

Severity breakdown: per severity, `{found, total}` — a tool that finds all the
lows but none of the criticals is not done.

## 3. Decoys — measuring the false-positive rate honestly

Include CLEAN samples in the corpus (T3MP3ST's `DECOY-clean-c`, `DECOY-clean-java`
pattern): code that LOOKS vulnerable but is not. Findings against decoys are
false positives by construction. A hunter that scores well on vulnerable samples
but flags decoys is pattern-matching, not analyzing — and the decoy FP rate is
the number that proves it.

## 4. Sample format (T3MP3ST bench layout)

```
bench/cve-hunt/samples/<CVE-ID>/
  source.<ext>          # the vulnerable (or decoy) source file
  ground-truth.yaml     # the answer key for this sample
```

`ground-truth.yaml` carries the vuln's id, title, severity, category,
matchKeywords, and points — the same shape as the challenge ground truth.
Real CVEs (Heartbleed, Shellshock, Log4Shell, etc.) make good anchors because
public writeups let you verify the answer key itself.

## 5. Running a hunt honestly

- The hunter must NOT see the ground truth. Score after the run.
- Fixed toolset and prompts per run so results are comparable.
- Log every finding with its evidence (file + line + reasoning) — the evidence
  gate from `evidence-vault-chain-of-custody` applies here too: prose-only
  findings are not detections.
- Report the full metric set, never just "it found bugs": precision, recall, F1,
  points, time-to-first-finding, severity breakdown, decoy FP rate.

## 6. OSA-specific wiring

- Score with a small Python script over JSONL findings `[LOCAL]` — keyword
  matching is a containment check, no dependencies needed.
- Store corpora under a scratchpad case dir; keep `ground-truth.yaml` out of the
  hunter's context (honor system: score only after the run).
- Use `delegate` to run hunter and scorer as separate agents — the scorer sees
  ground truth, the hunter never does.

## Attribution

Derived from [elder-plinius/T3MP3ST](https://github.com/elder-plinius/T3MP3ST)
(`src/benchmark/index.ts`, `bench/cve-hunt/`, `bench/cve-zero/`), Apache-2.0.
Patterns described; no code copied.

Part of the offensive skill library — see also `evidence-vault-chain-of-custody`
and `redteam-multi-agent-orchestration`.