verifying-review-findings v0.1.1A

Immutable. This exact content never changes and is served at /api/v1/blob/2f67fc7456939d70.

---
name: verifying-review-findings
description: "Checks each finding from a code review against the code it cites and keeps only the ones that hold. Drops findings whose quoted code is not in the file, then asks TypeSafe's Jev decision model whether the code shown really has the claimed defect, returning P(confirmed), P(refuted), a severity distribution and whether the proposed fix works. Use for \"verify these review findings\", \"filter false positives from the review\", \"is this bug real\", \"double-check the reviewer's claims\", \"get a second opinion on these findings\", a pre-commit or CI gate that blocks only on confirmed high-severity findings, or a finder-then-verifier review with a different model as the verifier."
metadata:
  version: 0.1.1
---

# Verifying review findings

A model that reviews code for defects also invents them: it misreads a guard
ten lines up, a loop bound, or a lock the caller holds. This skill splits the
review into two roles, as the open-source reviewer bluesky-social/roast does.
A **finder** (you, the `code-review` skill, any reviewer) writes each finding
with a verbatim quote of the code it is about. A **verifier** from a different
model family then reads that code and confirms or refutes the claim.

The verifier is Jev, a decision model from TypeSafe. It answers typed questions
(yes/no, multiple choice, ordered scale) with a probability for every answer in
about 0.3 s. Jev is not a Claude model, so when Claude finds and Jev verifies,
the verifier is less likely to share the finder's blind spots. Below,
P(x) means the probability Jev gives to answer x.

`scripts/verify.py` does the following for each finding:

1. **Grounding.** The quote must occur in the file, compared with runs of
   whitespace collapsed. A finding with no quote, or a quote that is not in
   the file, is dropped before Jev sees it.
2. **The window.** The verifier sees a *window* of the file: the top-level
   block (function, class, statement) that contains the quote, with at least
   12 lines on each side, capped at 160 lines. Lines are numbered and the
   quoted ones are marked `>`. With `--diff`, the file's diff hunks go along
   too.
3. **Three questions to Jev**, one call per finding, in parallel:
   - verdict: `confirmed`, `refuted` or `insufficient_context`
   - severity if the defect is real, on the scale `info`, `low`, `medium`,
     `high`, `critical`
   - when the finding proposes a fix, whether the fix works
4. **Thresholds.** P(refuted) of at least 0.7 drops the finding
   (`--refute-at`). P(confirmed) of at least 0.7 confirms it (`--confirm-at`).
   Anything else is `uncertain`. Only a refutation removes a finding; a
   verifier that is unsure keeps it.

## When NOT to use this skill

| Situation | Use instead |
|---|---|
| Nobody has looked for defects yet | `code-review` (or any reviewer) first; this skill only judges findings that exist |
| Checking that a README or docstring still matches the code | `verifying-claims` |
| An adversarial read of a whole deliverable (prose, analysis, a PR's design) | `challenging` |
| A routing, triage or yes/no question that is not about a code finding | `deciding-with-confidence`, or the Muninn `decide` tool directly |
| Building a deterministic check that must be able to go red | `gating` |
| The claim is about runtime behaviour the code cannot show (performance, a flaky test, production data) | run it; a verifier reading code can say `insufficient_context` at best |

Stop the run when most findings come back `uncertain` with reason
`needs_more_context`: the evidence for those claims lives outside their
windows (in a caller, another file, a config). Verify those findings by
reading the code yourself.

## Procedure

Throughout, `V=/mnt/skills/user/verifying-review-findings/scripts/verify.py`.

1. **Write the findings as a JSON list.** `file` (relative to the repository
   root) and `code_quote` are required. Copy the quote character for character
   from the file, one to six lines; a paraphrase is dropped.

   ```json
   [{"id": "1", "file": "src/util.py", "line": 6,
     "description": "Off-by-one: the loop skips the last element.",
     "code_quote": "for i in range(len(items) - 1):",
     "fix": "for i in range(len(items)):", "severity": "high"}]
   ```

   The other fields are optional. `id` defaults to the item's position.
   `line` picks between several matches of the same quote. `description` is
   the claim Jev judges. `fix` adds the fix question. `severity` is the
   finder's own estimate and is passed to Jev as a claim. Findings in the
   `ReportFindings` shape (`summary` and `failure_scenario` instead of
   `description`) are accepted too. When you are the finder, emit this shape
   directly.

2. **Pick the path.** `--transport auto`, the default, uses the first of
   `jev` and `gateway` whose credentials are set.

   | Path | Needs | How |
   |---|---|---|
   | Jev direct (`jev`) | `TYPESAFE_API_KEY`, a TypeSafe API key | `run` (step 3a) |
   | Jev via Cloudflare Workers AI (`gateway`) | `CF_ACCOUNT_ID` and `CF_API_TOKEN`, optionally `CF_GATEWAY_ID` | `run`; pass `--transport gateway` when both paths have credentials |
   | Muninn `decide` tool, which calls Jev | the Muninn MCP connector in the session | `emit`, then `apply` (step 3b) |
   | Claude Haiku via `deciding-with-confidence` | that skill installed next to this one | `run --transport decider`; the verifier is then also Claude |

3. **Run it.**

   a. With credentials in the environment:
   ```bash
   python3 $V run findings.json --repo . --format md      # default output is JSON
   git diff main... > /tmp/d.diff && python3 $V run findings.json --diff /tmp/d.diff
   python3 $V run findings.json --ref HEAD~1               # read files at a git ref
   ```

   b. Through the Muninn `decide` tool:
   ```bash
   python3 $V emit findings.json --out batch.json
   ```
   `emit` saves the grounded findings to `batch.json` and prints the Jev
   requests, one per surviving finding, each as `{"id", "state",
   "questions"}`. Call `decide` once per request, passing its `state` and
   `questions` unchanged; the calls are independent, so send them together.
   Save the replies in `answers.json` as an object that maps each finding's
   `id` to the full text `decide` returned, then:
   ```bash
   python3 $V apply batch.json answers.json --format md
   ```

4. **Read the report.** The top level has `complete` (false when any finding
   could not be verified), `counts` per status, and `gate`. Each finding keeps
   its input fields and adds:
   - `status`: `confirmed`, `uncertain`, `dropped`, or `unverified` (Jev was
     not reached or replied in an unexpected shape)
   - `reason`: why it is not confirmed. One of `missing_code_quote`,
     `ungrounded_code_quote`, `file_not_found`, `refuted_by_verifier`,
     `split_verdict`, `needs_more_context` (`insufficient_context` was the
     largest verdict), `no_answer`, or `verifier_error: ...`
   - `verdict`: the three verdict probabilities
   - `severity`: `probabilities` per level, and `expected`, the level nearest
     the probability-weighted average
   - `fix_ok`: the probability that the fix works
   - `claimed_severity`: the finder's own `severity` from the input
   - `fingerprint`: a hash of the file and the whitespace-collapsed quote. Use
     it to match a finding across runs, never to suppress one: two different
     claims about the same lines share a fingerprint.

5. **Act on it.**
   - Report `confirmed` findings as findings.
   - Report `uncertain` ones separately and read their code yourself; the
     verifier's mistakes cluster there.
   - Leave `dropped` findings out, but give their count and reasons so a
     reader can recover one that was refuted wrongly.
   - When a confirmed finding has `fix_ok` under 0.5, rewrite the fix before
     proposing it.

6. **Gate a commit or CI step** with `--fail-on LEVEL` (any severity level).
   Exit codes:

   | Code | Meaning |
   |---|---|
   | 0 | every finding verified, and no gate tripped |
   | 1 | a confirmed finding has P(severity at least LEVEL) of at least `--severity-at` (default 0.5) |
   | 2 | bad input, or no transport available |
   | 3 | some finding is `unverified`: the run is incomplete, not clean |

   A pre-commit hook that should not block on an outage can let the commit
   through on 3, as long as the hook prints the report so the gap is visible:
   ```bash
   python3 $V run staged.json --fail-on high --format md; rc=$?
   [ $rc -eq 3 ] && echo "verify: incomplete run, commit allowed" >&2 && rc=0
   exit $rc
   ```

## Earned exceptions

| Default | Change it when | Keep it when |
|---|---|---|
| `--refute-at 0.7` | a missed defect costs more than reading a false one (security review, release gate): raise it toward 0.9 so fewer findings are dropped | the review is advisory, and noise is what makes people stop reading |
| The gate counts only confirmed findings | the gate guards something irreversible: lower `--confirm-at` so more findings count as confirmed | `uncertain` findings get a human look before merge |
| One window per finding | the claim depends on another file (a caller, a type, config) | the evidence is in the same top-level block, which covers most of what reviewers flag |

## Common failure modes

- **`verify: no Jev transport`, exit 2** → neither `TYPESAFE_API_KEY` nor the
  Cloudflare pair is set. Use the `emit`/`apply` path, or `--transport decider`.
- **Every finding `dropped` with `ungrounded_code_quote`** → the finder quoted
  from memory or reformatted the code, or `--repo` points at the wrong
  checkout. Run `grep -F` with one quote against the file to tell which.
- **`file_not_found` for files that exist** → `file` is resolved against
  `--repo` (default: the current directory). A path that leaves the
  repository is treated as missing on purpose.
- **`unverified` with `verifier_error: HTTP 4xx`** → check the error text.
  When it names the input length, the window was too long for Jev's default
  context: lower `MAX_LINES` (the 160-line cap, a constant at the top of
  `verify.py`). The report says `complete: false` until those findings are
  re-run.
- **`unverified` with `no_answer` after `apply`** → `answers.json` has no
  entry for that finding's `id`. Ids are strings: `"1"`, not `1`.
- **A high `severity.expected` on a dropped finding** → severity answers "how
  bad if real". Read it only on findings that survive.

## Verification

```bash
python3 -m pytest /mnt/skills/user/verifying-review-findings/tests -q   # offline, no network
python3 $V eval /mnt/skills/user/verifying-review-findings/assets/eval.jsonl
```

`eval` runs 36 labelled findings (16 real defects, 16 false findings, 4
quotes not in the file) and prints the counts. Grounding is deterministic,
so `ungrounded_dropped` must be 4/4. Jev's part gave `real_kept` 16/16 and
`false_dropped` 11/16 with jev-1.14.0 on 2026-11-07. A lower count means the
Jev version or the window settings changed; compare per item with
`--out results.jsonl`.

A **bad success** is a report with `complete: true` in which every finding is
`confirmed` near 1.0, including ones you know are false: Jev is judging the
description without the code. Check that `state.code` in the requests `emit`
prints holds the numbered window.

The eval's per-item results, the threshold sweep and what it does not cover
are in `references/method.md`.

## Diagnosed failures

- **2026-11-07, eval item `ts-04`:** a finding claimed a race condition in
  synchronous single-threaded JavaScript. Jev confirmed it at 0.89, the only
  false finding confirmed. No threshold catches a confident error like this.
  Read the code behind any concurrency claim about single-threaded code yourself.
- **2026-11-07, eval items `py-05` and `sh-03`:** the evidence that refutes
  each claim was in the same file but in a different function, outside the
  window. Neither was dropped. `sh-03` came back `uncertain` with
  `needs_more_context` (P(insufficient_context) 0.82). `py-05` came back
  `uncertain` with `split_verdict` (P(confirmed) 0.55, P(insufficient_context)
  0.40). Both outcomes are correct: the finding stays in the report, flagged
  for a human to read.