git:20260810.8a876fa to git:20260930.e4f475f

- description: >

- Design or audit the clinical-validation study for an engineer-built medical-imaging model

- (segmentation, classification, or detection) before the validation report or manuscript is written.

- Covers patient-level split disjointness and the data-leakage taxonomy, tuning-on-test, internal

- versus genuine external validation, comparator design, single-run versus multi-seed variance,

- task-correct metric selection, test-set sizing, and CLAIM 2024 / TRIPOD+AI / STARD-AI reporting fit.

- Ships a deterministic split-leakage gate that proves patient disjointness by set arithmetic on the

- emitted split-assignment table. Does not build or train models — it integrates with MONAI / nnU-Net,

- it does not replace them.

- triggers: model validation, validate AI model, imaging model validation, data leakage, split leakage, train test split, patient-level split, internal validation, external validation, validation design, leakage audit, segmentation model validation, classification model validation, detection model validation, nnU-Net validation, deep learning validation, CLAIM 2024, generalizability, held-out test set

- tools: Read, Write, Edit, Bash, Grep, Glob

- model: inherit

- # Model-Validation Skill

-

- ## Purpose

-

- This skill pressure-tests the **validation study for an engineer-built medical-imaging model** — the

- common case where a clinical team receives a trained segmentation / classification / detection model

- from an engineering collaborator and must validate it and write it up. It is the imaging-model

- specialization of `/design-study`: where `design-study` covers general validity and

- `design-ai-benchmarking` covers AI-versus-human-expert reader studies, this skill owns the

- **partition, leakage, reproducibility, and metric-selection mechanics** that decide whether a reported

- Dice / AUROC / sensitivity is trustworthy.

-

- It is **advisory and deterministic-audit only**. It writes decision notes and runs a stdlib gate on the

- split table; it never builds, trains, or alters the model, and it never replaces MONAI / nnU-Net /

- TorchIO — those produce the model, this validates and publishes it.

-

- ## When to use

- - A trained imaging model (in-house, vendor, or open-weights) needs a clinical-validation study designed

- or audited before submission.

- - You have, or can produce, the **split-assignment table** (which patient went to train / val / test).

-

- ## When NOT to use

- - Building or training the model → out of scope (integrate MONAI / nnU-Net).

- - AI-versus-human-expert reader study → `/design-ai-benchmarking`.

- - LLM / MLLM evaluation → `/mllm-eval` (when available).

- - General study/validity review → `/design-study`.

- - Statistical execution (DeLong, ICC, bootstrap CIs, calibration tables) → `/analyze-stats`.

- - Item-by-item reporting-guideline audit of a finished manuscript → `/check-reporting`.

- - Reviewing a finished manuscript → `/self-review` or `/peer-review` (which load the MD0–MD11

- reviewer-side probe).

-

- ## Workflow

-

- The design/audit rationale behind Phases 2–7 — the full data-leakage taxonomy, the

- internal-vs-genuine-external validation ladder, comparator design, single-run vs multi-seed

- variance, test-set sizing, and the CLAIM 2024 / TRIPOD+AI / STARD-AI reporting map — is in

- `${CLAUDE_SKILL_DIR}/references/validation_design.md` (load on demand). The patient-disjointness

- verdict itself is proven by `scripts/check_split_leakage.py` (Phase 2), not from that prose.

-

- ### Phase 1 — Reconstruct the task, the intended-use horizon, and the analysis unit

- State the model's task (segmentation / classification / detection), its **intended-use horizon**

- (screening, triage, pre-procedure, post-hoc), the **single headline metric** the conclusion leans on,

- and the **analysis unit** the metric must respect (per-patient vs per-lesion vs per-image). Everything

- downstream is read against this.

-

- ### Phase 2 — Leakage audit (the deterministic gate, run first)

- The most metric-inflating defect is a split that is **not disjoint at the patient level**. Produce the

- emitted split-assignment table (`patient_id,split`) and run the gate:

-

- ```bash

- python3 ${CLAUDE_SKILL_DIR}/scripts/check_split_leakage.py \

- --splits <split_assignment.csv> --out qc/split_leakage.json --strict

- ```

-

- `PATIENT_OVERLAP` (a patient in ≥ 2 partitions) and `MISSING_SEED` (an unreproducible split) are

- proven by set arithmetic — not heuristics. Then walk the rest of the **leakage taxonomy** (Kapoor &

- Narayanan, *Patterns* 2023) that the table cannot show: **preprocessing-before-split** (normalisation,

- resampling, foundation-model embeddings, or ComBat harmonisation fit on the whole cohort before

- partitioning), **site / scanner / burned-in-label shortcuts**, and **temporal leakage** (a random split

- where future and past coexist). The decisive question: *could any value used in training have been

- computed only with knowledge of a test case?*

-

- ### Phase 3 — Validation tier (internal split vs genuine external)

- Classify the evidence honestly: apparent → internal random split → cross-validation → temporal →

- geographic / external (different site, scanner, vendor) → multi-site external. Cross-validation and

- bootstrap are development-time optimism corrections, **not** external validation. Flag a generalisability

- or deployment claim that outruns an internal-only design, and "developed with external validation" where

- the single external set was used for tuning. Also confirm the test set was touched **once** — no

- architecture search, hyperparameter sweep, early-stopping, or operating-point / threshold choice read

- the test set.

-

- ### Phase 4 — Comparator design

- Decide what the model is compared against: clinical-only baseline, incremental value over an existing

- score, or reader comparison. For a reader comparison, hand the rubric / inter-rater design to

- `/design-ai-benchmarking`.

-

- ### Phase 5 — Metric selection (task-correct, prevalence-aware)

- Match the metric to the task and the prevalence (Metrics Reloaded — Maier-Hein & Reinke et al.,

- *Nat Methods* 2024): **segmentation** pairs an overlap metric (Dice / IoU) with a **boundary metric**

- (HD95 / Normalised Surface Distance), per-structure not only global; **classification under imbalance**

- reports **AUROC and AUPRC with CIs** plus sensitivity / specificity and prevalence-dependent PPV / NPV,

- never bare accuracy on a balanced set; **detection** reports FROC / mAP with the IoU match criterion

- stated. Require the headline metric as **mean ± SD across ≥ 3 seeds / runs**, or a fixed reported seed

- with the determinism caveat. The per-case metric computation + the deterministic metric-reporting gate

- live in `/model-evaluation` (which emits the per-case table for `/analyze-stats`). For **interactive /

- promptable segmentation** (SAM2 / MedSAM2 / nnInteractive) the metric set adds the interaction axis —

- number of clicks / interactions-to-threshold (NoC), initial-vs-converged Dice, and per-case

- interaction / inference time (`/model-evaluation --task interactive`). When the evaluation runs two arms

- (simulated prompting + human-operator validation), record **protocol fidelity** across arms — identical

- prompt types, stopping rule, target threshold, and seeds — as an explicit validity item: arm-to-arm

- comparability is the precondition for reading the human-operator arm as validating the simulated one,

- and the human-operator arm design is in `/design-study`.

-

- ### Phase 6 — Test-set sizing

- Check the **events per class** in the test set, not the cohort total — a metric on a sparse positive set

- has a confidence interval spanning much of the usable range, and calibration needs roughly ≥ 100 events.

- Hand the formal sizing (diagnostic-accuracy / AUC precision / agreement) to `/calc-sample-size`.

-

- ### Phase 6.5 — Prospective evaluation & deployment-monitoring horizon

- Retrospective external validation shows accuracy *transfers*; it does **not** show the model

- is safe and useful *in the clinical workflow*. If the claim is clinical use — not just

- discrimination — design the higher tier explicitly: **silent / shadow deployment** (live

- cases, no care impact, prospective performance + calibration targets) → **prospective

- comparative / impact study or RCT** on a clinical endpoint → **post-deployment monitoring**

- for performance / dataset-shift / calibration drift with recalibration-or-withdrawal

- triggers and ongoing subgroup-performance audit. See `references/validation_design.md` §2b.

- Scope the claim to the tier reached — a retrospective external study must not claim

- deployment readiness or clinical-outcome benefit.

-

- ### Phase 7 — Reporting-guideline fit

- Map the study to its reporting standard via `/check-reporting`: **CLAIM 2024** (diagnostic imaging AI),

- **TRIPOD+AI** (prediction model), **STARD-AI** (diagnostic accuracy), **PROBAST+AI** (risk of bias),

- and — for a prospective/live evaluation (Phase 6.5) — **DECIDE-AI** (early clinical evaluation of

- decision-support AI) or **CONSORT-AI / SPIRIT-AI** (full AI trials / protocols).

-

- ### Phase 8 — Handoffs

- Carry the audited design into `/write-paper` (Methods), `/calc-sample-size` (sizing), `/check-reporting`

- (compliance), and — for the reviewer-side audit of the finished draft — `/self-review`, which loads the

- `model_development.md` (MD0–MD11) probe.

-

- ## Deterministic gate

-

- `scripts/check_split_leakage.py` — proves patient-level split disjointness + seed presence on the emitted

- split-assignment table (stdlib, network-free). Verdicts: `PATIENT_OVERLAP` (Major), `MISSING_SEED`

- (Major), `SINGLE_PARTITION` (Minor). Reproducible challenge:

- `bash ${CLAUDE_SKILL_DIR}/scripts/check_split_leakage_challenge/verify.sh`.

-

- ## Anti-Hallucination

-

- - **Never fabricate performance metrics, split assignments, event counts, or seeds.** Every number comes

- from the engineer's executed code, the supplied split table, or a re-run of the deterministic gate —

- never invented. A reported Dice / AUROC / overlap count with no underlying record is the failure mode

- this skill exists to prevent.

- - **Never report a split-audit "pass" without running `check_split_leakage.py`.** The patient-disjointness

- verdict is proven by the script, not asserted from prose.

- - **Never invent references, reporting-guideline items, or metric-selection rules.** Verify citations via

- `/search-lit` (confirmed DOI / PMID); mark unverified ones `[UNVERIFIED - NEEDS MANUAL CHECK]`. If a

- CLAIM 2024 / TRIPOD+AI / Metrics-Reloaded item is uncertain, flag `[VERIFY]` and ask the user rather

- than guessing.

- - **Do not claim external validation, generalisability, or deployment readiness the design does not

- support** — classify the validation tier honestly and let the evidence cap the claim.

-

- ## Boundaries — which skill to use, in what order

-

- ```

- design-study (general validity)

- └─ model-validation (this skill: leakage, split, comparator, metric, sizing handoff)

- ├─ check_split_leakage.py (deterministic patient-disjointness gate)

- ├─ calc-sample-size (test-set / event sizing)

- ├─ design-ai-benchmarking (reader-comparison rubric / IRR)

- ├─ check-reporting (CLAIM 2024 / TRIPOD+AI / STARD-AI)

- └─ write-paper -> self-review / peer-review (MD0–MD11 reviewer probe)

- ```

-

- It does not build the model (integrate MONAI / nnU-Net), compute publication statistics (`/analyze-stats`

- owns DeLong / ICC / calibration tables), or evaluate an LLM / MLLM (`/mllm-eval`).