design-study git:20260725.13ac006A

Immutable. This exact content is served forever at /api/v1/blob/113a91e5696f7720.

---
name: design-study
description: >
  Study design and validity review for radiology and medical AI research. Identifies analysis unit,
  cohort logic, leakage risks, comparator design, validation strategy, and reporting guideline fit before
  drafting or submission.
triggers: study design, leakage check, cohort design, analysis plan, validation strategy, comparator design, bias check
tools: Read, Write, Edit, Bash, Grep, Glob
model: inherit
---

# Design-Study Skill

## Purpose

This skill pressure-tests whether a study is answerable, interpretable, and defensible before large amounts of drafting or analysis work accumulate.

Use it when:
- a study question is known but the analysis plan is still fluid
- the user wants a methods sanity check
- a manuscript feels vulnerable to reviewer criticism
- a peer review requires explicit methodological diagnosis

---

## Communication Rules

- Communicate with the user in their preferred language.
- Use English for statistical, radiologic, and reporting-guideline terminology.
- Be direct about validity risks, but always propose the smallest feasible fix first.

---

## Core Review Questions

Always inspect these dimensions:

1. What is the exact research question?
2. What is the analysis unit: patient, lesion, exam, study, phase, report?
3. What is the index date or decision point?
4. How are inclusion and exclusion criteria applied?
5. Is there any information leakage?
6. What is the reference standard or endpoint definition?
7. What comparator is clinically meaningful?
8. What validation strategy is used?
9. What uncertainty reporting is required?
10. Which reporting guideline best fits?
11. Are exposure/outcome/covariate **definitions literature-grounded**, or invented ad-hoc from the data dictionary? If ad-hoc, defer to `/define-variables` before drafting Methods.

---

## Standard Output

```text
## Study Design Review
Question: ...
Study type: ...
Analysis unit: ...
Index date / prediction timepoint: ...

### Strengths
- ...

### Major validity risks
1. ...
2. ...

### Minimal fixes
- ...

### Reporting fit
- Recommended guideline: ...

### Decision
- Ready for analysis / Needs redesign / Drafting can proceed with limitations
```

---

## Workflow

### Phase 1: Reconstruct the study

Extract from protocol, draft, slides, tables, or notes:
- clinical problem
- intended use case
- population
- inputs
- outputs
- outcome definition
- timing of variable availability

**Gate:** Present the reconstructed study summary (question, analysis unit, intended use)
to the user. Confirm before proceeding — if the reconstruction is wrong, the entire
validity review will be misdirected.

### Phase 2: Check structural validity

#### A. Analysis unit

Look for mismatches such as:
- patient-level claim from lesion-level analysis
- exam-level split with patient overlap
- phase-level samples treated as independent

#### B. Leakage

Look for:
- postoperative features used for preoperative prediction
- normalization or thresholding performed before data split
- repeated exams across train/test
- reader annotations derived from outcome information
- **input-text contamination for NLP/LLM extraction tasks**: if the model input includes report
  sections such as clinical history, indication, impression, prior diagnosis, or referral text, confirm
  that those fields do not literally name or strongly imply the target label. If the target is already
  present in the supplied text, the task is information retrieval under label leakage, not phenotype
  inference; redesign the input mask, report a sensitivity analysis excluding leaky fields, or reframe the
  claim.
- **construct dependence** (a predictor that is a definitional component of the outcome). Two cases:
  (i) *mathematical definition* — an input that computes the outcome (when the outcome is HOMA-IR =
  f(fasting insulin, fasting glucose), those two inputs are not independent predictors); (ii)
  *near-tautological composite* — a ratio or score built from the outcome's defining components, which
  shows an inflated, near-circular association. Test: "could this predictor be derived, in whole or
  part, from the outcome's definition or the same measurement?" If yes, exclude it, or retain it only
  as a labeled calibration probe rather than a reported discovery.

#### F. Time origin & survivorship (incident / transition models)

For any time-to-event or incident/transition design, check before drafting:
- **Time origin per model.** Each incident model starts its at-risk clock at the correct origin. Watch for **immortal-time bias** (a span in which the event cannot occur, misattributed to one group) and **left-truncation / delayed entry** (subjects entering the risk set after the origin).
- **Mediator-ascertainment-window survivorship.** A "progressor" / transition label that is conditional on *surviving to* a later ascertainment (a second scan, a follow-up visit) is survivorship-biased; plan a landmark time or an explicit intermediate-state (multistate / illness-death) model.
- **Primary-analysis-set selection.** If the primary will not be the full cohort (e.g., complete-case while a large fraction is missing), pre-specify the selection justification and a MAR rationale; do not let the complete-case model become primary because it is the significant one (an outcome-dependent choice).
- A design that cannot yet answer these should say so honestly — but note that at review time a Methods/Limitations admission that the issue was *"not formally assessed"* is escalated to a MAJOR by the survival probe (S1), not waved through as a limitation.

#### C. Reference standard

Check:
- who established ground truth
- when it was established
- whether blinding was possible
- whether only a subset had gold standard verification
- **Construct ↔ nominal-definition match.** Does the exposure/finding *construct* stay inside its stated definition, or does it quietly exceed it? An "incidentaloma" defined as an *indeterminate* finding must not include frank malignancy reads; a label that overshoots its definition inflates the apparent cohort and breaks the κ. For each construct, restate the nominal definition and confirm every included case satisfies it.
- **Per-flag reference-standard concordance.** When the index finding is flagged against a reference standard, report the concordance *per flag category* (not just overall). A construct where a large fraction of flags do not match the reference standard (e.g., ~86% non-match) is measuring something other than the named construct.
- **Manuscript definition ↔ `variable_operationalization.md`.** The variable definitions written in Methods must match the operationalization table verbatim (dictionary-first). A blinded re-classification form must quote the analytic protocol's definition verbatim — paraphrase / "common-sense extension" in the form (but not the Methods) is the documented cause of a low κ that is a *definition mismatch*, not real disagreement. Cross-check with `/define-variables` output before drafting.

#### D. Validation

Classify:
- apparent only
- internal split
- cross-validation
- temporal validation
- external validation
- multi-center external validation

#### E. Reader / expert-elicitation studies (load on demand)

When the study elicits expert ratings — a reader study, an annotation panel, an AI-output
evaluation — the design decisions that matter are made **before data collection**, and the
acceptance ceiling of a perceptual / reader AI study is fixed at design time: no quality of
execution lifts a ceiling baked into the comparator, the estimand, or the reader cohort.

For an AI-system-versus-human-expert benchmark specifically, route to `/design-ai-benchmarking`,
which extends this subsection with arm definition, LLM-as-judge versus human-as-judge
adjudication, and a structured export schema.

| File | Read it when | Cost if read blindly |
|---|---|---|
| `references/reader_elicitation_design.md` | the design has a human-rater or expert-elicitation arm — rubric axes, calibration probes, operational rigor, human-as-operator, and the six ceiling decisions | ~2,400 tokens, none of which applies to a design with no reader arm |
| `references/dag_adjustment.md` | confounding control needs an explicit adjustment set | — |
| `references/target_trial_emulation.md` | the design emulates a target trial | — |
| `references/venue_accept_recipe.md` | it is a clinical DL / AI-validation study and the question is **which venue tier the achievable design can be accepted at, and the one design move that reaches the tier above** (the design→acceptance-tier ladder + the five acceptance levers, reverse-engineered from accepted OA papers; the bridge into `/find-journal`) | ~1,800 tokens; skip for a design with no publication-tier decision |
| `references/combine_models_ablation_design.md` | the model is built by **combining / adapting / fine-tuning existing models** (nnU-Net, TotalSegmentator, SAM/MedSAM, a pretrained backbone) — how to design the comparator as an **ablation** that proves the combination earns its complexity (beat the un-adapted base + the best single component + direct-train), reverse-engineered from accepted OA papers | ~1,600 tokens; skip if the model is trained de novo with no reused component |
| `references/multi_model_comparison_design.md` | the study's contribution is **comparing several models / architectures head-to-head** (CNN vs Transformer vs foundation backbone; N segmentation networks) — how to make the comparison **fair**: one frozen split + one preprocessing through every model, a strong self-configuring baseline (nnU-Net) not a hobbled one, **matched training/HPO/compute budget** (the #1 threat — "new≠better, just tuned harder") or disclosed, variance-over-single-run, a pre-specified primary metric + a **paired** delta test, honest ranking. Reverse-engineered from accepted OA papers | ~1,700 tokens; skip for a single-model study (use `combine_models_ablation_design.md` for an ablation of one model, `/design-ai-benchmarking` for AI-vs-human) |

### Phase 3: Clinical framing

Ask whether the comparator and endpoint support the stated claim:
- is the model better than current practice or just another model?
- is the endpoint clinically meaningful?
- does performance translate to action?
- **incremental value**: if the study frames the model/marker as adding value *beyond* / *on top of* / *incremental to* an existing tool (a clinical score, a routine test, a baseline model), the design must pre-specify the baseline comparator built from the in-routine-use predictors **and** an incremental-value metric — ΔC-index / ΔAUC (with a paired CI, e.g. DeLong), categorical or continuous NRI, IDI, or decision-curve net benefit. A standalone discrimination number ("our model's AUC was 0.84") does not support a "beyond X" claim; without the nested-model comparison the finding may be real but redundant. Plan this at design time — it cannot be added post hoc without the baseline model.
- **fine-tuning contribution baseline**: if an NLP/LLM study claims that fine-tuning, LoRA, prompt
  engineering, or a multi-agent wrapper improves extraction/classification, pre-specify a same-backbone
  zero-shot or few-shot comparator on the identical input, output schema, and test split. A comparison
  only against a weaker or unrelated baseline cannot establish that the proposed adaptation adds value. For an **imaging** model built by combining / adapting / fine-tuning existing models (nnU-Net, a foundation model, a pretrained backbone), design the full ablation ladder — un-adapted base, best single component, direct-train vs transfer — per `references/combine_models_ablation_design.md`. When the contribution is instead a **head-to-head comparison of several models** (which architecture wins), the decisive design question is comparison *fairness* — one frozen split/preprocessing through every model, a strong fairly-tuned baseline, a matched (or disclosed) compute budget, and a paired delta test — per `references/multi_model_comparison_design.md`.
- **endpoint↔conclusion scope**: decide up front what *kind* of conclusion the design can support, so the manuscript does not overreach. A cross-sectional / single-visit / prevalence design cannot support a prognostic or surveillance claim (rescreen interval, disease progression) — that needs longitudinal follow-up. A binary surrogate endpoint (present/absent, >0, dichotomized) is risk stratification, not a patient-care directive (defer/withhold/initiate therapy). At review time `/self-review` §D + `check_scope_coherence.py` flag `CROSS_SECTIONAL_PROGNOSTIC` / `SURROGATE_CARE_DIRECTIVE` against the conclusion.

### Phase 4: Reporting fit

Recommend one primary guideline:
- `TRIPOD-AI`
- `CLAIM`
- `STARD`
- `STROBE`
- `PRISMA`
- `CARE`
- `ARRIVE`
- journal-specific additions if needed

---

## Frequent Failure Modes

### Diagnostic AI
- no clinically relevant comparator
- exam-level split instead of patient-level split
- unclear reference standard
- AUROC-only reporting without threshold metrics

### Prognostic modeling
- unclear time zero
- immortal time bias
- feature timing mismatch
- no calibration

### Retrospective cohort / screening database
- **time zero misalignment**: cohort entry ≠ follow-up start → immortal time bias
- interval-censored outcomes treated as exact → underestimation of event times
- healthy volunteer bias unacknowledged → inflated external validity claims
- surveillance bias from unequal follow-up frequency between groups
- **3 bias classification (Hernan/Robins)**: selection bias (who enters), information bias (how measured), confounding (what else differs) — explicitly map each threat
- **comparative / causal question → emulate a target trial.** For a treatment-vs-treatment, screening-vs-no-screening, or drug-A-vs-drug-B question on routinely-collected data, specify the seven target-trial components (eligibility, strategies, assignment, **time zero**, outcome, causal contrast, analysis plan) before extraction — this is what prevents the immortal-time / prevalent-user / confounding-by-indication trio above and turns an association into a defensible causal contrast. New-user + active-comparator design, grace-period clone-censor-weight, and negative controls are in `references/target_trial_emulation.md`.
- **confounding completeness**: pre-specify the adjustment set from a DAG (not a Table-1 p < 0.05 rule), and plan to report whether any measured covariate that turns out imbalanced by exposure but outside the adjustment set leaves the primary estimate robust (an extended-adjustment sensitivity model). Build the DAG and pre-screen the proposed covariates with `scripts/adjustment_set_helper.py` (flags mediator / collider / descendant adjustment and omitted confounders, and proposes a candidate backdoor set), then derive the **minimal** sufficient set with dagitty — see `references/dag_adjustment.md`. At review time `/self-review` Phase 2.5e + the O1–O12 probes in `observational_confounding.md` check this against Table 1 (including O7 over-adjustment, O10 overlapping-subset-gradient discipline, for complex-survey data O11 design-based weighting and O12 data-driven-threshold mining, O13 — a cross-sectional mediation claim cannot order X→M→Y, and O14 — a synergy/joint-effect claim needs the additive interaction scale (RERI/AP/S), not a multiplicative-only test).

### Multimodal LLM / report generation
- no clear rubric for clinical correctness
- benchmark labels derived from noisy reports without adjudication
- unsupported claims about safety or workflow benefit
- input text contains the target label or diagnosis being predicted
- no same-backbone zero-shot/few-shot baseline for a fine-tuning or prompt-engineering claim

### Imaging meta-analysis
- overlapping cohorts
- paired modalities analyzed as independent
- heterogeneity metrics missing
- zero-cell handling unspecified

---

## Minimal-Fix Principle

Whenever possible, recommend the smallest feasible repair first:

- clarify the claim
- narrow the target population
- add a limitation statement
- add a clinically relevant baseline
- re-run one key sensitivity analysis
- redefine the endpoint more explicitly

Escalate to redesign only when the central claim is not defensible otherwise.

---

## Handoff Rules

- route to `analyze-stats` when the design is basically sound but analysis details need refinement
- route to `check-reporting` after the design is locked
- route to `self-review` when the user wants a pre-submission quality check on their own manuscript
- route back to `write-paper` only after the main validity risks are documented

---

## What This Skill Does NOT Do

- It does not compute statistics directly
- It does not draft full manuscript prose
- It does not resolve raw data engineering issues
- It does not replace a full peer review when journal-facing tone is required

## Anti-Hallucination

- **Never fabricate references.** All citations must be verified via `/search-lit` with confirmed DOI or PMID. Mark unverified references as `[UNVERIFIED - NEEDS MANUAL CHECK]`.
- **Never invent clinical definitions, diagnostic criteria, or guideline recommendations.** If uncertain, flag with `[VERIFY]` and ask the user.
- **Never fabricate numerical results** — compliance percentages, scores, effect sizes, or sample sizes must come from actual data or analysis output.
- If a reporting guideline item, journal policy, or clinical standard is uncertain, state the uncertainty rather than guessing.