git:20260724.f8a95c4 to git:20260930.e4f475f

- description: >

- Profile a medical-imaging dataset before any modelling decision is made — the acquisition grid,

- voxel spacing and orientation spread, the intensity domain, which label values are actually

- present, how much of the volume the target occupies, and how large the target is in millilitres —

- then gate that profile against the researcher's declared plan. Catches, at the point where it is

- still cheap, the dataset facts that otherwise surface after a training run: a "test set" that

- carries no ground truth, labels whose grid does not match their image, a stray label index, a

- target occupying a fraction of a percent while accuracy is planned as a metric, and acquisition

- heterogeneity nobody declared a resampling decision for. Emits a dataset-profile JSON and a

- deterministic gate that reads it (stdlib-only, so an audit travels with the JSON). It describes

- the data and audits the plan against it; it does not preprocess, split, or train.

- triggers: profile dataset, dataset profile, EDA, exploratory data analysis, explore the data, what does the data look like, imaging dataset, NIfTI, voxel spacing, slice thickness, orientation, intensity distribution, Hounsfield, class imbalance, foreground fraction, label sanity, empty label, label QC, dataset QC, data audit, before training, target volume, organ volume, is my test set labelled, research direction, where do I start

- tools: Read, Write, Edit, Bash, Grep, Glob

- model: inherit

- # Profile-Imaging Skill

-

- ## Purpose

-

- A dataset decides more of a study than the architecture does, and it decides it **first**. Before

- anything is preprocessed, split, or trained, a handful of facts are already true about the data, and

- each one closes off or opens up a research plan:

-

- - If the target occupies 0.4 % of the volume, accuracy is not a metric — predicting background

- everywhere scores 99.6 %.

- - If through-plane spacing runs 1.5–8 mm inside a single institution, resampling is not a default to

- accept quietly; it is the most consequential preprocessing choice in the study, and it is also the

- axis along which an external dataset will differ.

- - If the directory named `imagesTs` has no labels, it is not a test set, and the held-out set has to

- come from somewhere else — better known before training than after.

- - If the organ volume spans 56–502 mL when normal is roughly 100–250, the cohort contains disease

- that a subgroup analysis should be **pre-specified** for, rather than discovered post hoc.

-

- None of that requires a model, a GPU, or an engineer. It requires reading every file once and

- writing down what is there. This skill does that, and then audits the plan against it.

-

- It is the **front door** of the model-engineering lane:

- `profile-imaging (describe)` → `/design-study` + `/architecture-zoo` (decide) →

- `/preprocess-imaging` (plan the pipeline) → `/model-scaffold` (build) → `/model-validation` →

- `/model-evaluation`.

-

- ## When to use

- - You have a dataset and a task, and need to know what the data will and will not support before

- committing to a plan.

- - You inherited a dataset and need its integrity established (labels intact, splits labelled,

- label values as declared) before anyone trains on it.

- - You are about to write a Methods section that describes the cohort and its acquisition.

-

- ## When NOT to use

- - Tabular / clinical variables → `/generate-codebook` (data dictionary) and `/clean-data`.

- - Designing the preprocessing pipeline and auditing it for data-stage leakage →

- `/preprocess-imaging` (it consumes what this skill describes).

- - Auditing the train/val/test split table → `/model-validation`.

- - Choosing an architecture → `/architecture-zoo`. Building the repo → `/model-scaffold`.

- - Held-out metrics, calibration, subgroup results → `/model-evaluation` then `/analyze-stats`.

-

- ## Workflow

-

- ### Step 1 — profile every case

-

- ```bash

- python3 scripts/profile_imaging_dataset.py \

- --split train:imagesTr:labelsTr \

- --split test:imagesTs \

- --dataset "MSD Task09 Spleen" \

- --declared-labels 0=background,1=spleen \

- --target-label 1 \

- --plan resample=true,reorient=false,loss=dice_ce,metrics=dice+hd95 \

- --out eda/profile.json

- ```

-

- One record per case: grid, spacing, orientation, intensity percentiles, the label values actually

- present, foreground fraction, and target volume in mL. A `--split` given no label directory is

- recorded as **unlabelled** — which is itself a finding.

-

- **`--target-label` on a multi-structure atlas.** Foreground defaults to every non-zero index, which

- is the whole annotated anatomy. Run a single-organ study against a 15-organ atlas and the reported

- fraction describes the upper abdomen, not the target — measured on AMOS22 that is 3.2 % rather than

- the spleen's 0.2 %, so the pooled number sits *above* the 1 % imbalance threshold while the real

- target sits far below it, and the imbalance verdicts go quiet exactly where the risk is. Naming the

- target also makes `LABEL_EMPTY` mean *this case has no spleen*, which a multi-organ label file

- otherwise hides behind the other organs. Pass `--target-label all` for a genuinely multi-class

- study; leave it out on a multi-structure atlas and the gate raises `TARGET_LABEL_UNDECLARED`.

-

- Requires `nibabel` + `numpy` (it has to open images). The gate below does not.

-

- ### Step 2 — gate the profile against the declared plan

-

- ```bash

- python3 scripts/check_dataset_profile.py --profile eda/profile.json \

- --out qc/dataset_profile.json --strict

- ```

-

- Stdlib-only, so the audit re-runs anywhere the JSON travels. Verdicts:

-

- | Verdict | Severity | Fires when |

- |---|---|---|

- | `LABEL_SHAPE_MISMATCH` | Major | label grid ≠ image grid |

- | `LABEL_EMPTY` | Major | a labelled case has zero foreground |

- | `LABEL_VALUE_UNEXPECTED` | Major | label values outside the declared set |

- | `TEST_SET_UNLABELLED` | Major | a split whose name contains test/held-out/external/eval carries no labels |

- | `ACCURACY_UNDER_IMBALANCE` | Major | accuracy is planned while the target is a sliver of the volume |

- | `LABEL_MISSING` | Minor | a case in a labelled split has no label file |

- | `SPACING_HETEROGENEOUS` | Minor | spacing spans ≥ ratio on an axis and no resampling is declared |

- | `ORIENTATION_MIXED` | Minor | >1 orientation code and no reorientation declared |

- | `INTENSITY_SCALE_INCONSISTENT` | Minor | some cases sit on the HU scale and others do not |

- | `EXTREME_IMBALANCE` | Minor | median foreground below the threshold with no Dice-family loss |

- | `TARGET_LABEL_UNDECLARED` | Minor | >1 structure declared but no target named, so foreground pools them all |

-

- **The gate flags an undeclared decision, not variability itself.** A dataset with 5× spacing spread

- and two orientation codes passes cleanly once resampling and reorientation are declared —

- heterogeneity that has been dealt with is not a defect. That distinction is what the challenge card's

- clean fixture exists to prove.

-

- `--spacing-ratio` (default 2.0) and `--imbalance-frac` (default 0.01) are **screening defaults, not

- published cut-points**: 2× through-plane spacing changes what a fixed-size patch sees, and 1 %

- foreground is roughly where plain accuracy stops carrying information. Both are adjustable and both

- are printed in the output, so a reader knows what was applied.

-

- ### Step 3 — turn the profile into research decisions

-

- The profile is evidence; the decisions are yours, and the ones worth writing down are:

-

- 1. **Resampling target** — from the spacing distribution, not from a tutorial default. Carry it into

- `/preprocess-imaging` as a declared transform.

- 2. **Loss and metric family** — from the foreground fraction. Segmentation reports Dice **and** a

- boundary metric per structure (`/model-evaluation`); accuracy is not on the list.

- 3. **Pre-specified subgroups** — from the clinical spread the profile reveals (target volume,

- slice thickness, modality). Pre-specifying them here is what separates a subgroup finding from a

- post-hoc one.

- 4. **Where the held-out set comes from** — especially when the shipped "test" directory is unlabelled.

- 5. **What the cohort cannot support** — n, single-source acquisition, absent subgroups. This is the

- honest seed of the Limitations paragraph, written before the results can bias it.

-

- Record these in the study record so `/design-study`, `/preprocess-imaging`, and eventually

- `/write-paper` inherit them rather than re-deriving them.

-

- ## Outputs

-

- - `eda/profile.json` — per-case dataset profile (the artifact downstream skills read).

- - `qc/dataset_profile.json` — deterministic audit with verdicts.

- - Decision notes for the study record (resampling target, loss/metric family, pre-specified

- subgroups, held-out provenance, cohort limitations).

-

- ## Forbidden

-

- - Reporting a profile figure that was not computed from the files (no remembered spacings, no

- assumed label indices — open the labels and look).

- - Declaring an audit pass without running the gate.

- - Using a split's images as a held-out test set when the profile says it has no labels.

- - Reading `--spacing-ratio` / `--imbalance-frac` defaults as published thresholds.

-

- ## Anti-Hallucination

-

- - **Never report a profile figure that was not computed from the files.** Spacings, label indices,

- foreground fractions and organ volumes come from opening every image and label — not from a

- dataset's documentation, not from what a similar dataset looked like, and not from memory. A

- dataset's README can be wrong about its own label indices; the labels cannot.

- - **Never report a profile audit "pass" without running `check_dataset_profile.py`.** The verdicts

- are re-derived from the profile JSON by rule and arithmetic; a prose claim that the data "looks

- fine" is not the audit.

- - **Never treat an unlabelled split as a test set.** If the profile says a split has no labels, no

- held-out metric can come from it, however the directory is named.

- - **Never present `--spacing-ratio` / `--imbalance-frac` as published cut-points.** They are

- screening defaults; the values applied are printed in the output and belong in the Methods.

-

- ## Validation

-

- ```bash

- python3 scripts/check_dataset_profile.py --profile <profile.json> --strict

- bash scripts/check_dataset_profile_challenge/verify.sh # deterministic, network-free

- bash tests/test_dataset_profile.sh

- ```