dataset-split-review git:20260605.b3dee57A

Immutable. This exact content is served forever at /api/v1/blob/ef9af27be7519c98.

---
name: dataset-split-review
description: Use when reviewing how data is split into train/validation/test sets -- especially with time-series data, repeated entities (users, patients, sessions), or imbalanced targets.
---

## Purpose

A random split is often the wrong split. Incorrect splitting causes massive overestimation of model performance; this review verifies the split methodology matches the structure of the data.

## When to Use

- Reviewing splitting code before training or sign-off
- Data has a time component, repeated entities, or class imbalance
- Validation metrics look too good for the problem

## Inputs

- The splitting code
- The dataset's structure: time column? entity keys (user/patient/session)? target balance?

## Workflow

1. **Time-series data**: if the data has a time component, random `train_test_split` is strictly forbidden. Require a chronological split so the model cannot learn from the future.
2. **Group leakage**: if the dataset has multiple rows per user/patient/session, a standard split puts rows from the same entity in both train and test. Require GroupKFold or group-based splitting.
3. **Stratification**: for imbalanced targets, verify stratification maintains the target distribution across all splits.
4. **Verify in code, not in description**: read the actual splitting call and its parameters before issuing a verdict.

## Output

- An explicit pass/fail verdict on Time, Group, and Stratification safety, citing the splitting code

## Verification

- [ ] Time safety checked (chronological split whenever a time component exists)
- [ ] Group safety checked (no entity appears in both train and test)
- [ ] Stratification checked for imbalanced targets
- [ ] Verdict cites the actual splitting code, not the author's description of it

## Failure Modes

- **Trusting the description** — authors often believe the split is grouped or chronological when the code says otherwise. Read the code.
- **Random split on time data** — silently inflates metrics; flag as blocking, not advisory.
- **Partial grouping** — grouped train/test but a random validation split still leaks. Check every split boundary.