dataset-split-review · diff
git:20260602.4470d2c to git:20260605.b3dee57
37 added, 8 removed. Audit A to A.
---
name: dataset-split-review
- description: "Audit the methodology used to split data into train, validation, and test sets."
+ description: Use when reviewing how data is split into train/validation/test sets -- especially with time-series data, repeated entities (users, patients, sessions), or imbalanced targets.
---
- # Dataset Split Review
+ ## Purpose
- A random split is often the wrong split. Incorrect splitting causes massive overestimation of model performance.
+ A random split is often the wrong split. Incorrect splitting causes massive overestimation of model performance; this review verifies the split methodology matches the structure of the data.
- ## Review Protocol
+ ## When to Use
- 1. **Time-Series Data**: If the data has a time component, `train_test_split` is strictly forbidden. You must use a chronological split to prevent the model from learning the future.
- 2. **Group Leakage**: If the dataset has multiple rows for a single user/patient/session, a standard split will put rows from the same user in both train and test. You must use GroupKFold or group-based splitting.
- 3. **Stratification**: For imbalanced datasets, verify that stratification is used to maintain the target distribution across all splits.
- 4. **Action**: Review the splitting code and explicitly verify Time, Group, and Stratification safety.
+ - Reviewing splitting code before training or sign-off
+ - Data has a time component, repeated entities, or class imbalance
+ - Validation metrics look too good for the problem
+
+ ## Inputs
+
+ - The splitting code
+ - The dataset's structure: time column? entity keys (user/patient/session)? target balance?
+
+ ## Workflow
+
+ 1. **Time-series data**: if the data has a time component, random `train_test_split` is strictly forbidden. Require a chronological split so the model cannot learn from the future.
+ 2. **Group leakage**: if the dataset has multiple rows per user/patient/session, a standard split puts rows from the same entity in both train and test. Require GroupKFold or group-based splitting.
+ 3. **Stratification**: for imbalanced targets, verify stratification maintains the target distribution across all splits.
+ 4. **Verify in code, not in description**: read the actual splitting call and its parameters before issuing a verdict.
+
+ ## Output
+
+ - An explicit pass/fail verdict on Time, Group, and Stratification safety, citing the splitting code
+
+ ## Verification
+
+ - [ ] Time safety checked (chronological split whenever a time component exists)
+ - [ ] Group safety checked (no entity appears in both train and test)
+ - [ ] Stratification checked for imbalanced targets
+ - [ ] Verdict cites the actual splitting code, not the author's description of it
+
+ ## Failure Modes
+
+ - **Trusting the description** — authors often believe the split is grouped or chronological when the code says otherwise. Read the code.
+ - **Random split on time data** — silently inflates metrics; flag as blocking, not advisory.
+ - **Partial grouping** — grouped train/test but a random validation split still leaks. Check every split boundary.
+