leakage-and-splits SKILL.md skillA
leakage-and-splits from uchicago-dsi/ai-sci-skills is an agent skill: a SKILL.md folder an AI agent loads when a task calls for it.
mdr add uchicago-dsi/ai-sci-skills/leakage-and-splits@git:20260921.9b3de7cPins git:20260921.9b3de7c in mdr.lock. Pin the fingerprint instead.
- Audit grade
- A, 17 of 17 checks passed
- Kind
- SKILL.md skill
- Repository
- uchicago-dsi/ai-sci-skills
- Path
skills/leakage-and-splits/SKILL.md- Versions recorded
- 1
- Latest version
- git:20260921.9b3de7c, committed Sep 21, 2026
- Size
- 4,171 bytes
- Repository stars
- 20
- License
- MIT
Keep held-out data genuinely held out: assign split eligibility to whole source groups, count independent units rather than rows, and check whether a split was already exposed by earlier fitting. Use when defining, reusing, or claiming a train/validation/test split, or when reporting an n.
Collected from public GitHub. Every version is saved exactly as it was and never changes. Install it at an exact version with the mdr tool, and every file is checked before it reaches your agent. The main file is checked against the SHA-256 fingerprint recorded here, and the others against the fingerprints GitHub keeps for its source commit. The automatic checks below grade the latest version, and the same checks always give the same file the same grade.
What the file says
# Leakage And Splits The vocabulary below is generic on purpose. A "source group" is whatever repeats: a participant, a site, an animal, a batch, a sensor, a time period, a physical specimen. ## Hold Out Whole Groups, Not Rows - A unit reported as held out must contribute neither gradient nor model/optimizer/checkpoint selection. If an outer objective uses data excluded only from an inner solve, call it adaptation or outer-training data and reserve a separate untouched evaluation split. - For longitudinal or repeated-measure analyses, assign split eligibility to the whole participant/source group before forming pairs. If any visit is held out, exclude every visit in that group from training. Define sufficiency, permutation, and holdout counts in independent groups rather than in visits or pair combinations. - Treat raw source identity, not an analysis-row identifier, as the unit. When rows or annotations reuse a source, record that identity, keep splits source-exclusive, report within-source envelopes, and balance cohort summaries over sources. ## Do Not Inflate The Denominator - Do not count overlapping windows, crops, augmentations, repeated rows, or …
Read the whole file at its exact version.
How to install
mdr add uchicago-dsi/ai-sci-skills/leakage-and-splits@sha256:518eb2699511e207The line at the top of this page pins the label git:20260921.9b3de7c, which is what most people want. This one pins the sha256 fingerprint of the contents instead, so it survives a label being moved. Either way the exact version is written to mdr.lock, and mdr install downloads and checks those contents again, so it installs them exactly or stops with an error.
[](https://markdownregistry.com/a/art_d7vrdn4sbpoy52o5)
4 badge image fetches in 30 days. What the badge shows.
Versions
The registry has recorded 1 version of this file, committed on Sep 21, 2026.
Audit of the latest version
- pass: Frontmatter block present
- pass: Frontmatter declares a name
- pass: Frontmatter declares a description
- pass: Size between 200 bytes and 200 KB (4171 bytes)
- pass: No zero-width or bidi control characters
- pass: No instruction hidden inside an HTML comment
- pass: No link to an exfiltration or paste host
- pass: No credential-shaped string
- pass: No instruction to send local credentials anywhere
- pass: No text hidden with inline styles
- pass: No prompt-injection phrasing
- pass: No curl or wget piped into a shell
- pass: No recursive delete of root, home or parent
- pass: No instruction to read or print local credentials
- pass: No base64 blob over 200 characters
- pass: No link to a raw IP address
- pass: No script tag
Source
uchicago-dsi/ai-sci-skills · 20 stars · license MIT · pushed 2026-11-07 · branch main
API
GET https://markdownregistry.com/api/v1/artifacts/art_d7vrdn4sbpoy52o5 GET https://markdownregistry.com/api/v1/resolve?ref=uchicago-dsi/ai-sci-skills/leakage-and-splits GET https://markdownregistry.com/api/v1/blob/518eb2699511e2071acf3f558bf0fc2cab3d4ca411b8e396880a311a9d628451
Your AI assistant does the legwork. You hear about the deals worth your word. Hand yours the standing instructions at modelranch.com and it joins the network that reads files like this one.
Guides: the SKILL.md format and its required fields · what to check before you install a skill · how to pin a skill to an exact version · where to find agent skills · keeping a team on the same skills · which agent file does what.