leakage-and-splits skillA
leakage-and-splits is agent-read markdown (skill) from uchicago-dsi/ai-sci-skills: Keep held-out data genuinely held out: assign split eligibility to whole source groups, count independent units rather than rows, and check whether a split was already exposed by earlier fitting. Use when defining, reusing, or claiming a train/validation/test split, or when reporting an n..
Indexed from public GitHub and served as immutable, content-addressed versions. Install it pinned to an exact SHA-256 with the mdr CLI, and every file is verified against the hash recorded here before it reaches your agent. The deterministic audit below grades the latest version, and the same file always earns the same grade.
What the file says
# Leakage And Splits The vocabulary below is generic on purpose. A "source group" is whatever repeats: a participant, a site, an animal, a batch, a sensor, a time period, a physical specimen. ## Hold Out Whole Groups, Not Rows - A unit reported as held out must contribute neither gradient nor model/optimizer/checkpoint selection. If an outer objective uses data excluded only from an inner solve, call it adaptation or outer-training data and reserve a separate untouched evaluation split. - For longitudinal or repeated-measure analyses, assign split eligibility to the whole participant/source group before forming pairs. If any visit is held out, exclude every visit in that group from training. Define sufficiency, permutation, and holdout counts in independent groups rather than in visits or pair combinations. - Treat raw source identity, not an analysis-row identifier, as the unit. When rows or annotations reuse a source, record that identity, keep splits source-exclusive, report within-source envelopes, and balance cohort summaries over sources. ## Do Not Inflate The Denominator - Do not count overlapping windows, crops, augmentations, repeated rows, or …
Read the whole file at its exact version.
How to install
mdr add uchicago-dsi/ai-sci-skills/leakage-and-splits@git:20260921.9b3de7cmdr add uchicago-dsi/ai-sci-skills/leakage-and-splits@sha256:518eb2699511e207Pin to a label to follow the author's releases, or to a sha256 to freeze the exact bytes forever. Either way the resolved hash is written to mdr.lock, and mdr install reproduces it on any machine.
[](https://markdownregistry.com/a/art_d7vrdn4sbpoy52o5)
1 badge views in 30 days
Versions
Audit of the latest version
- pass: Frontmatter block present
- pass: Frontmatter declares a name
- pass: Frontmatter declares a description
- pass: Size between 200 bytes and 200 KB (4171 bytes)
- pass: No zero-width or bidi control characters
- pass: No instruction hidden inside an HTML comment
- pass: No link to an exfiltration or paste host
- pass: No credential-shaped string
- pass: No instruction to send local credentials anywhere
- pass: No text hidden with inline styles
- pass: No prompt-injection phrasing
- pass: No curl or wget piped into a shell
- pass: No recursive delete of root, home or parent
- pass: No instruction to read or print local credentials
- pass: No base64 blob over 200 characters
- pass: No link to a raw IP address
- pass: No script tag
Source
uchicago-dsi/ai-sci-skills · 20 stars · license MIT · pushed 2026-09-23 · branch main
API
GET https://markdownregistry.com/api/v1/artifacts/art_d7vrdn4sbpoy52o5 GET https://markdownregistry.com/api/v1/resolve?ref=uchicago-dsi/ai-sci-skills/leakage-and-splits GET https://markdownregistry.com/api/v1/blob/518eb2699511e2071acf3f558bf0fc2cab3d4ca411b8e396880a311a9d628451
Your agent does the legwork. You hear about the deals worth your word. Hand yours the standing instructions at modelranch.com and it joins the network that reads files like this one.