git:20260531.76c295c to git:20260930.913abeb

28 added, 93 removed. Audit A to A.

---

name: version-dataset

description: Dataset version control for research reproducibility. Builds a deterministic content-hash manifest of a dataset (file SHA-256 + tabular schema + per-column value hashes), verifies a later copy against it to detect drift (schema change, row-count change, value changes), and diffs two manifests. Use to prove an analysis ran on the intended data, lock a dataset version, or reproducibility-lock bundled demos.

triggers: version dataset, dataset version, data manifest, data hash, dataset drift, reproducibility lock, verify dataset, data provenance, did my data change, manifest.lock

tools: Read, Write, Edit, Bash, Grep, Glob

model: inherit

---

# Version Dataset Skill

- You help a medical researcher put a dataset under version control: fingerprint it,

- detect when it changes, and lock a reproducible version. This guards the

- data-integrity rule — an analysis must run on the data it claims to, with a fixed

- seed — by making any drift between runs loud instead of silent.

- ## Communication Rules

- - Communicate with the user in their preferred language.

- - Manifest fields, drift reports, and provenance notes are in English.

- ## Philosophy

- A dataset is an input to a result; if it changes silently, every downstream

- number is suspect. This skill records a deterministic fingerprint (file SHA-256 +,

- for tabular files, schema and per-column value hashes) so a later run can *prove*

- the inputs are unchanged. It does not alter data, and it records nothing

- non-deterministic (no timestamps unless explicitly passed), so the same data

- always yields the same manifest.

- ## Reference Files

- - **Manifest schema + workflow**: `${CLAUDE_SKILL_DIR}/references/manifest_schema.md` —

- the manifest.json structure, what each drift category means, and the non-

- deterministic-artifact policy (PPTX/DOCX timestamps). Read before interpreting drift.

+ An analysis must run on the data it claims to, with a fixed seed. This skill makes drift between

+ runs loud instead of silent: it records a deterministic fingerprint (file SHA-256 plus, for

+ tabular files, schema and per-column value hashes; no timestamp unless explicitly passed) so a

+ later run can *prove* the inputs are unchanged. It never alters data, and manifests hold hashes,

+ not the data itself. Provenance notes are in English.

## Deterministic Script

```bash

# Build a manifest (record the analysis seed + provenance)

python "${CLAUDE_SKILL_DIR}/scripts/version_dataset.py" manifest data.csv \

--out manifest.json --seed 42 --provenance "KNHANES 2018 extract v1"

# Verify a later copy against it (CI / pre-analysis gate)

python "${CLAUDE_SKILL_DIR}/scripts/version_dataset.py" verify --manifest manifest.json --strict

# Compare two manifests (what changed between versions)

python "${CLAUDE_SKILL_DIR}/scripts/version_dataset.py" diff --old v1.json --new v2.json

```

File hashing is stdlib-only; tabular schema/column hashing uses pandas when present.

`--ignore-cols` excludes volatile columns; `--base` makes manifest keys relative.

## Workflow

### Step 1: Lock the version (gate)

Build the manifest at the moment the dataset is frozen for analysis. **Gate:**

confirm with the user the seed and provenance note are correct before locking —

the manifest is the record they will cite as "this is the data the results came from."

+ The provenance note is the user's text; never invent it.

### Step 2: Verify before each run (gate)

- Before re-running an analysis (or in CI), `verify --strict`. **Gate:** if drift is

- reported, stop and show the user the drift report; do not proceed on changed data

- without their explicit acknowledgement and a re-lock. Silent re-run on drifted data

- is the failure this skill exists to prevent.

+ Before re-running an analysis (or in CI), `verify --strict`. Never say a dataset is unchanged

+ without running `verify`. **Gate:** if drift is reported (`MANIFEST_DRIFT`, exit 1), stop and

+ show the user the drift report exactly as computed — never downplay a changed column hash. Do not

+ proceed on changed data without their explicit acknowledgement and a re-lock. Silent re-run on

+ drifted data is the failure this skill exists to prevent.

+ Read `${CLAUDE_SKILL_DIR}/references/manifest_schema.md` before interpreting drift — it defines

+ each drift category. A tabular file is compared on its logical content (schema + per-column

+ value hashes), not raw bytes, so re-saving the same data, reordering columns, or an

+ `--ignore-cols` volatile column does not trip a false drift.

### Step 3: Diff across versions

When a dataset is intentionally updated, `diff` the old and new manifests and

present the change set (added/removed/changed columns, row-count delta) so the

user can record what changed and re-lock. **Gate:** the user approves the new

version before it replaces the locked one.

## Non-Deterministic Artifacts

- Some outputs (PPTX/DOCX with embedded timestamps, figures with render metadata)

- change byte-for-byte on every build even when the analysis is identical. Do not

- put these under strict byte verification — manifest only the deterministic inputs

- and tabular outputs (data files, result CSVs), or use `--ignore-cols` for volatile

- columns. See references for the policy.

- ## Scope Limitations

- ### Supported

- - Content-hash manifest of any file; schema + per-column hashes for tabular files

- (CSV/TSV/Parquet/Stata/SAS/Excel).

- - Drift verification and manifest-to-manifest diff.

- ### NOT Supported

- - Storing or transmitting the data itself (manifests hold hashes, not contents).

- - Cleaning, profiling, or de-identifying — use `/clean-data`, `/generate-codebook`, `/deidentify`.

- - Full pipeline-output reproducibility for non-deterministic binaries (see above).

- ## Cross-Skill Integration

- - **/generate-codebook** documents *what* is in the data; version-dataset locks *which version*.

- - **/deidentify** should run before a manifest is shared (example values are not stored, but provenance notes may carry context).

- - Demo reproducibility: each bundled `demo/*/` carries a `manifest.lock.json` (input data + deterministic result tables) that `verify --strict` checks.

- ## Worked Example

- Lock a freshly-frozen extract:

- ```bash

- python "${CLAUDE_SKILL_DIR}/scripts/version_dataset.py" manifest cohort.csv \

- --out manifest.json --seed 42 --provenance "KNHANES 2018 extract, frozen 2026-05"

- # -> {"files": 1, "out": "manifest.json"}

- ```

- Before re-running the analysis next month:

- ```bash

- python "${CLAUDE_SKILL_DIR}/scripts/version_dataset.py" verify --manifest manifest.json --strict

- # OK: 1 file(s) match the manifest. (exit 0 — safe to run)

- ```

- If someone silently re-exported the data with three extra rows:

- ```text

- =========================================

- Dataset Manifest Verify

- =========================================

- DRIFT (3):

- ROW COUNT cohort.csv: 3457 -> 3460

- CHANGED column cohort.csv:bmi

- CHANGED column cohort.csv:hba1c

- MANIFEST_DRIFT: dataset differs from manifest. (exit 1 — STOP)

- ```

- The analysis does **not** proceed: the result the manuscript will cite would no

- longer match the locked data. The researcher reviews the drift, decides whether

- the change is intended, and only then re-locks (`manifest` again) and records the

- new provenance. A tabular file is compared on its **logical content** (schema +

- per-column value hashes), not raw bytes — re-saving the same data, reordering

- columns, or an `--ignore-cols` volatile timestamp column does not trip a false drift.

+ Do not put PPTX/DOCX or figure binaries under strict byte verification — they embed timestamps

+ or render metadata and change on every build. Manifest only the deterministic inputs and tabular

+ outputs (data files, result CSVs), or use `--ignore-cols` for volatile columns; the policy is in

+ `references/manifest_schema.md`. Each bundled `demo/*/` carries a `manifest.lock.json` (input data

+ + deterministic result tables) that `verify --strict` checks; the reference gives the demo

+ command.

- ## Anti-Hallucination

+ ## Scope

- - Never claim a dataset is unchanged without running `verify`.

- - Manifests record only observed hashes/schema; no provenance is invented — the

- `provenance` note is user-supplied text.

- - Report drift exactly as computed; do not downplay a changed column hash.

+ - Run `/deidentify` before a manifest is shared: example values are not stored, but provenance

+ notes may carry context.

+ - Cleaning, profiling or de-identifying the data → `/clean-data`, `/generate-codebook`,

+ `/deidentify`. `/generate-codebook` documents *what* is in the data; this skill locks *which

+ version*.