git:20260808.245a9ea to git:20260817.a1ba97b

---

name: research-repo

description: Scaffold or audit an entire research project repository organized around its source library. Use whenever the user is starting, structuring, organizing, or reviewing a whole project — "set up a research repo", "how should I structure/organize this project", "initialize my sources folder", "new paper or literature-review project", "audit my repo structure", "is my sources folder set up right", "check my project layout". Builds the sources spine (original PDFs, tracked Markdown conversions, a drop zone, and the bibliography), the conversion and intake pipeline, and the archetype-appropriate analysis, manuscript, and review folders; or audits an existing repo and reports what is present, partial, or missing. NOT for intaking or converting a single PDF (use process-source) or building a publication replication package (use replication-package).

argument-hint: "[path to the research repo; defaults to the current directory]"

allowed-tools:

- Read

- Write

- Edit

- Bash

---

# Research Repository Scaffold & Audit

## Scope and organizing principle

This skill sets up a new research repository, or audits an existing one, around a single organizing principle: **the source library is the spine of the project, and everything else grows from it.**

`sources/og/` holds the original PDFs and documents — archival, **gitignored** (size + copyright), rarely read directly. `sources/md/` holds their Markdown conversions, **tracked in git**: this is the LLM-readable knowledge base, and *sources are always read from here, not from the PDFs*. `sources/unprocessed/` is the drop zone where new PDFs land until they are named, converted, and filed. `sources/references.bib` keys the corpus to the manuscript — one entry per source, author+year resolvable to its `md/` file, so a `\cite` key maps to a document actually read. A conversion script (`scripts/convert-sources.sh`, OpenDataLoader PDF) and a `process-source` intake command keep the three in sync. Get that spine wrong — PDFs scattered, no Markdown, a bib that does not match what was read — and every downstream skill (`literature-review`, `citation-check`, `fact-check`, `paper-tex`) is working on sand.

Everything else — `data/`, analysis `scripts/`, `manuscript/` or `paper/`, `review/` + `codebook/`, `figures/`, `tables/`, `replication/` — grows outward from that spine, conditioned on what kind of project this is.

## Instructions

### Step 1. Resolve the target and decide the mode

Use `$ARGUMENTS` as the repo path if given; otherwise default to the current working directory (confirm with the user once before writing anything into it). Then decide the mode from what is there:

- **Scaffold mode** — the directory is empty, is not yet a git repo, or has no `sources/` and no manuscript. You will build the spine and the outward folders.

- **Audit mode** — the directory already has research content (a `sources/` tree, a manuscript, analysis scripts, a bib). You will read what exists, compare it against the convention, and report present / partial / missing. **Never overwrite an existing file without explicit confirmation**; offer to create only what is missing.

If a repo is half-built (e.g. PDFs exist but no `md/`, or an `og/` with no convert script), that is audit mode with scaffolding gaps — report the gaps and offer to fill them.

### Step 2. Identify the project archetype

The bibliography is the spine of every project; the **outward** folders — and whether a PDF corpus is even appropriate — depend on what the project is. Detect the archetype from concrete signals in the directory, or ask one question if it is genuinely ambiguous.

| Archetype | Concrete signals | Outward folders to scaffold |

|-----------|------------------|------------------------------|

| **Literature / systematic review** | many `sources/md/` files; a coding protocol; an inventory CSV | `review/` (inventory CSV + per-source annotations), `codebook/` (evaluation protocol), `data/` (open datasets collected) |

| **Empirical analysis / paper** | data files (`*.csv`, `*.dta`) beside estimation scripts (`*.R`, `*.do`, `*.py`); a manuscript heading to a journal | `data/`, `scripts/` (analysis), `paper/` or `manuscript/`, `figures/`, `tables/`, `replication/` |

| **Corpus / digitization** | scanned images / OCR output; a large derived corpus | `data/` (raw + derived corpus), `scripts/` (OCR/cleanup pipeline), plus `sources/` for the methods literature |

| **Lightweight / theory paper** | a manuscript and a `.bib` but no archived PDFs and no large dataset | a `paper/` or `manuscript/` folder and its `.bib`; **no `sources/` PDF corpus** unless the user asks for one |

| **Mixed** | several of the above | scaffold the union; keep one shared bibliography |

Most projects have a literature corpus **and** head to a paper; when in doubt, scaffold the spine plus `data/`, `scripts/`, and the manuscript folder, and let the project grow the rest. But some repos legitimately keep no PDF corpus — a theory paper, or a short empirical paper with a hand-maintained `.bib`. There the spine is the bibliography alone: audit the `.bib` wherever it lives (often `paper/references.bib`) and *offer* — never impose — a `sources/og`/`sources/md` tree.

### Step 3. Scaffold the sources spine

For any project that reads and archives a literature, this is the core. (For a corpus-free repo — the lightweight/theory archetype in Step 2 — scaffold only the bibliography and the manuscript folder, then skip to Step 4.) Create:

```text

<repo>/

├── sources/

│ ├── og/ # original PDFs/docs — gitignored

│ ├── md/ # Markdown conversions — tracked

│ ├── unprocessed/ # drop zone for new PDFs

│ ├── references.bib # bibliography (one entry per source)

│ └── README.md # the convention, written down

├── scripts/

├── .claude/

│ └── commands/

│ └── process-source.md # per-PDF intake command

├── CLAUDE.md # project conventions for coding agents

├── AGENTS.md -> CLAUDE.md # symlink so Codex/others read the same file

├── .gitignore

└── .venv/ # python env for opendataloader-pdf (created in setup)

```

**First, make sure the repo is under version control** — the whole tracked/gitignored split (`og/` ignored, `md/` tracked) only takes effect once git exists:

```bash

git rev-parse --git-dir >/dev/null 2>&1 || git init

```

Write the templates from the **Templates** section below. A file written by the `Write` tool is mode 0644, so **make the conversion script executable immediately after writing it** — otherwise `./scripts/convert-sources.sh` fails with "permission denied", both in the smoke test below and in the two places the generated scaffold tells the user to run it (`sources/README.md` step 2 and `.claude/commands/process-source.md` step 3):

```bash

chmod +x scripts/convert-sources.sh

```

Then set up the conversion environment — but verify the toolchain first, because `convert-sources.sh` runs under `set -euo pipefail` and will abort opaquely if the Java backend is missing:

```bash

command -v python3 && python3 --version

command -v java && java -version # OpenDataLoader PDF needs Java 11+; if absent, stop and tell the user to install it

python3 -m venv .venv

.venv/bin/pip install --upgrade pip opendataloader-pdf

```

Link `AGENTS.md` to `CLAUDE.md` so a second agent family (e.g. Codex) reads the same root instructions:

```bash

test -e AGENTS.md || ln -s CLAUDE.md AGENTS.md

```

Finally, smoke-test the pipeline — on an empty `og/` it should print `Nothing new to convert.`:

```bash

./scripts/convert-sources.sh

```

### Step 4. Build outward (archetype-conditioned)

Create only the outward folders the archetype calls for (Step 2). Leave them empty with a one-line purpose in the README — the project fills them. Common folders and what they hold:

- **`data/`** — datasets. Raw inputs and derived analysis-ready files. Large/binary data is gitignored (see template); document restricted data rather than committing it.

- **`scripts/`** — analysis and utility code (the convert script already lives here).

- **`review/`** + **`codebook/`** (review projects) — the inventory CSV (one row per source), per-source prose annotations, and the coding protocol that governs them. This is where the `sources/md/` corpus gets turned into structured evidence.

- **`manuscript/`** or **`paper/`** — the draft. Cites `sources/references.bib`. Hand this to `paper-tex` to typeset.

- **`figures/`**, **`tables/`** — generated outputs. Pair with the `figures` / `tables` skills.

- **`replication/`** — the public reproducibility package. **Do not hand-build this here** — when the paper is ready, call `replication-package` to scaffold it properly.

- **`logs/`**, **`meetings/`** — session logs and notes; usually local-only (gitignored).

### Step 5. Document the intake pipeline

The spine only stays trustworthy if every new source flows through the same pipeline. Write it into `sources/README.md` and the `process-source` command, and state it to the user:

```

drop in sources/unprocessed/ → identify (title/authors/year/venue)

→ rename to author-year-slug, move to sources/og/

→ ./scripts/convert-sources.sh (→ sources/md/<name>.md)

→ add a BibTeX entry to sources/references.bib

→ (review projects: add inventory row + annotation)

```

The per-PDF mechanics belong to `process-source`, which the scaffolded `process-source.md` command points at. This skill guarantees the pipeline *exists and is wired*; `process-source` *runs* it.

### Step 6. The BibTeX contract

The bibliography — `sources/references.bib`, or wherever the project keeps it (e.g. `paper/references.bib`) — is the contract between the manuscript and the knowledge base: a `\cite` key is only trustworthy if it resolves to a source you have actually read and filed in `sources/md/`. Keep it honest:

- **One entry per source**, added at intake (Step 5), never in a batch at the end.

- **Citekey** follows the project's own key style (e.g. `hainmueller_hopkins_yamamoto_2014` or `hainmueller-etal-2014-conjoint`). Do not impose a scheme on an existing project — but keep every key **author+year resolvable to its `md/` filename**, because `citation-check` and `fact-check` map keys to source files by author and year. The filename uses `author-year-slug`; the bib key can differ in punctuation but must point at the same work.

- **`sources/missing.bib`** (optional, recommended) — a second bib for works that are cited but have **no PDF in `og/`**: paywalled articles with no preprint mirror, books, dissertations, authoritative web resources (a standard, a DOI registry). Record why each is missing and how to acquire it, so "cited but unfiled" stays visible instead of silently absent. A `needs_updates.md`-style flag file serves the same purpose if the project already keeps one.

### Step 7. Audit checklist (audit mode)

Read the existing repo and report each item as **present / partial / missing**. If the repo keeps no PDF corpus by design (the lightweight/theory archetype), mark the PDF-corpus items **n/a for this archetype**, not *missing*, and audit the bibliography wherever it actually lives — discover it with `find . -name '*.bib' -not -path './.venv/*' -not -path './.git/*'`.

**Spine**

- [ ] `sources/og/`, `sources/md/`, `sources/unprocessed/` all exist.

- [ ] `sources/og/` is gitignored; `sources/md/` is tracked (the LLM-readable corpus must be in git).

- [ ] Every `og/*.pdf` has a matching `md/*.md` (no unconverted sources). List the orphans.

- [ ] Every `md/*.md` has a `references.bib` entry, and every bib entry resolves to a source (flag bib entries with no file and files with no bib — *drift in either direction*).

- [ ] Filenames follow `author-year-slug` (lowercase, hyphens, ≤3 authors then `firstauthor-etal`).

- [ ] `sources/references.bib` exists; keys are author+year resolvable to filenames.

- [ ] `sources/README.md` documents the convention.

**Detection recipes** (run from the repo root; approximate starting points, not gospel):

```bash

# Orphan PDFs — in og/ but never converted to md/

comm -23 <(cd sources/og && ls *.pdf 2>/dev/null | sed 's/\.pdf$//' | sort) \

<(cd sources/md && ls *.md 2>/dev/null | sed 's/\.md$//' | sort)

# Bib keys — compare against md/ stems by author+year to catch drift in either

# direction (a source with no entry, an entry with no source). Keys may use

# underscores where filenames use hyphens.

grep -oE '^@[a-zA-Z]+\{[^,]+' sources/references.bib | sed 's/^@[a-zA-Z]*{//' | sort

# Filenames that violate author-year-slug (lowercase-hyphen, four-digit year, slug)

ls sources/og | grep -vE '^[a-z0-9]+(-[a-z0-9]+)*-(19|20)[0-9]{2}-[a-z0-9-]+\.(pdf|docx)$'

```

**Pipeline**

- [ ] `scripts/convert-sources.sh` exists and points at this repo's `sources/og` → `sources/md`.

- [ ] `.venv/` with `opendataloader-pdf` is set up (or setup is documented); Java 11+ available.

- [ ] A `process-source` command (or the global skill) is available for intake.

**Repo conventions**

- [ ] `CLAUDE.md` documents the project and its conventions; `AGENTS.md` is a symlink to it (or a copy).

- [ ] `.gitignore` excludes `sources/og/`, `.venv/`, large data, secrets, and OS/editor cruft.

- [ ] Outward folders match the archetype and have a stated purpose (README).

- [ ] No secrets, answer keys, or restricted data in the tracked path.

### Step 8. Report

Output a short report:

1. Mode (scaffold / audit) and detected archetype.

2. The tree created (scaffold) or the present/partial/missing diff (audit).

3. For audits: orphan PDFs, bib drift, naming violations, and pipeline gaps, each with a one-line fix.

4. The next three actions (typically: set up `.venv` and run `convert-sources.sh`; drop the first PDFs in `unprocessed/` and run `process-source`; fill in `CLAUDE.md` placeholders).

Do not commit unless asked.

## Templates

### `sources/README.md`

````markdown

# Sources

The source library is the spine of this project. Read sources from `md/`, never the PDFs.

## Structure

- `og/` — original PDFs and documents (gitignored; not pushed)

- `md/` — LLM-readable Markdown conversions (tracked in git)

- `<name>_images/` folders hold figures/tables extracted from the PDF

- `unprocessed/` — drop zone for new PDFs awaiting intake

- `references.bib` — bibliography (one entry per source)

- `missing.bib` — cited works with no PDF available (paywalled, books, web resources)

## Naming convention

`author-year-slug`, lowercase with hyphens:

- Up to three authors → use all three; more → `firstauthor-etal`

- Then year, then a 2–4 word content slug

- Example: `hainmueller-hopkins-yamamoto-2014-causal-inference-conjoint`

The PDF in `og/` carries this name through to `md/`.

## Adding a source

1. Drop the PDF (or `.docx`) into `sources/unprocessed/`

2. Run the `process-source` command (or do it by hand): rename to `author-year-slug`,

move to `og/`, run `./scripts/convert-sources.sh`, add a `references.bib` entry

3. The convert script only touches files without a matching `.md` — re-running is safe

4. Commit the new `md/<name>.md` (and its `_images/`); `og/` stays local

## Requirements

- Python venv at `.venv/` with `opendataloader-pdf` installed

```bash

python3 -m venv .venv

.venv/bin/pip install --upgrade pip opendataloader-pdf

```

## Why Markdown?

PDFs are awkward for LLMs. The Markdown versions preserve text, headings, and

references in a form agents can read directly, enabling source-grounded analysis

and writing. OpenDataLoader PDF: https://github.com/opendataloader-project/opendataloader-pdf

````

### `scripts/convert-sources.sh`

```bash

#!/usr/bin/env bash

# Convert new PDFs/docx in sources/og/ to Markdown in sources/md/.

# Skips files that already have a corresponding .md.

# Usage: ./scripts/convert-sources.sh # incremental (new files only)

# ./scripts/convert-sources.sh --all # reconvert everything

set -euo pipefail

PROJECT_ROOT="$(cd "$(dirname "$0")/.." && pwd)"

SOURCES="$PROJECT_ROOT/sources/og"

OUTPUT="$PROJECT_ROOT/sources/md"

VENV="$PROJECT_ROOT/.venv"

FORCE=false

[[ "${1:-}" == "--all" ]] && FORCE=true

mkdir -p "$OUTPUT"

source "$VENV/bin/activate"

converted=0

# PDFs via OpenDataLoader PDF

for pdf in "$SOURCES"/*.pdf; do

[ -f "$pdf" ] || continue

base="$(basename "$pdf" .pdf)"

if [[ "$FORCE" == false && -f "$OUTPUT/$base.md" ]]; then

continue

fi

echo "Converting $base.pdf..."

opendataloader-pdf "$pdf" --format markdown --output-dir "$OUTPUT/"

converted=$((converted + 1))

done

if [[ "$FORCE" == false && -f "$OUTPUT/$base.md" ]]; then

continue

fi

converted=$((converted + 1))

done

echo "Nothing new to convert."

else

fi

```

### `.claude/commands/process-source.md`

```markdown

---

description: Process new papers from sources/unprocessed/ through the full intake pipeline

---

# Process unprocessed sources

Check `sources/unprocessed/` for new PDFs/docs. For each one:

1. **Identify** — read it; determine title, authors, year, venue.

2. **Rename & move** — to `sources/og/` using `author-year-slug` (see `sources/README.md`).

3. **Convert** — run `./scripts/convert-sources.sh` to generate `sources/md/<name>.md`.

4. **BibTeX** — add an entry to `sources/references.bib` (author+year resolvable to the filename).

5. **(Review projects)** — classify per `codebook/` and add an inventory row + annotation.

6. **Clean up** — remove the file from `sources/unprocessed/`.

Flag anything needing human review (paywalled, poor OCR, data-integrity concerns).

If a cited work has no obtainable PDF, record it in `sources/missing.bib` instead.

Prefer this project's conventions; defer to the global `process-source` skill for the generic mechanics.

```

### `CLAUDE.md`

```markdown

# <Project name>

<One paragraph: what this project is, who it is for, and its current phase.>

## Repository structure

The source library is the spine. See `sources/README.md` for the intake convention.

- `sources/md/` — LLM-readable Markdown corpus. **Always read sources from here, not the PDFs.**

- `sources/unprocessed/` — drop zone for new PDFs.

- `sources/references.bib` — bibliography; the citation contract for the manuscript.

- `scripts/` — conversion and analysis code.

- <archetype folders: review/ + codebook/ | data/ + paper/ + figures/ + tables/ + replication/>

## Conventions

- Source naming: `author-year-slug` (lowercase, hyphens).

- New sources flow through the intake pipeline (`/process-source`), never dropped straight into `md/`.

- Read from `sources/md/`; cite from `sources/references.bib`.

- Large data, original PDFs, `.venv/`, and secrets are gitignored.

```

### `.gitignore`

```text

# Original PDFs and large binaries — keep local, don't push

sources/og/

*.pdf

*.xlsx

*.dta

*.7z

*.zip

# Large data files (keep code, not data)

data/**/*.csv

data/**/*.tsv

data/**/*.png

# Python environment

.venv/

# Sensitive — never commit

.env

.env.*

*.pem

*.key

*ANSWER_KEY*

*do_not_share*

# Local-only working areas

logs/

.claude/settings.local.json

# macOS / R / editors

.DS_Store

.Rhistory

.vscode/

.idea/

```

> Adjust the data rules to the project: if a dataset is public and small, track it; if it is large or restricted, gitignore it and document it in the README. The default keeps original PDFs out of git (copyright + size) while keeping the `md/` corpus tracked.

## When to reach for this skill vs. siblings

- **`research-repo`** (this) — create or audit the *working* repository's structure, anchored on `sources/`. Use at project start, or when a repo has grown messy.

- **`process-source`** (a global skill, not part of this plugin) — run the per-PDF intake into the structure this skill creates. Use every time a new paper arrives.

- **`replication-package`** — scaffold or audit the *public* reproducibility package built from the finished paper, near submission. This skill creates the working repo; that one creates the archive.

- **`literature-review` / `citation-check` / `fact-check`** — consumers of the `sources/md/` + `references.bib` knowledge base this skill establishes.