# docs/ — Documentation Hub

## Overview

Technical guide for the `docs/` directory — the central documentation hub for the Research Project Template.

## Agents: Start Here

If you are an AI agent picking up work in this repo, orient through these four doors before touching code or docs:

1. **Discover every skill.** Read [`_generated/skills_index.md`](_generated/skills_index.md) — the generated catalog of every `SKILL.md` found under the public infrastructure, script, exemplar and resource roots, `docs/prompts/`, `.agents/skills/`, and `.agents/workflows/`. It is the single inventory of what this repo can do.
2. **Find the agent-invocable CLI operations.** The machine-readable catalog of CLI operations (module, invocation, subcommands, exports) lives in [`../.agents/operations_manifest.json`](../.agents/operations_manifest.json). Regenerate or verify it with `uv run python -m infrastructure.skills operations-write` (write) and `uv run python -m infrastructure.skills operations-check` (verify against live discovery).
3. **Route workflow intents.** Send any research/manuscript/pipeline intent through the hub router [`prompts/SKILL.md`](prompts/SKILL.md), then follow the handoffs declared in [`prompts/MODE_REGISTRY.md`](prompts/MODE_REGISTRY.md). One intent → one child skill.
4. **THE TWO-TREE ROUTING RULE.** To **EDIT/extend an infrastructure module**, use that module's `infrastructure/<module>/SKILL.md`. To **RUN a research workflow**, use `docs/prompts/<skill>/SKILL.md`. Infrastructure skills are for changing the machinery; prompt skills are for operating it.

To compose a **custom** subset of pipeline stages rather than running the whole `--pipeline`, follow the **Workflow Composition Map**: [`prompts/COMPOSITION.md`](prompts/COMPOSITION.md).

## Directory Structure

Folder purposes live in [`README.md`](README.md). Generated roster and count
facts live under [`_generated/`](_generated/README.md).

## Key Conventions

- **Public exemplars are generated scope:** tracked canonical templates are listed
  in [`_generated/active_projects.md`](_generated/active_projects.md). Use
  [`projects/templates/template_code_project/`](../projects/templates/template_code_project/)
  as the default concrete walkthrough; private lifecycle mirrors rotate locally.
- **CodeGraph is local-only:** `.codegraph/` is generated agent-navigation state; see [`guides/codegraph-local.md`](guides/codegraph-local.md) before documenting or initializing it.
- Each sub-directory has a `README.md` (user-facing index) and `AGENTS.md` (technical guide)
- `documentation-index.md` is a curated flat map of long-lived documentation;
  generated inventories and source-owned directory guides remain authoritative
  for their own scopes
- Cross-references use relative paths with descriptive link text
- Documentation is intended to be evergreen; when behaviour changes, we may include dated notes so it’s clear which guidance is newer.

## Review and evidence contract

Deep documentation review must follow the fact to its producer and verifier. A review that cannot name the producing source or test must record the claim as unverifiable rather than repeat it - this contract is human review discipline, not something CI performs. Known-wrong input: prose edited to match accidental runtime behavior contradicts the docstring/test intent and must be reverted; tests such as `tests/infra_tests/validation/docs/test_consistency_lint.py` flag contract drift and fail the docs lane when the repair is skipped.
An edit that forces prose to agree with stale runtime output is the known-wrong outcome reviewers reject rather than accept.
This rule is a review discipline only; following it does not prove documentation correctness by itself, and disputes still require tracing to the owning source.
Do not resolve a contradiction by changing prose to match an accidental runtime
behavior when the code's docstring, public contract, and tests indicate the
opposite intent; record the discrepancy and repair the owning source or narrow
the claim. This contract binds review practice; no machine gate checks it
mechanically, so compliance remains human review.

| Surface | Producer / authority | Required review |
| --- | --- | --- |
| Commands, flags, defaults, precedence | CLI parser, config schema, implementation, focused tests | Execute or probe the exact path, including conflicting inputs and failure cases. |
| Generated repo facts | Producer named by [`_generated/AGENTS.md`](_generated/AGENTS.md) | Regenerate with the producer; never edit generated output directly. |
| Manuscript variables and statistics | Project analysis code and source artifacts, serialized to `output/data/manuscript_variables.json` | Trace every variable to inputs, estimand, sample size, uncertainty, units, rounding, and freshness. Source manuscript prose uses `{{TOKEN}}`; hydrated output is generated. |
| Figures and tables | Analysis code, figure/artifact registry, source data | Check data lineage, scales, labels, legends, uncertainty, palettes, resolution, panel ordering, and agreement among plot, caption, prose, and registry. |
| Captions and alt text | Manuscript source plus the rendered artifact | A caption interprets the display and states statistical context; alt text conveys the visual structure without pretending the caption alone is accessible text. |
| Scholarship and citations | Primary literature and bibliography records | Verify citation identity and support, qualify claim strength, disclose limitations and alternative explanations, and avoid citation laundering through secondary summaries. |
| Gates and release claims | Gate implementation, negative controls, fresh receipts, hosted CI, remote SHA | Name the verifier and a known-wrong test/fixture when claiming enforcement. Report local validation, hosted CI, approval, merge, publication, and archival state separately. |

When a value can change between runs, inject or generate it. When a statement is
interpretive, cite evidence and state its boundary. When a check is advisory,
do not describe it as a blocking gate.

### Documentation validation sequence

Run the smallest focused checks while editing, then the complete documentation
surface before handoff:

```bash
uv run python scripts/audit/lint_docs.py --json --repo-root .
uv run python scripts/audit/audit_documentation.py --format markdown
uv run python scripts/audit/check_template_drift.py --strict
uv run python scripts/docgen/counts.py --check
uv run python scripts/docgen/exemplar_roster.py --check
uv run python scripts/docgen/status_evidence.py --check
uv run python scripts/docgen/api_reference.py --check
uv run python -m infrastructure.skills check
uv run python -m infrastructure.skills check-contracts
```

The RedTeam audit is intentionally broad and advisory; triage its findings
rather than treating the number of heuristic matches as a quality score. The
other commands above fail on concrete drift. Changes to generated sources,
public behavior, or release contracts also require their focused tests and the
repository's normal pre-commit/pre-push checks.

## Entry Points

| Audience | Start Here |
| -------- | ---------- |
| New user | `core/how-to-use.md` → `guides/getting-started.md` |
| Developer | `core/architecture.md` → `architecture/two-layer-architecture.md` |
| **New project** | **`guides/new-project-setup.md`** → **`guides/new-project-one-shot-prompt.md`** (optional LLM scaffold) → `architecture/thin-orchestrator-summary.md` |
| Publication | `guides/publication-runbook.md` → `guides/publishing-guide.md` → `modules/guides/publishing-module.md` |
| Contributor | `development/contributing.md` → `development/testing/` |
| AI Agent | `prompts/SKILL.md` → `prompts/agentic-use/SKILL.md` → `rules/AGENTS.md` |
| Troubleshooter | `operational/troubleshooting/` → `reference/faq.md` |

## Learnings & Known Issues

Key discoveries from multi-project development are documented in:

- **[guides/new-project-setup.md](guides/new-project-setup.md)** — Setup checklist with all pitfalls
- **[guides/manuscript-semantics.md](guides/manuscript-semantics.md)** — Canonical manuscript syntax (citations, cross-references, sections, `{{TOKEN}}` substitution) shared by public template exemplars; project-specific overlays live in `projects/templates/template_*/manuscript/SYNTAX.md`
- **[operational/troubleshooting/common-errors.md](operational/troubleshooting/common-errors.md)** — Pipeline-specific error patterns

### ⚠️ Critical Rule: Root Venv Must Include All Project Dependencies

If a project in `projects/<name>/` has its own `pyproject.toml` but **no `.venv/` directory**, every package listed in that `pyproject.toml#dependencies` must also appear in the **root** `pyproject.toml`. Project-local analysis scripts run under the root venv through `scripts/pipeline/stage_02_analysis.py` in this case.

**Symptom when violated:** `❌ <project>: 4 stages, 7.7s` — Stage 4 (Analysis) fails silently in < 1s. No import error surfaces in the console.

**Fix:** Add missing packages to root `pyproject.toml`, then `uv sync`.

**Root venv vs project `pyproject.toml`:**

| Scope | Notes |
| --- | --- |
| [`projects/templates/template_code_project/`](../projects/templates/template_code_project/) | Stable exemplar; often uses a local `projects/templates/template_code_project/.venv/`. |
| Any other `projects/{name}/` | Listed in [_generated/active_projects.md](_generated/active_projects.md). If that project has **no** `.venv/`, every package in its `pyproject.toml` must also appear in the **root** `pyproject.toml`. |
| Plotting | `matplotlib` must be in root `[project.dependencies]` (not only optional groups) when analysis needs it under the root interpreter. |

### ⚠️ Critical Rule: `matplotlib` Must be in Core Dependencies

`matplotlib` must be in `[project.dependencies]`, not `[project.optional-dependencies]`. `uv sync` without flags does not install optional groups.

### Idempotency Contract for Analysis Scripts

Network-dependent analysis scripts may reuse existing outputs only under their
own explicit freshness and provenance contract. File existence or nonzero size
alone does not establish that an artifact matches current source, configuration,
inputs or required provider identity. Validate those bindings before reuse; a
missing or stale required artifact must trigger the supported rebuild or a clear
refusal. Preserve original failed receipts when rebuilding.

This avoids unnecessary network work without promoting stale output to current
evidence. Follow the owning producer and its negative controls rather than a
generic existence-only cache shortcut.

## See Also

- [README.md](README.md) — Quick navigation with Mermaid diagram
- [documentation-index.md](documentation-index.md) — Curated documentation map
- [Root AGENTS.md](../AGENTS.md) — System-level documentation

## Typed-Subfolder Project Lifecycle

Project lifecycle state is expressed as **typed subfolders under `projects/`**. Public exemplars are always discovered/rendered; optional `active/` entries are discovered/rendered when present; simplified sidecar `working/`, `ongoing/`, and `archive/` mirrors are non-rendered by default. Authoritative public roster: [`docs/_generated/active_projects.md`](_generated/active_projects.md).

| Subfolder | Purpose | Discovered + rendered? |
| ---------- | ------- | ------------------------ |
| `projects/templates/` | The git-tracked public exemplars (this repo) | ✅ Yes |
| `projects/active/` | Optional hot-seat render set — symlinks to deliberately reintroduced private `active/` | ✅ Yes when present |
| `projects/working/` | Simplified sidecar working set — symlinks, explicit targeted renders only | ❌ No |
| `projects/ongoing/` | Long-lived private work with no publication target — symlinks, explicit renders only | ❌ No |
| `projects/archive/` | Simplified sidecar archive — symlinks, historical/reference | ❌ No |

**Movement rules:**

- Only `projects/templates/*` and optional `projects/active/*` are discovered and rendered (qualified names `templates/<name>` and `active/<name>`).
- Private lifecycle projects live in the sidecar resolved by `TEMPLATE_PRIVATE_PROJECTS_ROOT` or `.private_projects_root` (default sibling `../projects`). See [`maintenance/private-projects-repo.md`](maintenance/private-projects-repo.md). Default folders are `working/` and `archive/`, with optional `ongoing/` for long-lived work without a publication target; each mirrors into matching typed subfolders on every `./run.sh` (or `uv run python -m infrastructure.orchestration link-projects`). Render explicitly with a lifecycle-qualified name such as `working/<name>` or `ongoing/<name>`.
- The git-tracked public exemplars under `projects/templates/` never move — they are owned by this repo. Every other path under `projects/` is local-only and never committed (enforced by `scripts/audit/check_tracked_all.py`).

**Configuring the active directory (advanced):**

To run the pipeline against a non-default directory (e.g., for smoke-testing an in-progress project without moving it):

```python
from pathlib import Path

from infrastructure.core.pipeline import PipelineConfig, PipelineExecutor

config = PipelineConfig(
    project_name="my_wip_project",
    repo_root=Path("."),
    projects_dir="projects/working",  # override the default "projects"
)
executor = PipelineExecutor(config)
executor.execute_core_pipeline()
```

The `projects_dir` field on `PipelineConfig` defaults to `"projects"`, preserving the standard `./run.sh` behaviour. The resolved project root path is the read-only property `project_dir` (`repo_root / projects_dir / project_name`). Call sites and helpers that accept an explicit directory typically take a `projects_dir` string (or derive paths from `PipelineConfig`).