→ tasks/<TASK-ID-A>-task-description.md ```  ### Backlog ledger line format `backlog.md` uses `BACKLOG_ITEM_RE`, not the active-index regex. It accepts the wider backlog status set (`pending`, `blocked-pending`, `cancelled`, `superseded`, `absorbed`, `deferred`, plus active states), priorities P0-P4, optional bold priority/complexity tokens, and a nonempty single-line inline description. **Pointer: optional for backlog entries.** An existing description may be linked, but a compliant inline entry is not a Doctor finding and `--fix` must not truncate it or invent a relocation.  ``` - <TASK-ID-B> · pending · P2 · L2 · <Inline pending-work description> - <TASK-ID-C> · blocked · P3 · L2 · <Title> → tasks/<TASK-ID-C>-task-description.md ```  Section headers and blank lines are allowed; only task bullet lines are schema validated. ### `activeContext.md` thin contract **Active-Tasks-only mirror.** The file is bounded (≤ 30 lines) and contains a strict mirror of `tasks.md § Active`:  ```markdown # Active Context ## Active Tasks  - <TASK-ID-A> · in_progress · P1 · L3 · <Title> → tasks/<TASK-ID-A>-task-description.md ```  The legacy `## Последние завершённые` section is **abolished** — `/dr-doctor` removes it during Pass 3. Completion history must be looked up in `documentation/archive/` or git log instead of being mirrored into a rolling section.  ### `progress.md` **Abolished.** `/dr-doctor --fix` deletes the file unconditionally during Pass 3. No rolling completion log is maintained anywhere; per-task notes live in `datarim/tasks/{TASK-ID}-task-description.md` § Implementation Notes, and historical context lives in `documentation/archive/`. ## Description File Contract `datarim/tasks/{TASK-ID}-task-description.md` is the **only** place that holds the task's content. Format: ### YAML Frontmatter (12 fixed keys, all required) ```yaml --- id: <TASK-ID> # regex ^[A-Z][A-Z0-9]{1,9}-[0-9]{4}$ title: <string> # ≤ 80 chars, single line status: <enum> # in_progress|blocked|not_started|pending|blocked-pending|cancelled priority: <enum> # P0|P1|P2|P3 complexity: <enum> # L1|L2|L3|L4 type: <string> # free-form (e.g. framework, infra, content, bugfix) project: <string> # free-form (e.g. Datarim, Billing, Storefront) started: <date> # YYYY-MM-DD parent: <TASK-ID|null> # null if no parent related: <list[TASK-ID]> # YAML list, empty list ok prd: <relpath|null> # e.g. prd/PRD-{TASK-ID}.md, null if none plan: <relpath|null> # e.g. plans/{TASK-ID}-plan.md, null if none --- ``` All 12 keys are mandatory. Schema is closed — additional keys are NOT added by `/dr-doctor`. Project-specific extensions go inside the body, not in frontmatter. ### Body (markdown, ≤ 250 lines total) Five canonical sections (in order): ```markdown ## Overview 2–5 sentences. Problem + outcome. ## Acceptance Criteria - [ ] AC-1: … - [ ] AC-2: … ## Constraints Bullet list of immutable boundaries (security, performance, compatibility). ## Out of Scope Explicit non-goals. What this task does NOT do. ## Related Cross-references: parent PRD, sibling tasks, prior reflection notes. ``` Implementation Notes (free-form scratch, optional) MAY follow as `## Implementation Notes`. Discussion / decisions log MAY follow as `## Decisions`. Anything beyond ~250 lines is a smell — split into a PRD or design doc. ## Migration Algorithm (`--fix`) — 8 passes Applied by `scripts/datarim-doctor.sh --fix`. Single transactional sequence guarded by the data-loss safety contract (see next section). Atomic per file via `mv tmp file`. ### Pass 0 — `## Backlog` section reject in `tasks.md` 1. If `tasks.md` contains a top-level `## Backlog` header, emit finding `'## Backlog' section forbidden in tasks.md — move bullets manually to backlog.md` and exit 1 in dry-run mode. 2. `--fix` does NOT auto-migrate the section (cross-task hunk corruption risk). The section is preserved verbatim; the operator manually relocates bullets to `backlog.md` with correct task IDs and statuses. 3. Rationale: bullets nested inside a wrong file have unknown semantic intent (pending vs in-progress, target prefix project) — schema-level reject is safer than guessing. ### Pass 1 — Description files (build cache) 1. Walk `datarim/tasks.md` and `datarim/backlog.md` for legacy block-style headings: `^### ([A-Z][A-Z0-9]{1,9}-[0-9]{4}(?:-[A-Za-z0-9]+)*):?\s*(.*)$`. Trailing colon is optional; compound IDs (e.g. `PREFIX-NNNN-FOLLOWUP-slug`) accepted. 2. For each legacy block, extract the body until the next `### ID:` heading or section break. 3. Parse known fields (case-insensitive, leading `- ` or `* ` allowed): - **Status / Status:** → frontmatter `status` - **Priority / Приоритет:** → frontmatter `priority`  - **Complexity / Уровень / Level:** → frontmatter `complexity`  - **Type / Тип:** → frontmatter `type`  - **Started / Дата / Date / Date Started:** → frontmatter `started`  - **Parent / Родитель / Parent task:** → frontmatter `parent`  - **Related / Связанные:** → frontmatter `related`  - **PRD:** → frontmatter `prd` - **Plan:** → frontmatter `plan` 4. Heading first capture group → `id`. Second capture group → `title` (truncated to 80 chars on word boundary). If title text is empty (e.g. `### PREFIX-NNNN-FOLLOWUP-slug` with no trailing text), synthesise title from the compound suffix: strip the literal `FOLLOWUP-` token, replace remaining hyphens with spaces, sentence-case the first character; append « follow-up » suffix when the literal `FOLLOWUP` appeared in the ID. 5. Remaining body → write under `## Overview` (until first sub-heading) and pass through other sub-headings unchanged. 6. Normalize fields: - Status case-folded; alias `pending` ↔ `not_started` resolved by source file (tasks.md → `not_started`/`in_progress`/`blocked`; backlog.md → `pending`/`blocked-pending`/`cancelled`). - Priority normalized to `P[0-3]`; missing → `P3`. - Complexity normalized to `L[1-4]`; missing → `L2` (most common default). - Started missing → today (UTC). - Project missing → derive from prefix (`TUNE` → `Datarim`, `INFRA` → `Infrastructure`, …) or `unknown`. 7. Write to `datarim/tasks/{TASK-ID}-task-description.md`. **Skip if already exists with valid frontmatter** (idempotent). ### Pass 2 — Operational files (rewrite indexes) 8. Generate one-liner per task ID (regex above). 9. Group by section: tasks.md → `## Active` (in_progress/blocked/not_started); backlog.md → `## Pending` (pending/blocked-pending/cancelled). 10. Atomic rewrite via `mv tasks.md.tmp tasks.md`. ### Pass 3 — `activeContext.md` + `progress.md` retirement 11. Convert any legacy `**Current Task:** {ID}` line into `## Active Tasks` list with the corresponding one-liner; mirror is bounded to ≤ 30 lines. 12. Strip the `## Последние завершённые` section if present (abolished — see § activeContext.md thin contract above).  13. Delete `progress.md` if it exists. ### Pass 4 — `backlog-archive.md` migration 14. AWK section-state machine + per-ID dispatch reads legacy `backlog-archive.md`. 15. Cancelled entries → synthesised stubs in `documentation/archive/cancelled/archive-{TASK-ID}.md` (header notes `synthesised from backlog-archive.md by datarim-doctor.sh Pass 4`). 16. Completed entries → verify-or-synthesise into area-specific `documentation/archive/{area}/archive-{TASK-ID}.md`; unrecognised area falls back to `general/`. 17. Conflict policy is configurable via `--conflict-policy=prompt|keep|overwrite|skip|abort` (default `prompt`; auto-`skip` in non-TTY); `--no-prompt` is the canonical CI alias for `skip`. 18. The legacy `backlog-archive.md` is preserved in-tree as `backlog-archive.md.pre-v2.bak` (sidecar; operator-visible). ### Pass 6 — Operational-files archive section migration Strips legacy archive sections (`## Archived` in `tasks.md` / `backlog.md`; `### Archived`, `### Recently Archived`, `## Последние завершённые` in `activeContext.md`) and migrates each archive bullet to a canonical `documentation/archive/{area}/archive-{TASK-ID}.md` doc. The canonical thin-index contract (§ activeContext.md thin contract above, § Operational File Schema) prohibits archive sections in operational files: completion history lives in `documentation/archive/`, recency hint is computed at runtime via `/dr-status --recent N`. Pass 6 enforces that contract.  Four archive-bullet shapes are recognised (priority S1 → S2 → S4 → S3): - **S1 (arrow-link):** `- **TASK-ID** — title (YYYY-MM-DD) → documentation/archive/{area}/archive-TASK-ID.md` - **S2 (status-paren):** `- **TASK-ID** (status, YYYY-MM-DD) — title` (status ∈ `completed | cancelled | …`) - **S4 (mid-bold-context):** `- **TASK-ID** context-words — title` (context word(s) between `**ID**` and em-dash) - **S3 (plain-bold):** `- **TASK-ID** — title` (no date, no link, no mid-bold context) Task IDs may be **compound** — `<PREFIX-NNNN>`, `<PREFIX-NNNN>-<SUFFIX>`, `<PREFIX-NNNN>-<FOLLOWUP-SLUG>`. Numeric component (`-[0-9]{4}`) is required; suffix `(-[A-Za-z0-9]+)*` is optional. Per bullet: 1. Validate task ID matches `^[A-Z][A-Z0-9]{1,9}-[0-9]{4}(-[A-Za-z0-9]+)*$`. Invalid → preserve in operational file with manual-migration marker. 2. **Explicit-pointer dispatch:** if bullet body contains `→ documentation/archive/{path}.md`, prefer that path as canonical; otherwise fall back to `prefix_to_area()` (prefix→area mapping) → `documentation/archive/{area}/archive-{ID}.md`. 3. Path-traversal safety: canonical path MUST stay under `documentation/archive/`; violation rejects explicit pointer and falls back to `prefix_to_area`. If fallback also escapes → preserve, warn. > `prefix_to_area()` resolves **universal area prefixes first**, so a consumer > `## Task Prefix Registry` row that repeats one of them (`DEV`, `QA`, `WEB`, > `INFRA`, …) is ignored rather than applied. The doctor emits a `WARN` naming > both the reserved value and the discarded project value. If a project's > archives already live under a different subdir than the probe reports, that > warning is the explanation — and step 5's defensive `find` is what still > locates them (`archive at unexpected area`). See > `skills/datarim-system/task-identity-and-context.md` § Project Prefix Resolution. 4. **Verified case** — canonical archive exists with `{ID}` literal inside → strip bullet from operational file. 5. **Missing case** — canonical absent at computed path → defensive `find documentation/archive/ -name "archive-{ID}.md"` (depth ≤ 3) checks every area subdir; if found with ID literal, strip bullet with warning `archive at unexpected area`. Otherwise synthesise stub with frontmatter (`id`, `title`, `status`, `{status}_at`, `source: synthesised from operational-file by datarim-doctor.sh Pass 6`, `original_block_sha`) + body = original bullet content; strip bullet. 6. **Collision case** — canonical archive exists but does NOT contain `{ID}` literal → invoke `resolve_conflict()` (`--no-prompt` defaults to skip in non-TTY); on skip, preserve bullet in operational file with `` marker; on overwrite, synthesise stub. **Headerless fallback:** operational files without any archive section header are processed line-by-line. Bullets parseable via S1–S4 are candidates; bullets with explicit non-terminal status (`in_progress`, `not_started`, `pending`, `blocked`, `approved`, `review`, `active`) are passed through as active content. Other parseable bullets follow the same dispatch as the headered branch (explicit pointer → defensive find → synthesise). Non-parseable lines (one-liner thin-index entries, headers, frontmatter) pass through unchanged. Unparseable bullets (no shape match) → preserve with warning `Pass 6: unparseable archive bullet`. Operator fixes manually. After per-file processing, Doctor logs a one-line summary: `Pass 6 {file}: parsed={N} stripped={M} synthesised={K} skipped={L}`. Distributed users see exactly what migrated; tarball backup covers rollback. **Idempotent:** files without any archive header early-return; second `--fix` on a migrated tree produces zero changes.  *Counter-example — what Doctor MUST NOT do (Approach D, rejected on QA):* add a whitelist exception that preserves archive sections by design. This would legalise non-compliant pattern, accumulate token-bloat in operational files (12-18 KB on typical installations × 10-30 reads per session = 120-540 KB lost tokens), and contradict the canonical contract `datarim-system.md § activeContext.md thin contract` («one section only», v1.19.1). Migration, not preservation, is the correct enforcement.  ### Pass 7 — HTML-comment archive notes (verified-strip) Repo-local convention emits a one-line HTML comment after each archival: ```  ``` Recognised status verbs: `archived` | `cancelled` | `superseded` | `closed` | `dropped`. Both arrow forms accepted: `→` and `->`. Algorithm per operational file (`tasks.md`, `backlog.md`, `activeContext.md`): 1. Line-by-line scan. Match the canonical regex (compound ID + verb + ISO date + relpath to `documentation/archive/...md`). 2. Path-traversal guard: `validate_relpath` against `documentation/archive/` root rejects `..` segments. 3. Filename match guard: basename of the cited relpath MUST equal `archive-{ID}.md` (no cross-ID strip). 4. Existence check: `[ -f "{ROOT_ABS}/../{relpath}" ]`. 5. **Verified-strip:** file exists → omit the comment line; counter `stripped++`. 6. **Preserve + WARN:** archive file missing → keep the comment line + emit `Pass 7: archive file missing for {ID}: {relpath} (preserving comment)`; counter `preserved++`. 7. Atomic write via `mv tmpfile target` only when `stripped > 0`. Log line: `Pass 7 {file}: stripped={N} preserved={M}`. **Idempotent:** second `--fix` finds the same set of comments minus the previously-stripped lines; `stripped=0` on second run. **Pass 7 WARN is an observability rail, not a fallback.** The `preserved++` counter and the `Pass 7: archive file missing for {ID}: {relpath}` log line exist to surface a genuine data gap — an archive comment citing a file that does not exist on disk. When this WARN fires, the correct operator action is to file a fixture against the framework repo reproducing the gap, not to suppress or silence the WARN; treating it as noise defeats the purpose of the verified-strip contract. **Bats fixture marker discipline.** Every new bats fixture section that embeds task-ID-shaped literals MUST use its own history-allowed marker pair as defined in `skills/evolution/history-agnostic-gate.md` § Escape Hatch. Do not rely on a marker pair added earlier in the file: the diff-only gate treats markers as line-scoped context, so a fixture section outside its own pair is not covered by one opened elsewhere. ### Pass 5 — Post-fix re-scan 19. After `--fix` finishes the four mutating passes, the script composes the existing scan dispatch in dry-run mode and re-validates the tree. 20. Asserts: post-fix zero findings, `.pre-v2.bak` sidecar present (when Pass 4 ran), and an immediate second `--fix` is a no-op (idempotency). 21. Any failure here is treated as a Pass-4 regression and triggers the safety contract's restore path. ### Idempotency Guard Before Pass 1: if every operational file is already in canonical shape — zero `### TASK-ID:` headings in `tasks.md` / `backlog.md`, no legacy `backlog-archive.md` (or only the `.pre-v2.bak` sidecar remains), `progress.md` does not exist, no `## Последние завершённые` section in `activeContext.md`, and every bullet line matches the regex applicable to its file — exit 0 immediately. Cheap probe used by `/dr-init` self-heal.  ## Data-Loss Safety Contract Defence-in-depth around `--fix` mode. Every `--fix` invocation MUST satisfy all four rails; violation of any rail aborts the run with state preserved. - **Pre-write tarball backup.** Before any mutation, the script writes a `umask 077` tarball of the entire `datarim/` root to `${DATARIM_DOCTOR_BACKUP_DIR:-/tmp}/datarim-backup-{TS}.tgz`. Path is surfaced in the success summary so the operator can locate it for manual rollback. - **Sidecar copy.** Every legacy file mutated by Pass 4 also gets a `.pre-v2.bak` sidecar in-tree alongside the original (operator-visible, survives normal git workflows). Pass 5 asserts the sidecar exists. - **Count invariant.** Doctor counts task entries before mutation (`PARSED_COUNT`) and after rewrite (`EMITTED_COUNT`). Invariant: `EMITTED_COUNT ≥ PARSED_COUNT`. Violation triggers `restore_backup_and_die()`: removes mutated state in-place, `tar -xzf` the pre-write tarball back over the tree, and exits 2 with `emitted=N < parsed=M (data loss detected)`. - **Pinned-runtime provenance.** The doctor runs as `${DATARIM_RUNTIME:?}/scripts/datarim-doctor.sh`, a copy installed from the framework source by `install.sh --project`; `.datarim-runtime/installation.json` names the source commit. A doctor binary edited or dropped into the runtime by hand is replaced by the next install — fix the source repository instead. The contract is a hard precondition for any future `--fix` change: new mutating passes MUST plug into the same `PARSED_COUNT` / `EMITTED_COUNT` accounting and respect the tarball restore path. ## Conflict Resolution ### Description file already exists - **Compliant frontmatter (12 keys, ID matches)** → skip; do not overwrite. - **Frontmatter missing or wrong ID** → backup existing to `tasks/{TASK-ID}-task-description.md.bak-{timestamp}` and write canonical version. Operator merges manually if needed. - **File exists for ID that has no entry in tasks.md/backlog.md** → leave alone (orphan description; operator decides). ### Duplicate IDs across tasks.md and backlog.md If a task ID appears in both files (legacy state): - Status comes from `tasks.md` (active wins over pending). - Backlog entry is dropped from `backlog.md`. - Single description file written. ### Status/priority/complexity contradicts Source-file state wins (tasks.md status overrides description's frontmatter `status` if they disagree). Description frontmatter is rewritten to match. ### Pass 4 archive-file conflicts Resolved by `--conflict-policy`: - `prompt` (default in TTY) — interactive choice per conflicting `archive-{TASK-ID}.md`. - `keep` — preserve existing archive file untouched, log skip. - `overwrite` — replace existing archive file with synthesised stub, original moved to `.bak-{TS}`. - `skip` — same as `keep` but quiet (default in non-TTY; `--no-prompt` alias). - `abort` — fail the migration on first conflict; safety contract restores the tree. ### Path traversal in legacy entries `scripts/lib/canonicalise.sh` rejects any path that resolves outside `$DATARIM_ROOT` (lexical canonicalisation, no I/O). Tool exits 4. Operator inspects the entry manually. ## Edge Cases - **Bash 3.2 (macOS default)** — tool uses two-pass grep+awk parser, NOT NUL-delimited reads. Verified across bash 3.2 / 4.4 / 5.x. - **Empty `datarim/tasks.md`** — exit 0; nothing to do. - **Title with `→` character** — escaped or rejected (regex disallows). Operator must rename. - **Non-UTF-8 file** — tool refuses; operator must convert (`iconv`). - **Concurrent invocation** — `flock $DATARIM_ROOT/.dr-doctor.lock`. Second instance exits 3. - **Read-only filesystem** — exit 2 on first write attempt; partial state preserved (atomic per file). Tarball restore covers the partial mutation. - **Missing `tasks/` subdirectory** — created with `mkdir -p` before any description file write. - **Missing `documentation/archive/{area}/`** — created with `mkdir -p` during Pass 4 dispatch. - **`wiki/_raw_/` semantic-orphan check** — advisory-only pass (scope `all`): flags a file whose basename shares no token (≥4 chars, alnum-only) with its first 300 bytes of content — a signal of an accidental paste into the wrong file. Report-only; `--fix` does not touch `wiki/_raw_/`. ## CLI Surface (reference) ``` scripts/datarim-doctor.sh [OPTIONS] OPTIONS: --fix Apply fixes (default: dry-run) --scope=<scope> One of: tasks|backlog|active|backlog-archive|progress|descriptions|all (default: all) --root=<path> Datarim root (default: walk up from $PWD) --quiet Exit-code only (used by /dr-init self-heal) --no-prompt Skip Pass 4 conflicts (alias for --conflict-policy=skip) --conflict-policy=<policy> One of: prompt|keep|overwrite|skip|abort (default: prompt; auto-skip in non-TTY) --help ENVIRONMENT: DATARIM_DOCTOR_BACKUP_DIR Override pre-write tarball directory (default: /tmp) EXIT CODES: 0 Compliant (or --fix succeeded) 1 Non-compliant findings (dry-run) 2 Migration error (--fix aborted; tarball restored, state preserved) 3 Concurrent invocation (lock held) 4 Path traversal / security violation 64 Usage error ``` ## Validation (CI gate) `pre-archive-check.sh` runs the line-format validator before `/dr-archive` proceeds. Lines that don't match the canonical regex block the archive with: `BLOCK: {file} contains non-compliant lines (run /dr-doctor)`. Escape hatch: `pre-archive-check.sh --no-schema-check` (used during in-flight migration; not for normal use). ## See Also - `commands/dr-doctor.md` — operator-facing wrapper. - `scripts/datarim-doctor.sh` — implementation. - `tests/datarim-doctor.bats` — covers compliance detection, the 8-pass migration, safety contract (tarball + invariant + restore), conflict policies, regex compliance, CLI/UX. - `skills/datarim-system/SKILL.md` — broader Datarim file layout and path resolution.

--- name: datarim-doctor description: Schema and migration semantics for /dr-doctor — thin one-liner contract, 8-pass migration, data-loss safety, conflict resolution. Loaded by self-heal. current_aal: 2 target_aal: 3 --- # Datarim Doctor — Schema and Migration Semantics This skill is the runtime knowledge module for `/dr-doctor`. It defines the strict active-index contract for `tasks.md` / `activeContext.md`, the relaxed line-oriented ledger contract for `backlog.md`, the **8-pass migration algorithm** (Pass 0–7) that `scripts/datarim-doctor.sh` applies, and the **data-loss safety contract** that wraps every `--fix` invocation. Loaded by: - `/dr-doctor` (always) - `/dr-init` self-heal (when `--quiet` probe returns exit 1) - `/dr-archive` line-format gate (on failure, to explain non-compliance) Not loaded by other commands — they read operational files as line-oriented ledgers and resolve a description pointer when one is present. ## Why Thin Indexes `tasks.md` and `activeContext.md` are strict pointer indexes. `backlog.md` is a pending-work ledger: each entry remains one line and machine-parseable, but may preserve its pending-work description inline until the task is promoted. Goals: - **Bounded context** — agents read 1 KB index instead of 100 KB monolith. - **Single source of truth for active tasks** — description, ACs, and constraints live in `datarim/tasks/{TASK-ID}-task-description.md`; pointerless backlog prose is intake context, not an active plan. - **Greppable state** — line format is machine-parseable; status changes are 1-line diffs. - **Idempotent migrations** — `/dr-doctor` can run any number of times without drift. `progress.md` is **abolished**, and the legacy `activeContext.md § Последние завершённые` rolling log is **abolished** as well. Completion history lives only in `documentation/archive/{area}/archive-{TASK-ID}.md` and git log. ## Operational File Schema Exact regex constants are sourced from `scripts/lib/schema-regex.sh`. Do not redefine them inside Doctor or a downstream validator. ### Strict active-index line format `tasks.md` and `activeContext.md` use `ONELINER_RE`: ``` ^- ([A-Z][A-Z0-9]{1,9}-[0-9]{4}) · (STATUS) · P[0-3] · L[1-4] · (.+) → tasks/\1-(task-description|init-task)\.md$ ``` `ONELINER_RE` accepts `in_progress|blocked|not_started|pending|blocked-pending|cancelled` for legacy and migration compatibility. Canonical active-index writers emit `in_progress|blocked|not_started`; `pending` and `blocked-pending` belong to the backlog intake flow, and `cancelled` is archived from the backlog rather than mirrored as an active task. **Active-index pointer: required.** Separator: `·` (U+00B7 MIDDLE DOT, not a bullet or period). Arrow: `→` (U+2192). The pointer task ID must match the entry task ID. Compliant active entry: ``` - · in_progress · P1 · L3 · → tasks/<TASK-ID-A>-task-description.md ```  ### Backlog ledger line format `backlog.md` uses `BACKLOG_ITEM_RE`, not the active-index regex. It accepts the wider backlog status set (`pending`, `blocked-pending`, `cancelled`, `superseded`, `absorbed`, `deferred`, plus active states), priorities P0-P4, optional bold priority/complexity tokens, and a nonempty single-line inline description. **Pointer: optional for backlog entries.** An existing description may be linked, but a compliant inline entry is not a Doctor finding and `--fix` must not truncate it or invent a relocation.  ``` - <TASK-ID-B> · pending · P2 · L2 · <Inline pending-work description> - <TASK-ID-C> · blocked · P3 · L2 · <Title> → tasks/<TASK-ID-C>-task-description.md ```  Section headers and blank lines are allowed; only task bullet lines are schema validated. ### `activeContext.md` thin contract **Active-Tasks-only mirror.** The file is bounded (≤ 30 lines) and contains a strict mirror of `tasks.md § Active`:  ```markdown # Active Context ## Active Tasks  - <TASK-ID-A> · in_progress · P1 · L3 · <Title> → tasks/<TASK-ID-A>-task-description.md ```  The legacy `## Последние завершённые` section is **abolished** — `/dr-doctor` removes it during Pass 3. Completion history must be looked up in `documentation/archive/` or git log instead of being mirrored into a rolling section.  ### `progress.md` **Abolished.** `/dr-doctor --fix` deletes the file unconditionally during Pass 3. No rolling completion log is maintained anywhere; per-task notes live in `datarim/tasks/{TASK-ID}-task-description.md` § Implementation Notes, and historical context lives in `documentation/archive/`. ## Description File Contract `datarim/tasks/{TASK-ID}-task-description.md` is the **only** place that holds the task's content. Format: ### YAML Frontmatter (12 fixed keys, all required) ```yaml --- id: <TASK-ID> # regex ^[A-Z][A-Z0-9]{1,9}-[0-9]{4}$ title: <string> # ≤ 80 chars, single line status: <enum> # in_progress|blocked|not_started|pending|blocked-pending|cancelled priority: <enum> # P0|P1|P2|P3 complexity: <enum> # L1|L2|L3|L4 type: <string> # free-form (e.g. framework, infra, content, bugfix) project: <string> # free-form (e.g. Datarim, Billing, Storefront) started: <date> # YYYY-MM-DD parent: <TASK-ID|null> # null if no parent related: <list[TASK-ID]> # YAML list, empty list ok prd: <relpath|null> # e.g. prd/PRD-{TASK-ID}.md, null if none plan: <relpath|null> # e.g. plans/{TASK-ID}-plan.md, null if none --- ``` All 12 keys are mandatory. Schema is closed — additional keys are NOT added by `/dr-doctor`. Project-specific extensions go inside the body, not in frontmatter. ### Body (markdown, ≤ 250 lines total) Five canonical sections (in order): ```markdown ## Overview 2–5 sentences. Problem + outcome. ## Acceptance Criteria - [ ] AC-1: … - [ ] AC-2: … ## Constraints Bullet list of immutable boundaries (security, performance, compatibility). ## Out of Scope Explicit non-goals. What this task does NOT do. ## Related Cross-references: parent PRD, sibling tasks, prior reflection notes. ``` Implementation Notes (free-form scratch, optional) MAY follow as `## Implementation Notes`. Discussion / decisions log MAY follow as `## Decisions`. Anything beyond ~250 lines is a smell — split into a PRD or design doc. ## Migration Algorithm (`--fix`) — 8 passes Applied by `scripts/datarim-doctor.sh --fix`. Single transactional sequence guarded by the data-loss safety contract (see next section). Atomic per file via `mv tmp file`. ### Pass 0 — `## Backlog` section reject in `tasks.md` 1. If `tasks.md` contains a top-level `## Backlog` header, emit finding `'## Backlog' section forbidden in tasks.md — move bullets manually to backlog.md` and exit 1 in dry-run mode. 2. `--fix` does NOT auto-migrate the section (cross-task hunk corruption risk). The section is preserved verbatim; the operator manually relocates bullets to `backlog.md` with correct task IDs and statuses. 3. Rationale: bullets nested inside a wrong file have unknown semantic intent (pending vs in-progress, target prefix project) — schema-level reject is safer than guessing. ### Pass 1 — Description files (build cache) 1. Walk `datarim/tasks.md` and `datarim/backlog.md` for legacy block-style headings: `^### ([A-Z][A-Z0-9]{1,9}-[0-9]{4}(?:-[A-Za-z0-9]+)*):?\s*(.*)$`. Trailing colon is optional; compound IDs (e.g. `PREFIX-NNNN-FOLLOWUP-slug`) accepted. 2. For each legacy block, extract the body until the next `### ID:` heading or section break. 3. Parse known fields (case-insensitive, leading `- ` or `* ` allowed): - **Status / Status:** → frontmatter `status` - **Priority / Приоритет:** → frontmatter `priority`  - **Complexity / Уровень / Level:** → frontmatter `complexity`  - **Type / Тип:** → frontmatter `type`  - **Started / Дата / Date / Date Started:** → frontmatter `started`  - **Parent / Родитель / Parent task:** → frontmatter `parent`  - **Related / Связанные:** → frontmatter `related`  - **PRD:** → frontmatter `prd` - **Plan:** → frontmatter `plan` 4. Heading first capture group → `id`. Second capture group → `title` (truncated to 80 chars on word boundary). If title text is empty (e.g. `### PREFIX-NNNN-FOLLOWUP-slug` with no trailing text), synthesise title from the compound suffix: strip the literal `FOLLOWUP-` token, replace remaining hyphens with spaces, sentence-case the first character; append « follow-up » suffix when the literal `FOLLOWUP` appeared in the ID. 5. Remaining body → write under `## Overview` (until first sub-heading) and pass through other sub-headings unchanged. 6. Normalize fields: - Status case-folded; alias `pending` ↔ `not_started` resolved by source file (tasks.md → `not_started`/`in_progress`/`blocked`; backlog.md → `pending`/`blocked-pending`/`cancelled`). - Priority normalized to `P[0-3]`; missing → `P3`. - Complexity normalized to `L[1-4]`; missing → `L2` (most common default). - Started missing → today (UTC). - Project missing → derive from prefix (`TUNE` → `Datarim`, `INFRA` → `Infrastructure`, …) or `unknown`. 7. Write to `datarim/tasks/{TASK-ID}-task-description.md`. **Skip if already exists with valid frontmatter** (idempotent). ### Pass 2 — Operational files (rewrite indexes) 8. Generate one-liner per task ID (regex above). 9. Group by section: tasks.md → `## Active` (in_progress/blocked/not_started); backlog.md → `## Pending` (pending/blocked-pending/cancelled). 10. Atomic rewrite via `mv tasks.md.tmp tasks.md`. ### Pass 3 — `activeContext.md` + `progress.md` retirement 11. Convert any legacy `**Current Task:** {ID}` line into `## Active Tasks` list with the corresponding one-liner; mirror is bounded to ≤ 30 lines. 12. Strip the `## Последние завершённые` section if present (abolished — see § activeContext.md thin contract above).  13. Delete `progress.md` if it exists. ### Pass 4 — `backlog-archive.md` migration 14. AWK section-state machine + per-ID dispatch reads legacy `backlog-archive.md`. 15. Cancelled entries → synthesised stubs in `documentation/archive/cancelled/archive-{TASK-ID}.md` (header notes `synthesised from backlog-archive.md by datarim-doctor.sh Pass 4`). 16. Completed entries → verify-or-synthesise into area-specific `documentation/archive/{area}/archive-{TASK-ID}.md`; unrecognised area falls back to `general/`. 17. Conflict policy is configurable via `--conflict-policy=prompt|keep|overwrite|skip|abort` (default `prompt`; auto-`skip` in non-TTY); `--no-prompt` is the canonical CI alias for `skip`. 18. The legacy `backlog-archive.md` is preserved in-tree as `backlog-archive.md.pre-v2.bak` (sidecar; operator-visible). ### Pass 6 — Operational-files archive section migration Strips legacy archive sections (`## Archived` in `tasks.md` / `backlog.md`; `### Archived`, `### Recently Archived`, `## Последние завершённые` in `activeContext.md`) and migrates each archive bullet to a canonical `documentation/archive/{area}/archive-{TASK-ID}.md` doc. The canonical thin-index contract (§ activeContext.md thin contract above, § Operational File Schema) prohibits archive sections in operational files: completion history lives in `documentation/archive/`, recency hint is computed at runtime via `/dr-status --recent N`. Pass 6 enforces that contract.  Four archive-bullet shapes are recognised (priority S1 → S2 → S4 → S3): - **S1 (arrow-link):** `- **TASK-ID** — title (YYYY-MM-DD) → documentation/archive/{area}/archive-TASK-ID.md` - **S2 (status-paren):** `- **TASK-ID** (status, YYYY-MM-DD) — title` (status ∈ `completed | cancelled | …`) - **S4 (mid-bold-context):** `- **TASK-ID** context-words — title` (context word(s) between `**ID**` and em-dash) - **S3 (plain-bold):** `- **TASK-ID** — title` (no date, no link, no mid-bold context) Task IDs may be **compound** — `<PREFIX-NNNN>`, `<PREFIX-NNNN>-<SUFFIX>`, `<PREFIX-NNNN>-<FOLLOWUP-SLUG>`. Numeric component (`-[0-9]{4}`) is required; suffix `(-[A-Za-z0-9]+)*` is optional. Per bullet: 1. Validate task ID matches `^[A-Z][A-Z0-9]{1,9}-[0-9]{4}(-[A-Za-z0-9]+)*$`. Invalid → preserve in operational file with manual-migration marker. 2. **Explicit-pointer dispatch:** if bullet body contains `→ documentation/archive/{path}.md`, prefer that path as canonical; otherwise fall back to `prefix_to_area()` (prefix→area mapping) → `documentation/archive/{area}/archive-{ID}.md`. 3. Path-traversal safety: canonical path MUST stay under `documentation/archive/`; violation rejects explicit pointer and falls back to `prefix_to_area`. If fallback also escapes → preserve, warn. > `prefix_to_area()` resolves **universal area prefixes first**, so a consumer > `## Task Prefix Registry` row that repeats one of them (`DEV`, `QA`, `WEB`, > `INFRA`, …) is ignored rather than applied. The doctor emits a `WARN` naming > both the reserved value and the discarded project value. If a project's > archives already live under a different subdir than the probe reports, that > warning is the explanation — and step 5's defensive `find` is what still > locates them (`archive at unexpected area`). See > `skills/datarim-system/task-identity-and-context.md` § Project Prefix Resolution. 4. **Verified case** — canonical archive exists with `{ID}` literal inside → strip bullet from operational file. 5. **Missing case** — canonical absent at computed path → defensive `find documentation/archive/ -name "archive-{ID}.md"` (depth ≤ 3) checks every area subdir; if found with ID literal, strip bullet with warning `archive at unexpected area`. Otherwise synthesise stub with frontmatter (`id`, `title`, `status`, `{status}_at`, `source: synthesised from operational-file by datarim-doctor.sh Pass 6`, `original_block_sha`) + body = original bullet content; strip bullet. 6. **Collision case** — canonical archive exists but does NOT contain `{ID}` literal → invoke `resolve_conflict()` (`--no-prompt` defaults to skip in non-TTY); on skip, preserve bullet in operational file with `` marker; on overwrite, synthesise stub. **Headerless fallback:** operational files without any archive section header are processed line-by-line. Bullets parseable via S1–S4 are candidates; bullets with explicit non-terminal status (`in_progress`, `not_started`, `pending`, `blocked`, `approved`, `review`, `active`) are passed through as active content. Other parseable bullets follow the same dispatch as the headered branch (explicit pointer → defensive find → synthesise). Non-parseable lines (one-liner thin-index entries, headers, frontmatter) pass through unchanged. Unparseable bullets (no shape match) → preserve with warning `Pass 6: unparseable archive bullet`. Operator fixes manually. After per-file processing, Doctor logs a one-line summary: `Pass 6 {file}: parsed={N} stripped={M} synthesised={K} skipped={L}`. Distributed users see exactly what migrated; tarball backup covers rollback. **Idempotent:** files without any archive header early-return; second `--fix` on a migrated tree produces zero changes.  *Counter-example — what Doctor MUST NOT do (Approach D, rejected on QA):* add a whitelist exception that preserves archive sections by design. This would legalise non-compliant pattern, accumulate token-bloat in operational files (12-18 KB on typical installations × 10-30 reads per session = 120-540 KB lost tokens), and contradict the canonical contract `datarim-system.md § activeContext.md thin contract` («one section only», v1.19.1). Migration, not preservation, is the correct enforcement.  ### Pass 7 — HTML-comment archive notes (verified-strip) Repo-local convention emits a one-line HTML comment after each archival: ```  ``` Recognised status verbs: `archived` | `cancelled` | `superseded` | `closed` | `dropped`. Both arrow forms accepted: `→` and `->`. Algorithm per operational file (`tasks.md`, `backlog.md`, `activeContext.md`): 1. Line-by-line scan. Match the canonical regex (compound ID + verb + ISO date + relpath to `documentation/archive/...md`). 2. Path-traversal guard: `validate_relpath` against `documentation/archive/` root rejects `..` segments. 3. Filename match guard: basename of the cited relpath MUST equal `archive-{ID}.md` (no cross-ID strip). 4. Existence check: `[ -f "{ROOT_ABS}/../{relpath}" ]`. 5. **Verified-strip:** file exists → omit the comment line; counter `stripped++`. 6. **Preserve + WARN:** archive file missing → keep the comment line + emit `Pass 7: archive file missing for {ID}: {relpath} (preserving comment)`; counter `preserved++`. 7. Atomic write via `mv tmpfile target` only when `stripped > 0`. Log line: `Pass 7 {file}: stripped={N} preserved={M}`. **Idempotent:** second `--fix` finds the same set of comments minus the previously-stripped lines; `stripped=0` on second run. **Pass 7 WARN is an observability rail, not a fallback.** The `preserved++` counter and the `Pass 7: archive file missing for {ID}: {relpath}` log line exist to surface a genuine data gap — an archive comment citing a file that does not exist on disk. When this WARN fires, the correct operator action is to file a fixture against the framework repo reproducing the gap, not to suppress or silence the WARN; treating it as noise defeats the purpose of the verified-strip contract. **Bats fixture marker discipline.** Every new bats fixture section that embeds task-ID-shaped literals MUST use its own history-allowed marker pair as defined in `skills/evolution/history-agnostic-gate.md` § Escape Hatch. Do not rely on a marker pair added earlier in the file: the diff-only gate treats markers as line-scoped context, so a fixture section outside its own pair is not covered by one opened elsewhere. ### Pass 5 — Post-fix re-scan 19. After `--fix` finishes the four mutating passes, the script composes the existing scan dispatch in dry-run mode and re-validates the tree. 20. Asserts: post-fix zero findings, `.pre-v2.bak` sidecar present (when Pass 4 ran), and an immediate second `--fix` is a no-op (idempotency). 21. Any failure here is treated as a Pass-4 regression and triggers the safety contract's restore path. ### Idempotency Guard Before Pass 1: if every operational file is already in canonical shape — zero `### TASK-ID:` headings in `tasks.md` / `backlog.md`, no legacy `backlog-archive.md` (or only the `.pre-v2.bak` sidecar remains), `progress.md` does not exist, no `## Последние завершённые` section in `activeContext.md`, and every bullet line matches the regex applicable to its file — exit 0 immediately. Cheap probe used by `/dr-init` self-heal.  ## Data-Loss Safety Contract Defence-in-depth around `--fix` mode. Every `--fix` invocation MUST satisfy all four rails; violation of any rail aborts the run with state preserved. - **Pre-write tarball backup.** Before any mutation, the script writes a `umask 077` tarball of the entire `datarim/` root to `${DATARIM_DOCTOR_BACKUP_DIR:-/tmp}/datarim-backup-{TS}.tgz`. Path is surfaced in the success summary so the operator can locate it for manual rollback. - **Sidecar copy.** Every legacy file mutated by Pass 4 also gets a `.pre-v2.bak` sidecar in-tree alongside the original (operator-visible, survives normal git workflows). Pass 5 asserts the sidecar exists. - **Count invariant.** Doctor counts task entries before mutation (`PARSED_COUNT`) and after rewrite (`EMITTED_COUNT`). Invariant: `EMITTED_COUNT ≥ PARSED_COUNT`. Violation triggers `restore_backup_and_die()`: removes mutated state in-place, `tar -xzf` the pre-write tarball back over the tree, and exits 2 with `emitted=N < parsed=M (data loss detected)`. - **Pinned-runtime provenance.** The doctor runs as `${DATARIM_RUNTIME:?}/scripts/datarim-doctor.sh`, a copy installed from the framework source by `install.sh --project`; `.datarim-runtime/installation.json` names the source commit. A doctor binary edited or dropped into the runtime by hand is replaced by the next install — fix the source repository instead. The contract is a hard precondition for any future `--fix` change: new mutating passes MUST plug into the same `PARSED_COUNT` / `EMITTED_COUNT` accounting and respect the tarball restore path. ## Conflict Resolution ### Description file already exists - **Compliant frontmatter (12 keys, ID matches)** → skip; do not overwrite. - **Frontmatter missing or wrong ID** → backup existing to `tasks/{TASK-ID}-task-description.md.bak-{timestamp}` and write canonical version. Operator merges manually if needed. - **File exists for ID that has no entry in tasks.md/backlog.md** → leave alone (orphan description; operator decides). ### Duplicate IDs across tasks.md and backlog.md If a task ID appears in both files (legacy state): - Status comes from `tasks.md` (active wins over pending). - Backlog entry is dropped from `backlog.md`. - Single description file written. ### Status/priority/complexity contradicts Source-file state wins (tasks.md status overrides description's frontmatter `status` if they disagree). Description frontmatter is rewritten to match. ### Pass 4 archive-file conflicts Resolved by `--conflict-policy`: - `prompt` (default in TTY) — interactive choice per conflicting `archive-{TASK-ID}.md`. - `keep` — preserve existing archive file untouched, log skip. - `overwrite` — replace existing archive file with synthesised stub, original moved to `.bak-{TS}`. - `skip` — same as `keep` but quiet (default in non-TTY; `--no-prompt` alias). - `abort` — fail the migration on first conflict; safety contract restores the tree. ### Path traversal in legacy entries `scripts/lib/canonicalise.sh` rejects any path that resolves outside `$DATARIM_ROOT` (lexical canonicalisation, no I/O). Tool exits 4. Operator inspects the entry manually. ## Edge Cases - **Bash 3.2 (macOS default)** — tool uses two-pass grep+awk parser, NOT NUL-delimited reads. Verified across bash 3.2 / 4.4 / 5.x. - **Empty `datarim/tasks.md`** — exit 0; nothing to do. - **Title with `→` character** — escaped or rejected (regex disallows). Operator must rename. - **Non-UTF-8 file** — tool refuses; operator must convert (`iconv`). - **Concurrent invocation** — `flock $DATARIM_ROOT/.dr-doctor.lock`. Second instance exits 3. - **Read-only filesystem** — exit 2 on first write attempt; partial state preserved (atomic per file). Tarball restore covers the partial mutation. - **Missing `tasks/` subdirectory** — created with `mkdir -p` before any description file write. - **Missing `documentation/archive/{area}/`** — created with `mkdir -p` during Pass 4 dispatch. - **`wiki/_raw_/` semantic-orphan check** — advisory-only pass (scope `all`): flags a file whose basename shares no token (≥4 chars, alnum-only) with its first 300 bytes of content — a signal of an accidental paste into the wrong file. Report-only; `--fix` does not touch `wiki/_raw_/`. ## CLI Surface (reference) ``` scripts/datarim-doctor.sh [OPTIONS] OPTIONS: --fix Apply fixes (default: dry-run) --scope=<scope> One of: tasks|backlog|active|backlog-archive|progress|descriptions|all (default: all) --root=<path> Datarim root (default: walk up from $PWD) --quiet Exit-code only (used by /dr-init self-heal) --no-prompt Skip Pass 4 conflicts (alias for --conflict-policy=skip) --conflict-policy=<policy> One of: prompt|keep|overwrite|skip|abort (default: prompt; auto-skip in non-TTY) --help ENVIRONMENT: DATARIM_DOCTOR_BACKUP_DIR Override pre-write tarball directory (default: /tmp) EXIT CODES: 0 Compliant (or --fix succeeded) 1 Non-compliant findings (dry-run) 2 Migration error (--fix aborted; tarball restored, state preserved) 3 Concurrent invocation (lock held) 4 Path traversal / security violation 64 Usage error ``` ## Validation (CI gate) `pre-archive-check.sh` runs the line-format validator before `/dr-archive` proceeds. Lines that don't match the canonical regex block the archive with: `BLOCK: {file} contains non-compliant lines (run /dr-doctor)`. Escape hatch: `pre-archive-check.sh --no-schema-check` (used during in-flight migration; not for normal use). ## See Also - `commands/dr-doctor.md` — operator-facing wrapper. - `scripts/datarim-doctor.sh` — implementation. - `tests/datarim-doctor.bats` — covers compliance detection, the 8-pass migration, safety contract (tarball + invariant + restore), conflict policies, regex compliance, CLI/UX. - `skills/datarim-system/SKILL.md` — broader Datarim file layout and path resolution.