---
name: evolution
description: Rules for proposing and applying framework improvements. Covers growth (new components) and maintenance (pruning, merging). Human approval required.
model: inherit
current_aal: 2
target_aal: 3
---

# Evolution — Framework Self-Update and Optimization Rules

## What is Evolution

Datarim improves itself in two ways:

1. **Growth** — on a passing `/dr-compliance` (Step 8.5), the `reflecting` skill analyzes lessons learned and proposes targeted improvements: new skills, updated agents, expanded templates. `/dr-archive` Step 0.5 generates reflection when the compliance stage did not produce a current artifact.
2. **Maintenance** — periodically or on demand, `/dr-optimize` audits the framework for bloat, duplicates, broken references, and inefficiencies, then proposes cleanup and consolidation.

Both paths require **explicit human approval** for every change. Evolution is how Datarim avoids repeating mistakes, accumulates institutional knowledge, and stays lean over time.

---

## When Triggered

### Automatic: Inside `/dr-compliance` Step 8.5 via `reflecting` skill (Growth); `/dr-archive` Step 0.5 is the fallback

The agent:
1. Reviews the reflection document just created
2. Identifies patterns that could benefit future tasks
3. Generates zero or more Evolution Proposals
4. **Bloat check**: Counts total skills, agents, commands. If any threshold is exceeded (see Health Metrics below), suggests running `/dr-optimize`
5. Presents proposals to the human for approval
6. Applies only approved changes

### Explicit: Via `/dr-optimize` (Maintenance)

The optimizer agent:
1. Performs a full audit of all framework components
2. Builds a dependency graph (commands → agents → skills)
3. Detects unused, oversized, duplicate, and broken components
4. Generates optimization proposals
5. Presents report and proposals for approval
6. Applies approved changes and syncs all documentation

### User-requested: Via `/dr-addskill` (Growth)

When creating new components, the skill-creator agent:
1. Audits existing framework before creating anything new
2. Prefers updating existing components over creating new ones
3. Follows the Anti-Bloat Rule (below)

**No automatic modifications.** Every change requires explicit human approval across all paths.

> **Historical note (v1.10.0):** the standalone `/dr-reflect` command was retired and consolidated into `/dr-archive` Step 0.5 via the `reflecting` skill. The primary trigger later moved to passing `/dr-compliance` Step 8.5; archive Step 0.5 remains the freshness-aware fallback, so reflection is still required before archive. Disaster-recovery procedures for runtime files live in `skills/utilities/recovery.md` (cross-referenced from `disaster-recovery.md`).

---

## Fragment Routing

Load only the fragment needed for the current sub-problem:

- `class-ab-gate.md`
  Use when evaluating whether a proposal changes the framework contract (Class A vs B operating-model gate, founding incident, decision aid, atomicity rule).
- `disaster-recovery.md`
  Use when runtime files in `${DATARIM_RUNTIME:?}/` are lost or corrupted. 5-channel recovery checklist.
- `examples-and-patterns.md`
  Use for reference when writing proposals or applying the deprecation pattern (forward-pointer annotations).
- `stack-agnostic-gate.md`
  MANDATORY pre-apply check before writing any approved Class A proposal to runtime. Rejects stack-specific content; whitelists `tech-stack.md` and the gate skill itself. CI helper at `scripts/stack-agnostic-gate.sh`.

---

## Proposal Categories

### Growth Categories (from the compliance reflection or archive fallback)

| Category | Target | Description |
|----------|--------|-------------|
| `skill-update` | `skills/{name}.md` | Improve existing skill — add recipes, fix inaccuracies, expand coverage |
| `agent-update` | `agents/{name}.md` | Refine agent capabilities, context loading, or decision criteria |
| `claude-md-update` | `AGENTS.md` | Update project-level rules, pipeline definitions, or conventions |
| `new-template` | `templates/{name}.md` | Create template for a recurring pattern |
| `new-skill` | `skills/{name}.md` | Create entirely new skill for an uncovered domain |
| `promote-recurring-incident-to-gate` | a blocking gate in a `commands/{name}.md` or `skills/{name}.md` | Promote a **recurring** incident-class into a blocking pipeline gate. Use when a lesson recurs (matches a prior reflection or describes a repeat of a known failure) and the existing contracts demonstrably did not prevent it. See the anti-self-suppression rule in `class-ab-gate.md`. |

### Optimization Categories (from `/dr-optimize`)

| Category | Target | Description |
|----------|--------|-------------|
| `prune-skill` | `skills/{name}.md` | Remove an unused or obsolete skill |
| `prune-agent` | `agents/{name}.md` | Remove an unused or obsolete agent |
| `prune-command` | `commands/{name}.md` | Remove an unused or obsolete command |
| `merge-skills` | `skills/{name}.md` | Combine two overlapping skills into one |
| `merge-agents` | `agents/{name}.md` | Combine two overlapping agents into one |
| `split-skill` | `skills/{name}.md` | Split an oversized skill into base + supporting files |
| `rewrite-skill` | `skills/{name}.md` | Restructure a skill for clarity and context efficiency |
| `fix-description` | any `.md` | Optimize description for better auto-triggering or shorter context |
| `fix-references` | any `.md` | Fix broken cross-references between components |
| `sync-docs` | `AGENTS.md`, `README.md`, `dr-help.md` | Update documentation counts and tables to match actual files |

---

## Proposal Format

Each proposal is a self-contained block.

```markdown
## Evolution Proposal

- **Category:** skill-update
- **Target:** skills/testing/SKILL.md
- **What:** Add property-based testing section with hypothesis/fast-check examples
- **Why:** Discovered during a prior incident that property tests caught edge cases unit tests missed.
- **Impact:** Medium — affects testing strategy for all future tasks
- **Risk:** Low / Medium / High
- **Diff preview:**
  ```
  + ## Property-Based Testing
  + When to use: data transformation, serialization, parsers, math operations.
  ```
```

### Required Fields

| Field | Description |
|-------|-------------|
| **Category** | One of the categories above (growth or optimization) |
| **Target** | File path relative to the framework root |
| **What** | Concise description of the change (one sentence) |
| **Why** | Evidence — from task reflection, audit findings, or user request |
| **Impact** | Low / Medium / High — how broadly this affects future tasks |

### Optional Fields

| Field | Description |
|-------|-------------|
| **Risk** | Low / Medium / High — how likely this is to break something |
| **Diff preview** | Approximate content to add/change |
| **Alternatives** | Other approaches considered |
| **Depends on** | Other proposals that must be applied first |

---

## Human Approval Gate

After presenting proposals, the agent MUST:

1. List all proposals with their number, category, target, risk, and one-line summary
2. Ask: "Which proposals should I apply? (all / none / comma-separated numbers)"
3. Wait for explicit response
4. Apply ONLY the approved proposals
5. Skip or discard rejected proposals without argument

**Never apply changes speculatively.** Never say "I'll go ahead and update this" without approval.

---

## Health Metrics

These thresholds trigger an optimization suggestion during reflection from `/dr-compliance` Step 8.5 or its `/dr-archive` Step 0.5 fallback:

| Metric | Threshold | Action |
|--------|-----------|--------|
| Total skills | >20 | Suggest `/dr-optimize` — check for merges |
| Total agents | >18 | Suggest `/dr-optimize` — check for merges |
| Total commands | >25 | Suggest `/dr-optimize` — check for duplicates |
| Any skill >500 lines | — | Suggest splitting into base + supporting files |
| Total description chars | >8000 | Suggest shortening descriptions |
| Any description >155 chars | — | Shorten to ≤155 chars (sufficient for discovery) |
| Orphan rate | >15% components unreferenced | Suggest `/dr-optimize` — prune orphans |

The optimizer reports these metrics in its audit. Healthy frameworks stay under all thresholds.

**Audit persistence:** When running `/dr-optimize`, write the structured report to `datarim/reports/optimize-audit-{YYYY-MM-DD}.md` for historical tracking. Chat-only findings are lost on context compaction.

---

## Anti-Bloat Rule

**Prefer updating existing files over creating new ones.**

Before proposing `new-skill`, `new-template`, or any new file:
1. Check if an existing skill could absorb the content
2. Check if an existing template could be extended
3. Only create a new file if the content is clearly a distinct concern

**Fragment-Resolution rule.** When a backlog item or proposal references `skills/<entry>.md § <section>` and `<entry>.md` is a thin router (no body content for that section — most split skills are a directory with a `SKILL.md` router plus sibling fragment files), the executing agent MUST `grep -rln "<section>" skills/<entry>/` for fragment files **before** choosing the insertion point. The router's own text rarely repeats section headings that live in a fragment; searching only the router file returns a false "not found" and risks either creating a duplicate section or inserting content into the wrong (router) file. Recurring papercut: a backlog text once named the retired flat testing entry, but the referenced content actually lived in the `skills/testing/live-smoke-gates.md` fragment.

Before proposing a merge:
1. Verify both components serve the same domain
2. Check that no information is lost in the merge
3. Update all cross-references after merging

Framework bloat degrades agent performance. Every new file adds to context loading overhead. Every description adds to the context budget. Keep the framework as small as it needs to be and no smaller.

---

## Evolution Log

All approved changes are logged in `datarim/history/evolution-log.md`. Create this file if it does not exist.

### Log Format

```markdown
# Evolution Log

| Date | Task ID | Category | Target | Change | Rationale |
|------|---------|----------|--------|--------|-----------|
| 2026-04-11 | OPT-001 | prune-skill | skills/<retired-skill>/SKILL.md | Removed unused skill | No agent references it |
```

For optimization runs, use `OPT-NNN` as the task ID. For growth proposals from compliance reflection or the archive fallback, use the current task ID.

---

## Rollback

Each Evolution change is a discrete edit to a specific file. Rollback strategy:

- **If using git:** Each approved set of changes should be a single commit with a message referencing the task ID. Revert via `git revert`.
- **If not using git:** The evolution log provides enough information to manually undo changes. The diff preview in the original proposal shows what was added.
- **For prune operations:** The optimizer creates a backup of deleted files in `documentation/archive/optimized/` before removal. Files can be restored from there.

> **Note:** Since 2026-04-22, `skills/`, `commands/`, `agents/`, `templates/` in `${DATARIM_RUNTIME:?}/` are symlinks to the Datarim git repo. Manual sync (`install.sh`) is no longer needed for these directories — changes are shared instantly. `install.sh` remains relevant only for first-time installation or rollback. See `skills/datarim-system/path-and-storage.md` § Symlink Architecture.

**Rule:** Never make changes that cannot be independently reverted. If two proposals modify the same file, apply them as separate edits so either can be rolled back without affecting the other.

---

## Pattern: Split-Architecture Metrics for Absorption Tasks

When a Class B task absorbs new components (skills, templates, fragments) into a **cold path** — meaning the new files are loaded on-demand at invocation time, not auto-loaded into every session — an aggregate token-budget metric (`total_chars_after / total_chars_before ≤ N%`) becomes meaningless because the hot path never sees the cold-path mass. Aggregate gates fail by design under structural absorption.

**Rule:** any absorption task whose plan adds files that are not auto-loaded MUST replace the aggregate metric with a split-axis metric:

1. **Idle hot-path budget** — measure only the entry skills/files loaded by every pipeline command (the discovery skill, system contract, mandatory loaders). Tight bound here protects per-invocation token cost.
2. **Per-existing-file delta** — bound the per-file growth on any file that existed in the baseline (e.g. `≤+30% chars`). Catches in-place bloat of touched components.
3. **On-demand exempt** — newly introduced files loaded only on invocation are reported informationally (count, total chars) but excluded from the gate.

**Why it matters:** absorbing N new on-demand skills inflates aggregate chars by Σ(new) regardless of runtime cost. A single aggregate gate forces a false trade-off between absorption and «no growth»; a split-axis gate aligns the metric with how runtime actually loads content.

**Falsifiability requirement:** each axis MUST have a concrete measurement command in `dev-tools/` that returns exit 0 (PASS) / exit 1 (FAIL). PRD AC text references the command directly so QA and Compliance re-runs are deterministic.

**Source:** Datarim v2.0.0 absorption (14 skills + 4 templates) — original aggregate ≤+10% gate violated by +27% structural addition; reformulated mid-task into hot-path ≤+16% + per-file ≤+30% + on-demand exempt.

---

## Pattern: Brand Hygiene for Absorbed External Skills

When a task absorbs or ports an external component (skill, agent, command, template) — typically an
MIT-licensed skill from another framework — the source material carries the *source project's* brand
surface: its skill-invocation prefix, namespace literals, tool names, and ecosystem paths. Ported
verbatim, those strings become dead pointers inside Datarim (the source component no longer exists in
this repo) or, worse, silently promote a competing framework's brand inside Datarim's own shipped
instruction surface. Every absorption MUST run the following four-step hygiene pass. This is a
mandatory contract, not a judgment call.

1. **Attribute the provenance in `CHANGELOG.md`.** Enumerate each absorbed component and its
   upstream source under the MIT licence in the CHANGELOG entry for the absorbing release. Attribution
   is required and is the *only* place the source brand is allowed to persist verbatim.

2. **Replace every source brand cross-reference with its local Datarim equivalent, post-merge.** After
   the port, find and replace the source framework's skill-invocation prefix, namespace literals, and
   internal tool names with the corresponding Datarim skill/agent/command reference. A cross-reference
   that points at a component the framework does not ship is a dead pointer and is not acceptable in a
   fully-absorbed artifact.

3. **Drop external path-interop strings unless Datarim declares a runtime dependency.** Absorbed files
   often reference the source ecosystem's on-disk paths (its worktree/config directories). Datarim owns
   none of those and, absent a *declared* dependency on the source tool, needs none of them. Drop the
   string and, where the behaviour still matters, restate it brand-agnostically ("paths owned by the
   external worktree-manager under its own config dir"). Keep a path only when a real dependency is
   declared.

4. **Enforce with the CI regression gate — extend it per new source brand.** A committed gate already
   exists: `tests/test-absorption-brand-hygiene.bats` asserts the source-brand skill-invocation prefix
   never reappears in `skills/ agents/ commands/ templates/`, and CI runs it via `bats tests/`
   (`.github/workflows/bats.yml`). Manual grep catches a point-in-time state; only the committed gate
   prevents regression across future commits. When a task absorbs a **new** external framework, it MUST
   extend that gate's banned-marker set with the new source's brand namespace and add a
   golden-violation fixture reproducing the leak, so the gate grows with each absorption instead of
   silently covering only the first-absorbed brand. (Writing a source-brand marker verbatim into any of
   the four scanned scopes will itself trip the gate — reference banned markers abstractly in shipped
   text, never literally.)

**Source:** `documentation/how-to/evolution-log.md` — the Class B proposal accepted after the v2.0.0
absorption of 14 external skills, its dead-cross-reference cleanup, and the regression gate that
followed. This pattern is the sibling of `## Pattern: Split-Architecture Metrics for Absorption
Tasks` above — both encode contracts an absorption task must satisfy.

---

## Pattern: Memory Rule → Executable Gate at Apply Step

When a user-memory rule (e.g. `~/.claude/projects/<proj>/memory/feedback_*.md`) repeatedly surfaces in reflection as a corrective action — meaning the rule was declared, then violated, then manually reverted — text-only memory is no longer sufficient at scale. Promote the rule to an **executable gate at the apply step** of the relevant pipeline command.

**Trigger threshold:** N ≥ 2 occurrences of the same memory rule appearing as a reflection corrective in distinct tasks within a short window (≤14 days).

**Gate anatomy:**

1. **Runtime contract** — `skills/<area>/<rule-name>-gate.md` documenting trigger, scope, allow/deny criteria, escape hatches, decision matrix. Markdown checklist agents can apply via Read+grep when no script is reachable.
2. **CI helper** — `scripts/<rule-name>-gate.sh` (pure bash, dependency-free). Same logic, scriptable for bats and CI. Exit codes: `0` PASS / `1` FAIL / `2` invocation error.
3. **Hooks** — embed «MUST run gate» line in every command/skill that reaches the apply step the rule should guard. Failure to embed = silently disabled gate.
4. **Bats fixtures** — at minimum 1 FAIL fixture (golden violation reproducing the original incident) + 1 PASS fixture (legitimate negative case) + 1 regression-invariant test on the gate's own host file (catches re-introduction of the violation).
5. **Whitelist + escape hatch** — declare any file that is exempt by design (e.g. `tech-stack.md` for stack-keywords). Provide a per-block escape marker (e.g. HTML-comment fence) for legitimate illustrative content. Use sparingly; reviewers should challenge each usage.

**Source incident:** `feedback_datarim_stack_agnostic.md` declared 2026-04-25, violated the next day across three artefacts. Memory rule advisory; gate enforces.

**Reuse candidates:** consider this pattern for any future recurring memory rule — e.g. «no-secrets-in-code», «no-personal-paths», «no-deprecated-API-XXX», ecosystem-specific keyword bans.

---

## Pattern: Helper Extends Doctrine — Same-Task Reconcile

When a runtime helper (a predicate, branch, return value, accepted-name set, allowed-flag list) covers a strictly wider set of conditions than the doctrine that documents it, the same task that widens the helper MUST close the gap in one of two ways. Doctrine-only and code-only states are both unstable: the next reader follows the narrower doctrine, judges the wider branch redundant, reverts it, and the runtime regresses to the original incident.

**Trigger:** a code change introduces or expands a branch / accepted-name / matched-flag / fallback path that is not literally enumerated in the matching doctrine (the convention bullet, comment block, ADR, or contract sentence that explicitly names the helper). The new branch is provably reachable in production — not dead code, not a defensive belt-and-braces — and the test suite asserts it.

**Required disposition (one of the two, in the same task):**

1. **Update the doctrine.** Add the new condition / name / flag to the doctrine prose in the same task. The doctrine and the helper now describe the same set.
2. **Record narrower-doctrine-intentional.** If the doctrine is deliberately narrower than the helper (e.g. the helper accepts a deprecated form on the way to removal, or one branch is platform-specific), add an inline disposition next to the helper itself: a one-line comment of the form `// doctrine narrower than helper: <reason> — see <link/anchor>` or the language-equivalent. The disposition is required in the same change so a future reader does not need to reconstruct the intent.

**What does NOT close the gap:**

- A reflection note that says «doctrine to update later». Drift looped twice between archives in source incident below — text-only memory is not load-bearing.
- A backlog follow-up to «decide simplify-helper vs amend-doctrine». The follow-up is itself the gap; the decision is the disposition above.
- A comment on the helper that says «matches both X and Y» without the **why** — the next reviewer reads the doctrine, sees only X, and removes the Y branch as dead code.

**Verification at /dr-compliance:** when Step 3 (References) detects that a helper accepts more values / names / shapes than the doctrine enumerates, FAIL Layer 4 unless one of the two dispositions is in the same diff. Operator override sets the disposition explicitly and re-runs.

**Source incident:** an `isAbortError` predicate accepted two cancel-marker names rather than one because the runtime emitted both shapes (the native cancel name plus a library-specific cancel name from a wrapped HTTP client). The matching doctrine in the project's convention file named only the native shape. The doctrine was amended to match the helper, reverted, then re-amended only after the drift surfaced for a second time across two review rounds. Two QA rounds and one compliance round wasted on the loop. The pattern generalises: any helper that quietly broadens its accept-set creates a future-reader trap unless the doctrine moves with it in the same task.

**Reuse candidates:** error-name normalizers (Abort/Cancel/Timeout), allowed-MIME-type lists, dialect-flag fallbacks, locale-tag aliases, soft-delete predicates that accept multiple sentinel values, schema migrations that tolerate transitional column shapes.