--- name: tlamatini-self-modify-inclusion description: Sweep the whole codebase and keep the SELF-MODIFY source snapshot complete — so the TlamatiniSourceCode/ tree that a `--self-modify` build ships carries every source/build asset Tlamatini needs to read, modify, and REBUILD herself, with heavy media/secrets correctly omitted+redacted and omitted binaries listed in the restore manifest. Invoke whenever you add/rename a new agent, source file, top-level dir, dependency, build input, prompt/identity file, migration, static/template asset, or any rebuild input — and ALWAYS after a feature like Blenderer or the self-update capability lands. Owns copy_source_assets.py and audits it against five invariants with a runnable sweep that GENERATES a snapshot and proves a rebuild's inputs all survive. Sibling of tlamatini-self-update-inclusion (which guards the runnable release). --- # Tlamatini — Self-Modify Inclusion Sweep > **Audience:** Claude Code working ON the Tlamatini codebase for **Angela**. > **Goal:** guarantee the **source snapshot** a self-able-modify build ships > (`TlamatiniSourceCode/`, generated by `copy_source_assets.py`) carries **everything a > rebuild needs** — so a running Tlamatini can read → modify → **regenerate her own > `Tlamatini.exe`** without missing the minimal thing — while staying lean (media/secrets > out) and safe (no leaked keys). This is the twin of `tlamatini-self-update-inclusion`. That one guards the **runnable release** (what users download). **This one guards the SOURCE tree** Tlamatini carries of herself. They overlap on one fact: a rebuild from the snapshot must be able to re-run the *same* `build.py` — so **every input `build.py` consumes must survive the snapshot**. ### The acid test (the whole job in one sentence) > Can `cd /TlamatiniSourceCode && python build.py --self-modify` succeed using > **only** the snapshot plus the documented restore steps? If yes, the snapshot is complete. --- ## The file this skill owns **`copy_source_assets.py`** (repo root) — generates the snapshot. Six knobs decide what ships: | Knob | Controls | |---|---| | `EXCLUDED_DIR_NAMES` | directory names pruned everywhere (`.git`, `node_modules`, `build`, `dist`, `pools`, `staticfiles`, `Temp`, `Templates`, `agents_backup`, `updater`, `TlamatiniSourceCode`, …) | | `EXCLUDED_EXTENSIONS` | file types dropped (`.pdf/.pptx/.png/.mp4/...`, binaries, generated state). **`.ico`/`.wav`/`.svg` are deliberately KEPT** (build-required) | | `EXCLUDED_FILE_NAMES` | exact files dropped (`db.sqlite3`, `data.keys`, `settings.local.json`, `_version.py`, …) | | `EXCLUDED_FILE_GLOBS` | patterns dropped (`*.version.txt`, `*_log.*`) | | `RESTORE_FROM_INSTALL` | binaries the build NEEDS but the snapshot OMITS → restored from the install at rebuild (`jd-cli.jar`, `XAIHT-Tlamatini.mp4`). The rebuild's escape hatch. | | `REQUIRED_SNAPSHOT_FILES` + `_wants_redaction()` | the completeness floor (raises if a critical file is missing) and the secret-redaction rules (config.json + agent config.yaml) | It also writes `_SOURCE_SNAPSHOT_MANIFEST.json` (counts/restore list) and `_REBUILD_INSTRUCTIONS.md` (the restore + rebuild runbook). The walk is **generic** — any new `.py/.js/.css/.yaml/.pmt/...` ships automatically. The danger is the *exclusions* wrongly dropping a rebuild input, an omitted binary missing from the restore manifest, a new always-present capability file not in `REQUIRED_SNAPSHOT_FILES`, or a secret leaking. --- ## The FIVE invariants ### Invariant 1 — INCLUDE: every rebuild SOURCE input survives Every file `build.py` reads to build (the build scripts, `requirements.txt`, all `--add-data` source trees, `optional/required_file_copies`, `optional_dir_copies` sources, `support_files`, the icon/wav/svg, every agent template, every `SKILL.md`, `prompt.pmt`, `Tlamatini.md`, `agentic_skill.md`, `monitoring-prompt.pmt`) must appear in the snapshot — **or** be in `RESTORE_FROM_INSTALL`. A source file dropped by an exclusion rule = a rebuild that fails or degrades. ### Invariant 2 — OMIT-LEAN, but RESTORE what the build needs Heavy media, dev trees, regenerable state, and secrets stay out (keep it ~10 MB, leak-free). But **anything omitted that the build genuinely consumes must be in `RESTORE_FROM_INSTALL`** (restored from the install) or covered by a regeneration step in `_REBUILD_INSTRUCTIONS.md` (e.g. `collectstatic` for `staticfiles`). Optional inputs (the `images` gallery) may degrade gracefully — decide per asset whether degradation is acceptable or it needs a restore entry. ### Invariant 3 — REDACT: no live secret survives `config.json` and every agent `config.yaml` are scrubbed to `` placeholders; `data.keys` is excluded outright. The snapshot must contain **zero** live keys. ### Invariant 4 — COMPLETENESS: critical files are guaranteed, not incidental `REQUIRED_SNAPSHOT_FILES` raises if a build-/run-critical source is missing. When you add a **new always-present capability** (a new build script, a new identity file, a flagship new agent), add its canonical file here so a future careless exclusion can't silently drop it. ### Invariant 5 — REBUILD COHERENCE: the manifest + instructions are true `_SOURCE_SNAPSHOT_MANIFEST.json` and `_REBUILD_INSTRUCTIONS.md` exist, `errors == []`, and the restore steps name real omitted binaries with correct install-relative sources. --- ## THE SWEEP — run this every time ### Step 0 — run the deterministic checker (does the heavy lifting) ```bash python .claude/skills/tlamatini-self-modify-inclusion/scripts/sweep_self_modify.py # add --keep to leave the generated snapshot under Temp/_self_modify_sweep for inspection ``` It **actually generates** a snapshot into `Temp/` and verifies: it builds error-free with all `REQUIRED_SNAPSHOT_FILES`; **every resolvable `build.py` input is carried or restorable** (distinguishing source-drop FINDINGS from intentionally-omitted media/regenerable NOTES); no live secret survives (re-running copy_source_assets' own redaction logic on the OUTPUT + a config-file value scan); the restore manifest + instructions exist and name real binaries; and no source-like extension is in the exclude set. Exit code = findings (0 == clean). Fix every `[FINDING]`; read every `[note]` and confirm the degradation/regeneration is intended. ### Step 1 — diff since the last release and classify new paths ```bash git diff --name-status "$(git describe --tags --abbrev=0 --match 'v[0-9]*')"..HEAD # brand-new TOP-LEVEL repo entries — the highest risk for a wrongly-excluded rebuild input git diff --name-status "$(git describe --tags --abbrev=0 --match 'v[0-9]*')"..HEAD \ | awk '$1=="A"{print $2}' | awk -F/ '{print $1}' | sort -u ``` For each new path ask the taxonomy question below. New `.py/.js/.css/.yaml/.pmt/SKILL.md` inside already-walked trees auto-ship (note, no action). The risks are: a new **binary** build input, a new **excluded** dir/ext/name that hides source, a new **secret-bearing config file**, or a new **always-present** file that belongs in `REQUIRED_SNAPSHOT_FILES`. ### Step 2 — the judgment passes the script can't fully make ```bash # (a) New binary build inputs the build reads (icons/fonts/wav/jar/etc.) that EXCLUDED_EXTENSIONS would drop grep -nE "--add-data=|--icon|optional_dir_copies|required_file_copies|support_files" build.py | grep -iE "\.(ico|wav|svg|png|jpg|ttf|otf|jar|dll|bin|dat)" # (b) New config files that may carry secrets but AREN'T covered by _wants_redaction() grep -rilE "api[_-]?key|token|secret|password|client_secret" Tlamatini --include=*.json --include=*.yaml --include=*.yml \ | grep -viE "/agents/.*/config.yaml$|agent/config.json$" # (c) New dependency / runtime added (cross-check the self-update skill too) git diff "$(git describe --tags --abbrev=0)"..HEAD -- requirements.txt ``` - (a) Any binary the build needs and the snapshot drops → add to `RESTORE_FROM_INSTALL` (omit + restore) or to the KEEP exceptions (if small + truly source-required, like the `.ico`/`.wav`/`.svg` already are). - (b) Any NEW config file that can hold a live key → extend `_wants_redaction()` to scrub it (and confirm Step 0's redaction check then covers it), or exclude it by name. - (c) A new dep doesn't affect the snapshot directly (it ships `requirements.txt`), but the rebuild `pip install`s it — make sure `requirements.txt` is current (the self-UPDATE skill owns the *bundled* side). **v1.48.14 External-MCP gate:** the snapshot must carry `agent/runtime_provisioner.py`, `agent/external_mcp_defaults.py`, and the manager/build/secret-regeneration source, but must not copy the downloaded `%LOCALAPPDATA%\Tlamatini\runtimes` tree or persistent Memory graph. Treat `external_mcps.json` as a special preserved/tracked config: the snapshot and public build receive a scrubbed/default-only form, never the keyed developer catalog. Verify `REQUIRED_SNAPSHOT_FILES`, `_wants_redaction()`, and the generated rebuild instructions agree. **v1.48.17 source-coherence gate:** the snapshot must carry Grepper's encoding-safe source, `agent_verdict.py`, `services/agent_contracts.py`, Kuberneter, and their regression tests, including `test_grepper_encodings.py`, `test_status_vocabulary.py`, and `test_preserved_user_state.py`. The status vocabulary must remain defined once in `agent_verdict.py`; do not copy its sets into a snapshot helper. Confirm the generated snapshot preserves `returncode`/`success`/semantic-status fields and the updater's `Uninstaller.exe` rule without copying local runtime state or keyed catalogs. **v1.48.18 worktree-target source-coherence gate:** the snapshot must carry the complete NetSpeed-Calculator source/config/wiring/tests and migrations 0195-0197, `sqlite_copy.py` plus its WAL backup/swap tests, Googler's structured-dork and two-tier plain-HTTP-first resilience source/config, `test_googler_dorks.py`, and the optional visible dork-hunt harness, migration 0194's Deep Internet Research prompt, and the `adding_external_mcp` skill tree. It must also carry the private-builder source that performs contact synchronization, but it must never copy `contacts.json`, `contacts.private.json`, a frozen-install contact book, or live External-MCP secrets. Verify current 88/66/108/29/197 counts from the generated snapshot rather than copying prose counts. ### Step 3 — fix every finding, re-run Step 0 until clean, eyeball the notes. --- ## Asset taxonomy — type → ship in snapshot? → action | New asset | In snapshot by default? | Action | |---|---|---| | New `.py/.js/.css/.html/.yaml/.pmt/.proto/SKILL.md` (source) | ✅ auto (generic walk) | none | | New **agent** (`agent/agents//` source + config.yaml) | ✅ auto; config.yaml auto-**redacted** | none (verify redaction in Step 0) | | New **migration** (`.py`) | ✅ auto | none | | New **build script / build input file** at repo root | ✅ if not excluded | add to `REQUIRED_SNAPSHOT_FILES` if always-present | | New **small build-required binary** (`.ico/.wav/.svg`-class) | ✅ (those exts KEPT) | if a NEW binary ext, KEEP it (remove from excludes) or restore-map it | | New **heavy binary** the build needs (`.jar/.mp4/large media`) | ❌ omitted | add to `RESTORE_FROM_INSTALL` (+ it lands in `_REBUILD_INSTRUCTIONS.md`) | | New **optional media** (gallery image) | ❌ omitted | leave omitted if degradation OK; else restore-map | | New **config file with secrets** | ✅ shipped — ⚠️ maybe UNREDACTED | extend `_wants_redaction()` or exclude by name | | New **runtime-state / generated dir** | should be ❌ | add its name to `EXCLUDED_DIR_NAMES` | | New **flagship capability file** (e.g. `self_update.py`, `Tlamatini.md`) | ✅ auto | add to `REQUIRED_SNAPSHOT_FILES` so it's guaranteed | The single most dangerous omissions: a **source file hidden by an over-broad exclusion** (Step 0 flags it as a source-drop FINDING) and a **new secret-bearing config that `_wants_redaction()` doesn't cover** (Step 2b — the raw scan in Step 0 catches obvious key shapes, but a novel config path needs the redaction rule extended). --- ## Where to make each fix (`copy_source_assets.py`) - **A source file got excluded** → narrow the offending rule in `EXCLUDED_EXTENSIONS` / `EXCLUDED_FILE_NAMES` / `EXCLUDED_FILE_GLOBS` / `EXCLUDED_DIR_NAMES`, or add a KEEP carve-out. - **An omitted binary the build needs** → add `"snapshot/rel/path": "install/rel/source"` to `RESTORE_FROM_INSTALL` (it auto-renders into `_REBUILD_INSTRUCTIONS.md`). - **Guarantee a critical file** → add its snapshot-relative path to `REQUIRED_SNAPSHOT_FILES`. - **A new secret-bearing config** → extend `_wants_redaction()` to return `'json'`/`'yaml'` for it (and add the key suffix to `_SECRET_KEY_RE` if it's a novel secret-key name). - **A new state/output dir leaking in** → add its name to `EXCLUDED_DIR_NAMES`. --- ## Done criteria (all must hold) 1. `sweep_self_modify.py` exits clean (no `[FINDING]`); every `[note]` is reviewed + intended. 2. Every new top-level repo path from the since-last-tag diff is classified and handled. 3. The generated snapshot builds error-free and contains every `REQUIRED_SNAPSHOT_FILES` entry. 4. Every `build.py` SOURCE input is in the snapshot or in `RESTORE_FROM_INSTALL`. 5. Zero live secrets in the snapshot (redaction check + value scan pass). 6. The manifest + `_REBUILD_INSTRUCTIONS.md` are accurate; restore targets are real. 7. `python -m ruff check copy_source_assets.py` is clean. 8. (Ideal) you eyeballed a `--keep` snapshot and spot-confirmed the newest feature's source files are physically present. --- ## Companion references - `tlamatini-self-update-inclusion` — the sibling skill guarding the **runnable release** (`build.py` carriers + the update swap). Run BOTH after any feature: one proves users can *receive* the feature, this one proves Tlamatini can *rebuild herself with* it. - `docs/claude/architecture.md` → *Self-Knowledge & Self-Modification* and `docs/claude/gotchas.md` → *Building & Packaging* for the `--self-modify` flag mechanics. - `VERSIONING.md` — the rebuild produces the next version from the git tag.