tlamatini-self-update-inclusion git:20260916.92a5830A

Immutable. This exact content is served forever at /api/v1/blob/89baa39b35e6f000.

---
name: tlamatini-self-update-inclusion
description: Sweep the whole codebase and keep the SELF-UPDATE pipeline complete — so every new asset/feature added to Tlamatini is actually carried into a release by build.py AND survives (or is correctly replaced by) the self-update swap. Invoke whenever you add/rename a new agent, top-level file or directory, dependency, bundled runtime, migration, static/template asset, or any "ship it next to the exe" artifact — and ALWAYS after a feature like Blenderer or the self-update capability lands. Audits the three files that own the pipeline (build.py, agent/self_update.py, apply_update.ps1) against four hard invariants, with a runnable sweep script that never forgets the minimal thing. Pairs with copy_source_assets.py (the self-modify snapshot) and VERSIONING.md.
---
<!--
═══════════════════════════════════════════════════════════════════
  ✦  T L A M A T I N I  ✦   —   "one who knows"
  Created by  Angela López Mendoza   ·   @angelahack1
  Developer · Architect · Creator of Tlamatini
  Tlamatini Author Banner — do not remove (Angela's name is kept in every build)
═══════════════════════════════════════════════════════════════════
-->

# Tlamatini — Self-Update Inclusion Sweep

> **Audience:** Claude Code working ON the Tlamatini codebase for **Angela**.
> **Goal:** guarantee that *every* asset a new feature introduces is (1) **carried into
> the release** by `build.py`, and (2) **handled correctly by the self-update swap** —
> either preserved (user data) or replaced (app code). Nothing minimal forgotten, ever.

This skill is the **safety net for shipping**. A feature can be 100% correct in source and
still be invisible to every existing user after they click *About ▸ Check for updates* —
because the asset was never bundled, or because the swap deleted/kept the wrong thing. This
sweep makes that class of bug impossible to ship silently.

---

## Pipeline files this skill owns

| File | Role in the pipeline |
|---|---|
| **`build.py`** (repo root) | Assembles the release tree (`dist/manage` → `pkg.zip`). If an asset isn't carried by one of its 7 mechanisms (below), it is **not in the download**, so a self-update can never deliver it. |
| **`Tlamatini/agent/self_update.py`** | In-app updater: checks GitHub, downloads + unzips + stages the new build, hands off to the PowerShell swapper. Its **docstring preserve list** documents what survives. |
| **`apply_update.ps1`** (repo root) | The external file-swapper. Its **`$Preserve` array** is the *authoritative, executed* contract for what is kept vs replaced. Renames `agents → agents_backup`, then full-replaces everything not preserved. |

> `apply_update.ps1` must itself be shipped by `build.py` (`required_file_copies`) so a self-updated
> install carries the *next* updater. The pipeline is self-hosting — this is invariant #1's
> most easily-forgotten case.

---

## How the release is assembled — the 7 carrier mechanisms in `build.py`

Every runtime asset needs at least one of these carriers (some deliberately have two destinations). When you add an asset, ask
"which carrier moves it?" If the answer is "none", it will NOT ship.

| # | Mechanism (`build.py`) | Carries | Auto-includes new files? |
|---|---|---|---|
| 1 | **PyInstaller import graph → PYZ** | every `.py` reachable from the import graph (incl. lazy `from . import x`) — views, tools, registries, `self_update.py`, **migrations** | ✅ yes, if imported/in a collected package |
| 2 | **`--add-data` list** | whole trees: `agent/templates`, `agent/static`, `staticfiles`, `agent/skills_pkg`; single files: `config.json`, `prompt.pmt`, `Tlamatini.md`; dependency data files | ✅ for files *inside* an already-listed tree; ❌ for a **new** top-level tree |
| 3 | **`optional_dir_copies` → install root** | `agent/images`, **`agent/agents`** (the whole agent-template tree → new agents auto-ship), `agent/skills_pkg` | ✅ new agents/skills inside these dirs |
| 4 | **`optional_file_copies` / `required_file_copies` → install root** | `config.json`, `prompt.pmt`, `Tlamatini.md`; required `README.md`, `agents_descriptions.md`, `apply_update.ps1`, `preserved_user_state.json`, standalone `sqlite_copy.py` | ❌ a **new** root-level required file must be added by hand |
| 5 | **`support_files` → install root** | the `.ps1` helpers (`Tlamatini.ps1`, `register_flw`/`unregister_flw`, `CreateShortcut`/`RemoveShortcut`), `Tlamatini.ico`, `CreateShortcut.json`, `cat_art.py` | ❌ a **new** root-level support file must be added by hand |
| 6 | **bundled runtimes + deps** | carried Python, `jre`, `git`, `ms-playwright`; PyInstaller **hidden imports**, **`--collect-all`** (e.g. `ffpyplayer`); **`requirements.txt`** | ❌ a **new** runtime / hidden import / collect-all / pip dep must be added by hand |
| 7 | **bespoke `shutil.copytree` block → install root** | **`security/`** (Angela's Blue-hat operator toolkit: `tlamatini_defender.ps1`, `tlamatini_whitelist_v2.ps1`, the `.bat` UAC launchers, `README.md`, and `automated_tests_of_security_assets.py`), copied near the end of the build with `ignore_patterns("security_logs", "*.log", "__pycache__")` | ✅ new files *inside* `security/`; ❌ a **new** bespoke tree needs its own block |
| — | **DB delivery** | `build.py` step 8a runs `migrate`, so the shipped `db.sqlite3` carries every migration's seeded rows (new agent row, `chat_agent_*` tool row, demo prompts) | ✅ rows ship; ⚠️ see the DB special case |

**The forgettable carriers are 2 (new top-level tree), 4, 5, 6, and 7** — anything that lives at
the repo root or needs an explicit PyInstaller flag or its own copy block. Mechanisms 1 and 3 are
automatic, which is exactly why a new *agent* needs no build edit but a new *root-level script* does.

> `security/` is mandatory. Missing source/copy failures abort the build, and
> `build_runtime_assets.py` inventories the tree (excluding evidence/logs) and checks
> every carried file. Register new runtime trees in `SOURCE_TREES` as well as their
> actual copy/add-data carrier; optional copying is unsuitable for required code.

---

## How an update swaps — preserve vs replace

`apply_update.ps1` does: validate staged build and backup helpers → stop running app
→ verify a WAL-aware SQLite backup → `agents → agents_backup` → **delete** old install
except `$Preserve` → **move in** new build except `$Preserve` → relaunch and migrate.

So every top-level entry is in one of three buckets:

- **PRESERVED** (`$Preserve`) — user data / runtime state. Kept across updates. Must equal the
  shared `preserved_user_state.json` contract, including runtime-writable dirs.
- **SWAP-BACKED** — `agents` only (renamed to `agents_backup`, then replaced). One backup kept.
- **REPLACED** — everything else (the exe, `python`/`jre`/`git`, `.ps1`/`.ico`, `prompt.pmt`,
  `Tlamatini.md`, `README.md`, `agents_descriptions.md`, `images`, `skills_pkg`, **`db.sqlite3`**).

---

## The FOUR invariants (this is the whole job)

### Invariant 1 — CARRY: every new asset reaches the release
For each asset a feature adds, a carrier (table above) moves it into `dist/manage`/`pkg.zip`.
The high-risk cases: a **new repo-root file** (→ `support_files` or `*_file_copies`), a **new
top-level tree** (→ `--add-data` or `optional_dir_copies`), a **new dependency** (→
`requirements.txt` + maybe hidden-import / `--collect-all`), a **new bundled runtime**.

### Invariant 2 — PRESERVE PARITY: the two preserve lists are identical
`apply_update.ps1` `$Preserve` (executed) **==** `self_update.py` docstring "Preserved across
the swap" list (documented). A drift here means the docs lie about what survives.

### Invariant 3 — PRESERVE CORRECTNESS: state preserved, code replaced
`$Preserve` **==** the names in `preserved_user_state.json` **==** the installer fallback
**==** the updater docstring list (case-insensitive sets). The top-level names from
`build.py::empty_dirs` must be a subset; catalogs, contacts and the separately built
uninstaller are also legitimate preserved entries. Add new state to all consumers.
`uninstall.py` intentionally has a different removal policy. Conversely an
**app-code** top-level entry (anything under `optional_dir_copies` like `images`/`skills_pkg`,
or `python`/`jre`/`git`) must **NOT** be preserved, or users get stuck on stale code forever.

### Invariant 4 — DB DELIVERY: new migration rows actually reach users
The live database sits inside replaced `_internal/`, but its data is preserved
through `DB/ToLoad`. After stopping the app, carried Python runs the shipped
`sqlite_copy.py` helper to produce a verified online backup including committed WAL
pages. Only success permits the swap and creates `post_update_migrate.flag`.
The next startup restores that database and applies new migrations. Never replace
this with a plain copy of `db.sqlite3`, and never preserve the old DB without
the first-launch migration path.

---

## THE SWEEP — run this every time

### Step 0 — run the deterministic checker (does 90% of the work)

```bash
python .claude/skills/tlamatini-self-update-inclusion/scripts/sweep_self_update.py
```

It parses the three files and reports `[PASS]` / `[FINDING]` for invariants 2, 3, the
root-`.ps1` census (invariant 1's worst case), app-dir-must-not-be-preserved (invariant 3),
and a migrations-since-last-tag count (invariant 4). Exit code is non-zero if any finding —
so it's usable as a pre-release gate. Fix every `[FINDING]` before shipping.

### Step 1 — diff since the last release and classify every new path

```bash
# what changed since the last shipped tag
git diff --name-status "$(git describe --tags --abbrev=0 --match 'v[0-9]*')"..HEAD
# any brand-new TOP-LEVEL repo entries (the highest-risk for "forgot to ship")
git diff --name-status "$(git describe --tags --abbrev=0 --match 'v[0-9]*')"..HEAD \
  | awk '$1=="A"{print $2}' | awk -F/ '{print $1}' | sort -u
```

For each **new top-level file or dir**, run it through the **asset taxonomy** below and confirm
its carrier is wired. New nested files inside `agent/static`, `agent/templates`,
`agent/agents/<x>`, `agent/skills_pkg`, or any `.py` in a package are auto-carried — note them
but they need no edit.

### Step 2 — the carrier checks the script can't fully judge (do by eye)

Run these greps and reconcile each hit against `build.py`:

```bash
# (a) New third-party imports in pool agents / app → requirements.txt + maybe hidden-imports/collect-all
grep -rnE "^\s*(import|from)\s+([a-z0-9_]+)" Tlamatini/agent/agents --include=*.py \
  | grep -ivE "import (os|sys|json|re|time|subprocess|socket|threading|pathlib|typing|shutil|logging|urllib|zipfile|tarfile|wave|base64|struct|math|datetime|tempfile|argparse|queue|signal|ctypes|glob|io|collections|functools|itertools)\b"
grep -nE "hiddenimports|--hidden-import|--collect-all|--collect-submodules" build.py
# (b) Any external EXE/runtime a new agent shells out to (like jre/git) → must be bundled
grep -rnE "subprocess|Popen|shutil.which|\.exe\b" Tlamatini/agent/agents --include=*.py | grep -iE "\.exe|which\(" | head
# (c) New requirements vs what build pins
grep -nE "_agent_libs|AGENT_DEP|pip install|requirements" build.py | head
```

If a new agent imports a new library, it must be in `requirements.txt` **and** importable by the
**carried Python** (see `bundle_carried_python` / the `_agent_libs` verify list) — otherwise the
frozen pool agent crashes at runtime even though the source is correct.

### Step 3 — confirm the asset actually lands in `pkg.zip` (ground truth)

The only 100%-sure check is to look at a real bundle. If a recent `dist/manage` or `pkg.zip`
exists, list it; otherwise note that a build is required to verify physically:

```bash
[ -f pkg.zip ] && python - <<'PY'
import zipfile
names = zipfile.ZipFile("pkg.zip").namelist()
for probe in ("apply_update.ps1","db.sqlite3","agents/blenderer/blenderer.py","Tlamatini.exe"):
    print(("OK  " if any(probe in n for n in names) else "MISS"), probe)
PY
```

If no bundle exists, do NOT claim it ships — say "verified in source/wiring; physical bundle
check needs a `python build.py` run."

### Step 4 — fix every finding, then re-run Step 0 until clean.

---

## Asset taxonomy — type → carrier → preserve?

| New asset | Carry via (build.py) | Preserve on update? |
|---|---|---|
| New **agent** (`agent/agents/<x>/`) | mech 3 (`optional_dir_copies` agents) + mech 1 (PYZ) — **automatic** | No (arrives via `agents` swap) |
| New **migration** (seeds rows) | mech 1 (PYZ) + build-time `migrate`; first-launch `migrate` applies it to the preserved user DB | User data survives; new schema/seed changes are applied |
| New **repo-root `.ps1`/script** | mech 5 `support_files`, or mech 4 `required_file_copies` for mandatory helpers such as `apply_update.ps1` — **manual** | No (app code, replaced) |
| New **repo-root required data file** | mech 4 `*_file_copies` — **manual** | No |
| New **top-level source tree** (new package dir to ship as data) | mech 2 `--add-data` / mech 3 `optional_dir_copies` — **manual** | No |
| New **static/template/skill** file (inside existing tree) | mech 2 / 3 — **automatic** | No |
| New **pip dependency** | mech 6 `requirements.txt` (+ hidden-import / `--collect-all` if dynamic) — **manual** | n/a |
| New **bundled runtime** (CLI the agent shells out to) | mech 6 bundler (mirror `bundle_git`/`bundle_java_runtime`) — **manual** | No (replaced) |
| New **runtime-writable dir** (app writes user data here) | mech `empty_dirs` (ship empty) — **manual** | **YES — update shared JSON, installer fallback, swapper and updater docs** |
| New **operator-facing toolkit tree** at the repo root (e.g. `security/`) | mech 3 `optional_dir_copies` (preferred) or mech 7 a bespoke `copytree` — **manual** | **No — it is APP CODE.** A fixed defender must reach a user who installed a broken one. Preserve only the *evidence* inside it (see below) |
| New **evidence/log dir INSIDE a replaced app tree** (e.g. `security/security_logs/`) | ignored by the carrier (never shipped) | **Not via `$Preserve`** — it would pin the whole parent. Stash-and-restore around the swap instead |
| New **config key with a secret** | already in `config.json` (preserved) | Yes (config.json preserved) |

The single most dangerous omission is the last-but-one row: a **new runtime-state dir** added to
`empty_dirs` but not to `$Preserve` + the `self_update.py` docstring → **wiped on every update.**
The sweep script flags exactly this.

---

## Where to make each fix

- **Carry a root file** → add to `support_files` (scripts/icons) or `required_file_copies`
  (data) in `build.py`, with a one-line comment on *why it must be next to the exe*.
- **Carry a new tree** → add an `--add-data` line (if read from the bundle) or an
  `optional_dir_copies` entry (if read from the install root).
- **Carry a dep** → `requirements.txt`; if PyInstaller can't see it, add a hidden-import or
  `--collect-all`; if a pool agent imports it, confirm the **carried Python** has it
  (`_agent_libs`).
- **Preserve a new state dir** → add the top-level name to `apply_update.ps1` `$Preserve`
  **and** shared JSON, installer fallback and `self_update.py` docs (keep them identical), and ship it empty via
  `build.py` `empty_dirs`.
- **Stop preserving stale app code** → remove it from `$Preserve` (+ docstring).
- **DB** → preserve committed data with verified SQLite online backup into `DB/ToLoad`, then restore it and run first-launch `migrate`. Keep `db.sqlite3` out of the top-level preserve set because it lives inside replaced `_internal/`.

---

## Done criteria (all must hold)

### PDF and database-helper carrier gate (2026-09-16)

The PDF backend and Image-Interpreter engine run inside the frozen web process.
Keep their explicit hidden imports / archive requirements and
`pyinstaller_hooks/hook-pymupdf.py`; the separate carried Python cannot satisfy
imports inside `Tlamatini.exe`. PDF.js assets are carried by the static trees,
and `context_files/pdf_canvas/` survives through the existing preserved
`context_files` directory.

`apply_update.ps1`, `preserved_user_state.json` and standalone `sqlite_copy.py`
are REQUIRED root assets, not warning-only support copies. The in-app updater
prefers the incoming swap script so new preservation rules apply immediately.
The swapper checks the backup helper/runtime before shutdown, requires verified
online backup before changing application files, and uses carried Python with
`-I` rather than a machine-installed interpreter. Keep helper source in the
self-modify snapshot and `agent.sqlite_copy` in the frozen archive too.

### Frontend carrier gate (v1.48.13)

Because `agent/static` is tree-carried, `dialog_theme.css`, `dialog_policy.js`, and `release_notes_renderer.js` ship automatically only if they remain inside that tree and are referenced by both source templates and collected-static output. Verify template load order and the startup-derived static cache stamp after JavaScript/CSS/template changes. Run tests only when the user's instructions permit them. Confirm the updater still uses the shared renderer/policy and that long-operation locks restore `data-bs-toggle`.

### External-MCP runtime/default carrier gate (v1.48.14 target)

`runtime_provisioner.py` and `external_mcp_defaults.py` are application code and must ship through the normal frozen/source carriers. The downloaded private runtime and persistent Memory graph live under `%LOCALAPPDATA%\Tlamatini`, outside the install swap, so they must **not** be added to installer payloads, `empty_dirs`, or `$Preserve`. `external_mcps.json` is different: it is preserved user state and a gitignored local catalog, replaced with maintained inactive defaults for public builds. Verify public `build.py` output contains only inactive `memory` + `sequential-thinking`, keyed/private builds take the explicit private path, `regen_secrets.py` handles catalog env secrets, and the public live-secret gate aborts unsafe output.

### v1.48.17 updater/parser and public-private build gate

The staged swap must retain `Uninstaller.exe`; Windows comments that explain this policy stay on standalone PowerShell lines so parser-sensitive continuations remain valid. Verify `agent/test_preserved_user_state.py` source-derives the code-seeded default catalog, proves the public builder clears `TLAMATINI_BUNDLE_EXTERNAL_MCPS`, and proves only the explicit private builder supplies it. Do not reintroduce an assertion that the environment variable is absent from `build.py`: the variable is intentionally read by the shared builder, while release entry points control whether it exists.

### v1.50.0 security-toolkit carrier + evidence-carryover gate

**`security/` is the first asset to use carrier mechanism 7** (a bespoke `copytree`, not a list),
and the first to need a **third bucket** beyond preserve/replace. Verify all three halves:

1. **CARRIED** — `build.py`'s security block copies `security/` to the install root, skipping
   `security_logs` / `*.log` / `__pycache__`. It fails CLOSED, with per-file receipt
   validation as well as the Step-0 source census.
2. **REPLACED, deliberately** — `'security'` is **NOT** in `apply_update.ps1` `$Preserve` and must
   stay out. It is application code: a corrected defender has to be able to reach a user running a
   broken one. Do **not** "protect" it by preserving it — that would freeze the toolkit forever.
3. **EVIDENCE CARRIED OVER** — `security/security_logs/` (alerts.log, monitor.log, the visible
   asset-test proof) is the operator's forensic evidence living *inside* that replaced tree, the
   same shape of problem as `db.sqlite3`. `apply_update.ps1` step **3c stashes** it to
   a unique `Temp/_security_logs_carryover_<id>` before deletion and step **5b restores** it.
   A failed stash ABORTS before deletion; failed restore LEAVES the stash, never
   overwriting other evidence. `self_update.py`'s
   docstring mirrors this. Never "fix" a failed restore by dropping the stash.

Also verify `security_logs` remains in `SKIP_DIRS` in **both** `build_complete_public_release.py`
and `check_private_data.py` (kept mirrored): the release scrubber must not rewrite forensic
artifacts, and the private-data scanner must not drown in the operator's own usernames / IPs.

### v1.50.0 release carrier gate

Verify `agent/agents/netspeed_calculator/`, migrations 0195-0197, its wrapped-tool wiring, prompt harness, and CSS/JS connector assets ship through the normal agents/static/migrations carriers. Verify `agent/sqlite_copy.py` is present in the frozen application and both DB menu views plus pre-Django swap import it. Verify Googler's updated `googler.py`, `config.yaml`, `test_googler_dorks.py`, and optional visible dork-hunt harness are carried together so neither the structured builder nor the two-tier plain-HTTP-first/browser-fallback resilience path can ship without its syntax/preset/retry/fallback contract. Verify the four HTTP routes, seven browser routes, explicit-engine Tier-0 bypass, tolerant booleans, and answer-route attribution together. Verify `skills_pkg/adding_external_mcp/` and all references ship with the skill tree. Private contact synchronization may create gitignored `contacts.private.json` for an explicit keyed build, but public output and `TlamatiniSourceCode/` must remain contact-empty; preserved runtime user state remains `contacts.json`, not the private build staging file. Resolve the current annotated release and `HEAD` using Git; do not reuse historical version/count claims.

1. `sweep_self_update.py` exits clean (no `[FINDING]`).
2. Every new top-level repo path from the since-last-tag diff has a wired carrier.
3. Shared JSON, swapper, installer fallback and updater docs agree; all `empty_dirs` top-level names are covered.
4. No app-code dir is preserved; no runtime-state dir is left unpreserved.
5. New deps are in `requirements.txt` and importable by the carried Python.
6. The DB story is coherent (replaced + honest docs, OR preserved + first-run migrate).
7. If a physical bundle was available, the probe in Step 3 shows the new assets `OK`; otherwise
   you stated a build is needed to physically confirm.
8. `python -m ruff check` clean on any edited `.py`; `apply_update.ps1` still parses.

---


### Integrity and release-mode gate (2026-09-16)

- Read `build_runtime_assets.py`, `preserved_user_state.json`, `install.py` and
  both complete-release wrappers as well as the three primary pipeline files.
- The checker parses `required_file_copies` with AST and `support_files`; a helper
  moved from one to the other is not a missing-carrier finding.
- Carry `build_runtime_assets.py` at the install root AND in the frozen module
  archive. The installer imports the same checker. `runtime-assets.json` records
  every file's size/hash, exact membership, version and boolean self-modify mode.
- Validate ZIP paths, membership and streamed hashes before publication,
  installer extraction and updater staging. Recheck staged bytes before shutdown.
  Reject legacy packages without receipts; rebuild them. A receipt is NOT a
  signature and cannot establish publisher authenticity.
- `--self-modify` requires both identity locations and the complete snapshot;
  default/`--no-self-modify` requires neither. Missing `pkg.zip` aborts wrappers.
- The PowerShell swapper requires the exact install-local `Temp/_update/staging`
  boundary, rejects reparse ancestors, and never falls back to a self-killing
  `taskkill /T`. Existing user state wins; absent state receives new defaults.
- Reinstallation retains the live `_internal/db.sqlite3` and WAL companions and
  requests first-launch migration; preserving only top-level `DB/` is insufficient.
- Retain the 1,990,000,000-byte final outer ZIP ceiling. Never prune required
  assets to meet it. Local frontend and PDF.js bytes stay pinned and CDN-free.
- Respect a no-tests request: these sweeps inspect/copy files only. Do not launch
  Django, rebuild, migrate, install, swap, or execute application test suites.
  Report source/snapshot evidence separately from a freshly built runtime.

## Companion references
- `copy_source_assets.py` (repo root) — the **self-modify** snapshot generator; its
  `REQUIRED_SNAPSHOT_FILES` completeness check is a sibling guarantee (that the *source* tree
  ships), distinct from this skill (that the *runnable release* ships + survives an update).
- `VERSIONING.md` — the git-tag version contract (a self-update compares tags via
  `self_update.is_newer`).
- `docs/claude/architecture.md` → *Self-Knowledge & Self-Modification* for the build flags.