git:20260822.94279fe to git:20260901.484ab7d

- # mempalace — Project Guide for Claude Code

-

- ## 0. Pre-apply simplicity, reuse, and boundary review

-

- This rule runs before any edit, patch, backlog mutation, commit, deploy, or

- delegated executor prompt. Review the planned change before applying it; finding

- the problem after implementation is already a process failure. Write one verdict

- line for every check below. An unwritten check was not run.

-

- - **DRY** — search by behavior before searching by name. Use

- `mempalace_code_search` or `mempalace_explain_subsystem` first when available,

- then bounded `rg -l`, signature, and call-site checks. Name the existing owner

- found. Extend it when it has the same contract, lifecycle, and state store.

- - **KISS** — state the task in one sentence and select the smallest change that

- closes it. Prefer deletion or extension over addition.

- - **YAGNI** — add no layer, abstraction, option, mode, gate, or generality that

- the stated task does not require now.

- - **Duplicates** — create or retain no second helper, client, config loader,

- mutation route, manifest, or documentation paragraph. Consolidate shared

- production behavior into its existing owning module under `mempalace_code/`;

- consolidate script-only behavior into the existing owning script. Do not

- create catch-all `utils`, `helpers`, or `scripts/lib` modules without a proven

- incompatible owner or lifecycle. Run the owning suite before and after shared

- code changes.

- - **Garbage** — remove what the change supersedes. Leave no dead code or flags,

- orphaned files, stale docs, scratch or `tmp/` artifacts, commented-out blocks,

- or one-shot scripts.

- - **Drunk-user path** — assume a human can lose context, hold stale assumptions,

- omit or contradict facts, provide malformed input, repeat or reorder steps,

- and retry after an ambiguous result. Use safe defaults, explicit state and

- authority checks, bounded idempotent actions, contradiction detection,

- confirmation proportional to irreversibility, and one concrete recovery

- command. Test critical paths with lost context, stale state, malformed input,

- duplicate or reordered actions, and partial execution. Never require retained

- conversational context to avoid destructive or irreversible behavior.

- - **Drunk-LLM path** — apply the same failure model to prompts, specifications,

- schemas, APIs, handoffs, approvals, and recovery procedures across

- human-to-LLM, LLM-to-human, and LLM-to-LLM boundaries. Avoid near-identical

- names or paths, silent flag meaning changes, ambiguous omitted-argument modes,

- mutating defaults, and split instructions that disagree. Make state,

- authority, inputs, outputs, invariants, and recovery machine-verifiable where

- practical. If an agent can plausibly choose the wrong path, fix the design.

-

- A failed check ends the attempt. Revise the plan, write all seven verdicts again,

- and proceed only when every check passes. Re-run the gate after scope, state,

- authority, or ownership changes.

-

- Reuse-first is mandatory before creating a file, module, helper, script, route,

- manifest, or documentation block. A new implementation requires one concrete

- incompatibility: a different owner, contract, lifecycle, or state store.

- "Cleaner", "safer", "isolated", "easier to reason about", effort, and fear of

- changing the existing owner are not incompatibilities. A near-copy is a defect;

- fixing the existing owner is the task.

-

- Preserve the requested change class and use the existing generator, validator,

- commit, and release path for data, copy, layout, and generated-artifact changes.

- Limit the diff to the owning source, required generated artifacts, and the

- smallest existing regression. The first unrelated tool or gate failure ends that

- attempt: restore or confirm safe state, report the blocker, and request a separate

- owner decision before changing tooling, publishers, runbooks, or architecture.

- Stop when a repair grows beyond roughly three times its initial estimate or

- crosses the stated component boundary.

-

- Add a gate only for a failure that already occurred in this repository, citing

- its date, commit, or backlog item. Data loss and a live user outage are the only

- exceptions. The default number of new gates is zero; prefer backup, atomic

- replace, and post-state verification. Ask the owner before adding any other gate,

- naming the prior failure and what breaks without the gate.

-

- ## Stack

-

- - **Python** 3.11+ (supports 3.11–3.14)

- - **Storage**: LanceDB (core, crash-safe vector DB — no server required)

- - **Embeddings**: sentence-transformers (local, no API key)

- - **Config**: PyYAML

- - **Linting / formatting / typing**: Ruff + Pyright

- - **Tests**: pytest

- - **Package manager**: uv (preferred) or pip

-

- ## Dev Setup

-

- ```bash

- # With uv (preferred)

- uv pip install -e ".[dev]"

-

- # With pip

- pip install -e ".[dev]"

- ```

-

- No Docker required. Everything runs locally in a venv or with pipx.

-

- Optional extras:

- - `.[chroma-migration]` — ChromaDB-to-LanceDB migration bridge; capped below

- ChromaDB 1.x while GHSA-f4j7-r4q5-qw2c affects the available 1.x line

- - `.[chroma]` — deprecated compatibility alias for the migration bridge

- - `.[spellcheck]` — autocorrect support for room/wing names

-

- ## Running Tests

-

- **Important**: Use the Python from your mempalace virtualenv (pipx venv, `.venv`, etc.), not the system Python which may lack dev deps.

-

- ```bash

- # Full suite (stop on first failure)

- python -m pytest tests/ -x -q -m "not needs_network"

-

- # Per-module

- python -m pytest tests/test_storage.py -v

- python -m pytest tests/test_mcp_server.py -v

- python -m pytest tests/test_miner.py -v

- python -m pytest tests/test_convo_miner.py -v

- ```

-

- ## Linting / Formatting

-

- ```bash

- # Check

- ruff check mempalace_code/ tests/ scripts/

-

- # Format check

- ruff format --check mempalace_code/ tests/ scripts/

-

- # Type check (gating in CI — must exit 0)

- python -m pyright --pythonpath "$(python -c 'import sys; print(sys.executable)')"

-

- # Auto-fix lint

- ruff check --fix mempalace_code/ tests/ scripts/

-

- # Auto-fix format

- ruff format mempalace_code/ tests/ scripts/

- ```

-

- Line length: 100. Target: py311. Quote style: double.

-

- ## Key Modules

-

- | Module | Purpose |

- |--------|---------|

- | `storage.py` | LanceDB vector storage — add, search, delete, health_check, recover |

- | `backup.py` | Tarball backup/restore — `mempalace-code backup`, scheduled backups |

- | `miner.py` | Code project miner — walks source files, extracts drawers |

- | `convo_miner.py` | Conversation miner — ingests Claude/ChatGPT/Slack exports |

- | `searcher.py` | Semantic search — query palace with optional wing/room filters |

- | `knowledge_graph.py` | Temporal KG — entity-relationship triples with validity windows |

- | `layers.py` | Tiered context loading — L0/L1/L2/L3 wake-up layers for local models |

- | `palace_graph.py` | Graph traversal and tunnel detection across wings/rooms |

- | `mcp_server.py` | MCP server — exposes palace tools to Claude Code and other MCP clients |

- | `watcher.py` | File watcher — `watch_and_mine`, `watch_all`, launchd/cron schedule rendering |

- | `cli.py` | `mempalace-code` CLI entry point — init, mine, mine-all, watch, search, health, repair, backup |

-

- ## Architecture Principles

-

- 1. **Verbatim-first** — store content exactly as provided; do not summarize or compress drawers.

- 2. **Local-first** — no external APIs required; all embeddings run on-device via sentence-transformers.

- 3. **Zero-API-by-default** — LanceDB and sentence-transformers work offline; API integrations are opt-in.

-

- ## Storage Backend

-

- - **LanceDB** is the core backend (installed by default via `lancedb>=0.20`).

- - **ChromaDB** is supported only as migration input through

- `mempalace-code migrate-storage SRC DST --verify`. Install

- `.[chroma-migration]` for that bridge. The deprecated `.[chroma]` alias is

- retained for existing scripts, and the dependency stays capped below 1.x while

- GHSA-f4j7-r4q5-qw2c affects the available 1.x line.

-

- ## Embedding Model Policy

-

- - **Current default**: `all-MiniLM-L6-v2` (384d, sentence-transformers).

- - **No-regression rule**: any embedding model change must match or beat MiniLM on LongMemEval R@5 (text retrieval). Text quality is non-negotiable — this is a code-first fork but natural language search (conversations, commits, decisions) must not degrade.

- - **No code-only models**: CodeBERT, UniXcoder, etc. improve code at the expense of prose. Only general-purpose sentence-transformers that handle both are candidates.

- - **Gate**: model upgrades are gated behind A/B benchmark results.

-

- ### Benchmark Results — 2026-04-09 (BENCH-EMBED-AB)

-

- Code retrieval on the mempalace repo (20 known-answer queries, 469 chunks):

-

- | Model | R@5 | R@10 | Embed(s) | Query(ms) | Index(MB) |

- |--------------------|-------|-------|----------|-----------|-----------|

- | all-MiniLM-L6-v2 | 0.950 | 1.000 | 15.2 | 15.9 | 17.0 |

- | all-mpnet-base-v2 | 0.900 | 1.000 | 47.5 | 30.5 | 17.7 |

- | nomic-embed-text-v1.5 | 0.950 | 1.000 | 85.4 | 45.8 | 17.7 |

-

- Per-category R@5:

-

- | Model | architecture | class_lookup | cross_file | function_lookup |

- |--------------------|:---:|:---:|:---:|:---:|

- | all-MiniLM-L6-v2 | 0.800 | 1.000 | 1.000 | 1.000 |

- | all-mpnet-base-v2 | 0.600 | 1.000 | 1.000 | 1.000 |

- | nomic-embed-text-v1.5 | 1.000 | 1.000 | 1.000 | 0.833 |

-

- **Recommendation: minilm remains default.**

-

- - mpnet regresses on code R@5 (0.900 vs 0.950) while being 3× slower to embed and 2× slower at query. Eliminated.

- - nomic ties minilm on code R@5 (0.950) but is 5.6× slower to embed and 2.9× slower at query, with a 550 MB model vs 80 MB. No net gain.

- - Text-gate (LongMemEval) evidence was not collected — prerequisites missing (`benchmarks/data/longmemeval_s_cleaned.json` not present, `fastembed` not installed). Any future upgrade must pass the text gate before switching.

- - Full results: `benchmarks/results_embed_ab_2026-04-09.json`.

-

- ## Git Workflow

-

- - **Branch naming**: `feat/<slug>`, `fix/<slug>`, `chore/<slug>`, `docs/<slug>`

- - **Conventional commits**:

- - `feat:` — new feature or capability

- - `fix:` — bug fix

- - `docs:` — documentation only

- - `test:` — test additions or changes

- - `bench:` — benchmarks

- - `chore:` — maintenance, deps, tooling

- - **No force-push to `main`**.

- - PR merges go through the `feat/*` → `main` flow; squash if the branch is noisy.

-

- ## Operational Lessons

-

- - **Keep this file public-safe.** `CLAUDE.md` is part of the publishable repository. Do not write private remotes, hostnames, credentials, local machine paths, customer/project details, incident specifics, or non-public operational history here. Put private or machine-local lessons in a local-only note outside the published tree.

- - **Record reusable lessons only when they are public knowledge.** When a session exposes a project gotcha, publish step, verification boundary, or agent-behavior correction, add a concise durable note here only if it is safe for public readers and useful to future contributors.

- - **Verify the environment that will actually run the change.** GitHub Actions runtime changes are not proven by Python tests alone. Use local YAML/static checks such as `actionlint`, then verify the real hosted workflow run when action runtime behavior matters.

- - **Name the verification boundary.** If a workflow is tag-only or release-only, say that it was syntax-checked and version-checked but not execution-tested unless a real trigger was run. Do not imply full local coverage for hosted-only behavior.

- - **Check the intended public release target.** Before publishing, verify the repository, branch, tag, and workflow that public users will see. Do not assume local remote names or private mirrors represent public release truth.

- - **Treat release status as multiple independent facts.** Branch Tests, tag-triggered PyPI publish, GitHub Release creation, PyPI version visibility, and any deployment/release-environment status can diverge. Check all of them before calling a release published or latest; if one is red or missing, either fix it now or record the explicit remaining blocker.

- - **Test dependency drift with a fresh resolver.** Local `.venv` and `uv.lock` success can hide what GitHub Actions or users get from an unlocked `pip install`. For dependency-sensitive failures, reproduce in a clean pip environment matching the hosted workflow before declaring the tests fixed.

- - **Audit dependency targets before raising bounds.** For runtime, dev, and optional extras, check current and target versions against OSV or an equivalent advisory source, then run a resolver-level audit on a fresh environment. Do not raise optional legacy backends into advisory-affected ranges; hold or cap them and backlog the upgrade gate instead.

- - **Separate public and local release information.** Public docs may name package ranges, workflow categories, advisory IDs, and reproducible commands. Private remotes, tokens, local paths, hostnames, and non-public incident details belong only in ignored local notes such as `.codex-local/LESSONS.md`.

- - **Keep benchmark gates tied to measured baselines.** If a release benchmark fails, reproduce it locally against the pinned fixture, update the CI threshold only to the observed stable baseline, and backlog any desired quality increase separately.

- - **Do not call tests "local feature testing."** When asked to test new features locally, run the public CLI/MCP/API behavior itself, not only pytest. For each new feature, exercise at least one success path and one important failure/guard path when safe, record the exact command or request, and name any behavior that was covered only by tests.

- - **Exercise real integration surfaces before release claims.** Direct handler calls are useful for MCP compatibility, but they are not the same as a separate stdio MCP client. CLI help is not the same as executing the command. A release-readiness summary must distinguish unit tests, focused integration tests, direct API smoke, real CLI execution, and hosted/daemon behavior that was not run.

- - **Clean up smoke-test artifacts immediately.** Real backup, cleanup, and benchmark smokes can create archives, temp palaces, and result JSON. Put them under disposable temp paths, verify success/failure, then remove artifacts or explicitly report what remains.

- - **Treat palace disk growth as storage forensics first.** Compare backup size, live storage stats, row counts, and cleanup output before deleting anything. Preserve non-regenerable manual drawers and KG data, stop active writers when needed, use supported cleanup APIs, and verify health/status afterward.