# MIND-Mem — Persistent AI Memory System ## Overview BM25F + vector hybrid search memory system for AI agents. Published on PyPI: `pip install mind-mem` **v4.1.1** (released 2026-06-14) — Postgres backend parity: the recall / scan / governance / export / reindex / daemon feature layer was reading the local Markdown corpus + SQLite FTS index and ignoring the configured `block_store` backend, so Postgres users' blocks were invisible to those features. Added a shared backend-aware `storage.iter_active_blocks()` + `PostgresRecallBackend` (BM25 + pgvector RRF), backend-aware/read-only-safe `build_index`/`index_status`, pgvector cross-schema detection, daemon cron module path, `init_workspace` PG setup, MCP `_check_workspace` backend-aware gate. **SQLite/markdown default unchanged** (5566 tests, no regression); 81 new both-backend tests. Builds on **v4.1.1** (released 2026-06-05) — security-hardening companion to v4.0.16 (same audit). At-rest encryption relabeled honestly (256-bit HMAC-SHA256 keystream + encrypt-then-MAC, **not** AES-256/SQLCipher; recall index not encrypted) + 256 MiB encrypt/decrypt DoS cap + `.mind-mem-keys` 0700 / salt 0600 / decrypt-audit-jsonl 0600. `verify_model` warns on in-tree-key (self-describing) verification + manifest skips sidecars only at root (no smuggling). MIC-b parser bounds `n_syms`/`n_types`. gRPC binds 127.0.0.1 by default (was `[::]`). `tenant_kms` (real AESGCM) unchanged. **v4.0.16** (released 2026-06-05) — correctness-hardening release from a six-agent audit. Postgres backend: `delete_block` silent-rollback (missing `deleted_blocks` table aborted the txn), `diff()` AND/OR precedence + `active` metadata leak, `restore()` `file_path` wipe, new `ping()` health probe surfaced in `mm doctor`. Replication: `ReplicatedPostgresBlockStore` now forwards `embedding` and implements `hybrid_search`/`backfill_embedding`/`ping`. Recall: FTS5 `bm25()` weight shift (`block_id` stole `statement`'s weight) + unified `_apply_post_filters()` so sqlite/vector paths no longer drop `lifecycle`/`event_id`/`min_maturity`. Governance/concurrency: `federation.resolve_conflict` rowcount guard, `apply_proposal` re-validate under lock (double-apply), `conflict_resolver` winner/loser hash mapping, `ConnectionManager.close()` write-lock, tier-schema ALTER race. MCP: DB errors become structured responses instead of crashing the stdio server. 5465 tests (live PG). No `mind-mem-4b` retraining. Builds on **v4.0.15** (MindLLM backend + token rotation + time-bounded recall + N-08/T-007). **v4.0.12** (released 2026-05-19) — `build_index` perf fix (#530: 55.9 s → 0.19 s by bounding `extract_facts` scan text) + Windows CI green (cross-platform `ConnectionManager` teardown in the recursion regression test). Builds on **v4.0.11** (security code-scanning alerts #181–#189: CRLF log-sanitization + audited nosec), **v4.0.10** (recall↔query_index recursion fix, 100 KB block_parser truncation fix, observability/logging crash-safety, telemetry kill-switch, query-expansion synonym narrowing), and **v4.0.9** — Predicate.register() runtime API + CI matrix fully green across 12 OS×Python-version rows. * **Added**: `Predicate.register(name)` on `knowledge_graph` — returns a `_RuntimePredicate` sentinel (str subclass with `.name`/`.value` enum-shaped attrs) so downstream adapters can extend the predicate set without forking the closed enum. * **Fixed**: live LLM HTTP calls in two test files (`test_cross_encoder_auto_enable`, `test_query_expansion_auto_enable`) that hung Windows CI; circuit-breaker timing flake on Windows (recovery 0.05s → 0.1s; sleep 0.06s → 0.15s — margin > Windows 15.6ms tick); mypy `_RuntimePredicate` slot annotations. * **CI infra**: `--cov` on ubuntu-3.12 only (other matrix rows skip instrumentation to fit GitHub's 7 GB budget); `pytest-timeout=120s --timeout-method=thread` surfaces hangs by name; OOM-prone concurrency files + slow `build_index` regression (refs #530) marked file-level `pytestmark = pytest.mark.stress`. CI run id + job count are recorded in CHANGELOG.md per release, not pinned here (they go stale on every flake — see `feedback_ci_green_claim_discipline`). Builds on **v4.0.8** (released 2026-05-14) — closes 4 open issues. **#526** ACL `_get_request_scope` fail-closed: introspection exceptions return `"deny"` sentinel (was silent `None`→`"user"` downgrade); decorator short-circuits to reject the call; `acl_introspection_failed` log + `mcp_acl_introspection_failed_total` metric (Critical). **#527** THREE_WAY_MERGE upserts winner_version into `block_tier_vclock` for the synthetic merge agent AND both fork agents — resolved conflicts no longer recur (Critical, functional). **#528** every three-way merge emits a `three_way_merge_resolved` log with SHA-256 hashes of left/right/merged payloads for operator audit (Critical, HTTP transport). **#529** FederationClient hardened — scheme allowlist (rejects `file://`/`ftp://`/etc.), same-origin redirect handler (blocks SSRF pivot to cloud metadata), `MAX_RESP_BYTES` 1 MiB response-size cap (High). 18 new regression tests in `tests/test_issue_52[679]_*.py` + 75 existing pass. Plus 5 missing file-level `pytestmark = pytest.mark.stress` markers added to OOM- prone concurrency files (`test_concurrency_stress.py`, `test_filelock_stress.py`, `test_v4_concurrency.py`, `test_v4_round4_concurrency.py`, `test_concurrent_integration.py`). Builds on **v4.0.7** (released 2026-05-14) — test-only fix: the post-#508 ACL hardening (defence-in-depth) gates every decorator call against `ADMIN_TOOLS ∪ USER_TOOLS` even with `MIND_MEM_ACL_DISABLED=true`, so `tests/test_mcp_v140.py::TestObservabilityDecorator::test_failure_increments_failure_counter`'s ad-hoc `failing_tool` was rejected before its body ran (the test expected `ValueError` to propagate). Patched the test to register `failing_tool` in `USER_TOOLS` for the test scope (both `mind_mem.mcp.infra.acl.USER_TOOLS` and the import-time binding in `mind_mem.mcp.infra.observability`). Decorator unchanged — the defence-in-depth behaviour is correct. Builds on **v4.0.6** (released 2026-05-14) — PyPI badge alignment + CI green. README badge block was rendering uncentred on PyPI because lines 9–26 used 2/4-space leading indentation; PyPI's strict CommonMark treats 4-space indented lines as a code block. Flushed left. Also: applied `ruff format` to 10 drifted files (closes the `lint` Format-check failure); added `-m "not stress"` to both CI pytest steps (closes the OOM kills on ubuntu 3.12/3.14 from `test_niah`- class stress tests); marked Python 3.14 matrix rows as `continue-on-error` (still pre-release). No source/test changes. Builds on **v4.0.5** (released 2026-05-14) — Docs/badges aligned + release workflow idempotent. README badges + comparison table now match ground truth (`tests-5155+`, `clients-18`, `audit-cross-model`, v4.0.5's `84` MCP tools); CLAUDE.md drift cleared (`MCP Tools (81) → (84)`, `16 → 15` clients). `.github/workflows/release.yml` `pypa/gh-action-pypi-publish` step passes `skip-existing: true` so tag re-pushes and local-twine races stop turning the Release badge red. Same wheel surface as v4.0.3, no code/test changes. Builds on **v4.0.4** (released 2026-05-14) — docs-only patch: PyPI README logo now resolves (relative `assets/logo.png` rewritten to absolute GitHub raw URL — PyPI does not resolve relative paths against the source repo). Same wheel surface as v4.0.3. No code/test changes. Builds on **v4.0.3** (released 2026-05-14) — Postgres-backed recall pipeline fix. `recall()` in `_recall_core.py` now dispatches to the configured backend at the library entry-point (previously only `python -m mind_mem.recall` honored it, so `mm recall` against a PG workspace returned `[]`). `_load_backend` now tolerates non-dict `recall` config (falls through to BM25 scan instead of crashing). `mm doctor --rebuild-cache` creates the SQLite `recall.db`, runs `_init_schema`, and populates the FTS5 `blocks_fts` virtual table on first run against a PG-backed workspace. Closes #524 + #525. Targeted recall/doctor/rebuild/error_paths/ sqlite_index suite: 636 passed, 6 skipped, 0 failed. `mind-mem-4b` weights unchanged — CLI/library fix only, zero retraining required. Builds on **v4.0.2** (released 2026-05-13) — Security + correctness audit pass over the v4.0.1 surface: 1 Critical / 12 High / 18 Medium / 12 Low / 3 Info findings closed. HMAC-equal token compare, Origin allowlist, OPTIONS rejection, per-client sliding-window rate limit, symlink TOCTOU close, arch-mind flag-injection guard, `--token` CLI removal, LWW wall-clock semantic, BEGIN IMMEDIATE rowid pin, `±Inf`/`None` preimage rejection, RRF date-freshness dedup, copy-on-write temporal decay, lru-cached query-type detection. `mind-mem-4b` weights unchanged — zero probe-surface overlap. Builds on **v4.0.1** (2026-05-11) — federation wire transport (`/federation/{vclock,write,resolve,conflicts}` over HTTP + stdlib `FederationClient`). Builds on **v4.0.0** (released 2026-05-10) — Cognitive kernel, knowledge graph, resilience suite, observability layer. All surfaces flag-gated under `v4.` in `mind-mem.json`. No breaking changes. `mind-mem-4b` retrained at **109/109 = 100%** on the un-softened harness (14 new `V4_SURFACES` probes; `qg.escape_hatch` and `lin.cites=0.8` gaps from v3.12.1 confirmed fixed). 376 v4 unit + 38 concurrency + 22 held-out paraphrase tests. Architecture audited at unanimous **10/10** in a cross-model review. Builds on **v3.12.1** (2026-05-10) — patched eval, model card honesty. Builds on **v3.12.0** (2026-05-09) — strict quality gate, lineage→staleness. Builds on **v3.11.0** (2026-05-08) — typed lineage edges, recall explainability. Builds on **v3.9.0** — **81 tools**, 4000+ tests, native MCP for 17 AI clients, Postgres+pgvector, full-fine-tune local model, at-rest encryption, tier decay, governance alerting, MIC/MAP wire format, HTTP transport, daemon, inbox ingestion, pipeline hash, persona projection, Kahn walkthrough. ## Architecture ``` src/mind_mem/ — Main package (src layout; flat modules, not core/governance/ingestion sub-packages) block_store.py — BlockStore (+ connection_manager.py pooled access) contradiction_detector.py, drift_detector.py, audit_chain.py, alerting.py, governance_gate.py — governance: contradiction/drift detection, proposals, audit chain, alerting hooks _recall_core.py, hybrid_recall.py, recall_vector.py — retrieval core mcp_server.py — MCP server monolith (107 tools, 8 resources) mcp/ — per-domain MCP tool modules (mcp.tools.*) inbox.py, entity_ingest.py, ingestion_pipeline.py — auto-ingestion. ingestion_pipeline was deleted in 5.0.0 as an unreachable webhook write path; RESTORED and WIRED in 5.0.1, and the concern that justified deleting it is answered by construction: the drain path IS the gate, so every event goes through admit_block. skill_opt/ — Skill optimization hook_installer.py — Client hook installation (v3.1.1+) http_transport.py — Stdlib HTTP REST adapter (v3.9.0) daemon.py — Background dream-cycle/intel-scan loop (v3.9.0) inbox.py — File-drop folder ingestion (v3.9.0) pipeline_hash.py — Hash-of-code invalidation (v3.9.0) personas.py — brief/detailed/technical projection (v3.9.0) walkthrough.py — Dependency-ordered learning sequence (v3.9.0) quality_gate.py — Deterministic quality validation (v3.11.0) block_lineage.py — Typed relationship edges (v3.11.0) lineage_staleness.py — BFS staleness propagation (v3.12.0) v4/ — the v4.0.0 surfaces. Twelve were deleted in 5.0.0 because nothing in the product imported them (cognitive_kernel, surprise_retrieval, block_kinds, block_metadata, kind_summaries, embedding_pipeline, pq, hnsw_kind_index, backpressure, health, logging_context, observability). ALL are restored and wired as of 5.0.1. They had tests -- up to 29 files each -- which was read at the time as evidence that test count does not prove reachability. True, and it proves nothing about WORTH either, which is the mistake that cost 47 modules. `scripts/check_reachable_modules.py` now reports "deliberately waiting" (with the trigger that flips it) separately from "no recorded decision", so an unwired module is never again read as a dead one. federation.py — VClock + conflict log + MergeStrategy (v4.0.0) federation_client.py — federation transport client self_editing.py — block_edits + propose/approve/reject (v4.0.0) block_versioning.py — block version history circuit_breaker.py — CircuitBreaker + @circuit_breaker (v4.0.0) feature_flags.py — 52 flags + is_enabled/require_enabled tests/ — pytest suite (12,566 test functions incl. 376 v4 unit + 38 concurrency + 22 paraphrase probes) mind/ — MIND scoring kernels (.mind) docs/ — User + integration docs (35+ files) ``` ### Key Components - **BM25F retrieval** with Porter stemming + RM3 query expansion - **Hybrid search**: BM25 + vector search with RRF fusion (sqlite-vec on the SQLite backend; pgvector + HNSW on the Postgres backend) - **Backends**: SQLite (default, zero-deps) or Postgres (pgvector + HNSW + GIN, with replicated read/write routing in v3.9) - **A-MEM blocks**: metadata evolution (importance, access, keywords) - **9-type intent router** with adaptive confidence weights - **Cross-encoder reranking** (opt-in, config-gated) - **Governance engine**: contradiction detection, drift, proposal queue, alerting hooks (webhook / Slack) - **ConnectionManager**: Thread-safe SQLite pool with WAL read/write separation - **Delta-based snapshot rollback**: MANIFEST.json for O(manifest) restore - **At-rest encryption** (v3.0.0+, opt-in): 256-bit HMAC-SHA256 keystream + encrypt-then-MAC over BlockStore content files (NOT AES/SQLCipher; the FTS5/sqlite-vec recall index is not encrypted) - **Tier decay** (v3.0.0+): TTL/LRU aging for lower tiers - **Audit-integrity patterns** (v2.10.0+): Q16.16 fixed-point scoring in audit hash preimages, TAG_v1 NUL-separated composition for collision resistance - **Local model** (v4.0.0): `star-ga/mind-mem-4b` — full fine-tune of Qwen3.5-4B retrained for v4.0.0 (v4 weights revision). Trained on the 83-tool surface, including v4 features (cognitive kernel, block kinds, tier memory, self-editing), and the corrected `KIND_DECAY['cites']=0.8` value. Prior v3.12.0-fullft weights pinned at `v3.12.0` HF revision. Prior v3.0.0 QLoRA at `v3.0.0`. These historical evaluations do not establish competence over the current server's additional tools. See `docs/mind-mem-4b-setup.md`. - **Client integrations:** the registry contains 20 AI clients, including 11 with MCP configuration formats. `mm install-all` uses each client's supported integration. See `docs/client-integrations.md`. ### MCP Tools (107) Grouped surfaces (full list in `docs/api-reference.md` and `src/mind_mem/mcp_server.py`): recall, hybrid_search, find_similar, prefetch, intent_classify, retrieval_diagnostics, propose_update, approve_apply, reject_proposal, rollback_proposal, delete_memory_item, export_memory, memory_evolution, scan, list_contradictions, reindex, reindex_dirty, index_stats, category_summary, verify_chain, verify_merkle, list_evidence, anchor_root, anchor_history, get_mind_kernel, list_mind_kernels, mic_convert_tool, mic_inspect_tool, compile_truth_walkthrough, recall_with_persona, pipeline_status, encrypt_file, decrypt_file, graph_query, traverse_graph, ontology_load, lint, lint_autofix, and additional tools listed in the API reference. Names are verified against the registry by `scripts/check_tool_surface.py --check-doc-names`, which fails on a tool name that no longer exists -- this list previously carried 11 ghosts (create_snapshot, briefing, import_memory, audit_replay, ...). ## Config - Config file: `mind-mem.json` (NOT `mem-os.json` — renamed) - Auth header: `X-MindMem-Token` (NOT `X-MemOS-Token` — renamed) - `governance_mode` (NOT `self_correcting_mode` — renamed) - LLM backend: default `ollama`; switch to `openai-compatible` to point at a vLLM/exllamav2 endpoint running `mind-mem-4b`. ## Testing ```bash pytest # full suite (12,566 test functions) pytest tests/test_retrieval.py # specific module pytest -x --tb=short # stop on first failure python3 scripts/check_docs_alignment.py --print # the test-function count (static; a run's count is machine-dependent) ``` ## Conventions - Python 3.10+ with type hints everywhere - src layout: `src/mind_mem/` - All public functions have docstrings - No `print()` in library code — use `logging` - SQLite WAL mode for concurrent reads ## Benchmarks (LoCoMo, external LLM judge) Canonical number lives in `docs/benchmarks.md` — do not restate a figure here. Current: full 10-conversation set (1986Q), Acc>=50 73.8%, Mean 70.5, BM25-only backend. (Prior `77.9`/`82.3`/`88.5` was a mislabeled 199-question conv-0 subsample, not the full set — see `docs/benchmarks.md` for the correction.) ## Git - Remote: star-ga/mind-mem (public) - Author: `STARGA Inc ` - CI: GitHub Actions (enabled); release workflow on tag push with trusted PyPI publishing via OIDC (`environment: pypi`)