DESIGN.md · git:20260903.0f6ca2c · 2026-09-03 · sha256 13e187ade9c1f826
DESIGN.md git:20260903.0f6ca2cF
Immutable. This exact content is served forever at /api/v1/blob/13e187ade9c1f826.
# Quarry Design Decision Log
This file is the authoritative record of design decisions, prior approaches, and their outcomes. **Every design change must be logged here before implementation.**
For system architecture, module responsibilities, configuration, and deployment, see [`docs/architecture.tex`](docs/architecture.tex). For user-facing documentation, see [README.md](README.md). For test tiers, counts, and strategy, see [`docs/architecture.tex`](docs/architecture.tex) §15.
## Rules
1. Before proposing ANY design change, consult this log for prior decisions on the same topic.
2. Do not revisit a settled decision without new evidence.
3. Log the decision, alternatives considered, and outcome.
---
## DES-001: Fire-and-Forget MCP Pattern
**Date:** 2026-02-08
**Status:** SETTLED
**Topic:** How side-effect MCP tools handle latency
### Design
Side-effect MCP tools (ingest, delete, sync, register, deregister, remember) return an optimistic response immediately and process in a bounded `ThreadPoolExecutor(max_workers=4)`.
Settings and database connections are **snapshotted at the tool boundary** and passed into background `_do_*` functions. This avoids a race condition where the mutable `_db_name` ContextVar could change between the tool call and the background thread's execution.
### Why This Design
MCP tool calls block the LLM's response stream. A 30-second ingest blocks Claude from responding. The bounded pool prevents resource exhaustion under concurrent requests. Exceptions in background threads are logged, not raised — the caller already received the optimistic response.
---
## DES-002: Protocol-Based Typing
**Date:** 2026-02-08
**Status:** SETTLED
**Topic:** Type safety for third-party libraries without stubs
### Design
Third-party libraries without type stubs (LanceDB) are typed via Protocol classes in `types.py`: `LanceDB`, `LanceTable`, `LanceQuery`, `ListTablesResult`, `OcrBackend`, `EmbeddingBackend`.
### Why This Design
Exact type checking without `Any`. Mockable interfaces for testing. No runtime cost (protocols are erased at runtime).
---
## DES-003: Named Databases
**Date:** 2026-02-10
**Status:** SETTLED
**Topic:** Work/personal separation
### Design
`resolve_db_paths(settings, name)` maps a database name to a path under `QUARRY_ROOT`. Each database is fully isolated — its own LanceDB directory, sync registry, and vector index. The `use` command (CLI) and `use` tool (MCP) switch the active database.
---
## DES-004: Embedding Model Selection
**Date:** 2026-02-08
**Status:** SETTLED
**Topic:** Which embedding model to use
### Design
snowflake-arctic-embed-m-v1.5: 768-dimensional, 512 token context, int8 quantized ONNX (~120 MB). Auto-downloads on first use.
### Why This Design
Strong retrieval quality for its size. Runs efficiently on CPU (no GPU required). Permissive license (Apache 2.0). The model is fixed — there is no configuration to swap it. Changing the model invalidates all existing embeddings, requiring full re-ingestion.
---
## DES-005: Bearer Token Authentication
**Date:** 2026-02-19
**Status:** SETTLED
**Topic:** HTTP API authentication
### Design
`quarry serve --api-key` gates all HTTP endpoints except `/health` behind `Authorization: Bearer <key>`.
- `/health` exempt — load balancers need unauthenticated health checks.
- OPTIONS exempt — CORS preflight requests carry no auth.
- `hmac.compare_digest` for token comparison — prevents timing attacks.
- Case-insensitive scheme per RFC 7235.
- Empty key = no auth — prevents accidental trivial bypass.
- No key = auth disabled — local use requires no key.
- Server refuses non-loopback binding without `--api-key`.
---
## DES-006: ASGI Server (Starlette + uvicorn)
**Date:** 2026-02-19
**Status:** SETTLED
**Topic:** HTTP server framework
**Supersedes:** Threaded Request Handling (stdlib `http.server`)
### Design
`quarry serve` runs a Starlette ASGI app on uvicorn. REST handlers are sync functions — Starlette runs them in its threadpool. The `/mcp` WebSocket endpoint is async. Default port 8420.
### Why This Design
MCP-over-WebSocket required native async support, which stdlib `http.server` cannot provide. Thread safety invariants are unchanged: `_QuarryContext` fields set once at startup, LanceDB handles concurrent reads, ONNX Runtime sessions are thread-safe.
---
## DES-007: MCP-over-WebSocket
**Date:** 2026-02-23
**Status:** SETTLED
**Topic:** Daemon architecture for shared embedding model
### Design
The `/mcp` WebSocket endpoint enables multiple Claude Code sessions to share a single quarry daemon via mcp-proxy. Each WebSocket connection gets its own asyncio Task with a `ContextVar` for database selection. `FastMCP._mcp_server` accessed via `getattr` with runtime guard; `mcp` pinned to `<2.0.0`.
Security: Origin-based CSWSH protection, Bearer auth before WebSocket accept, session keys sanitized (control chars stripped, truncated to 64 chars, CWE-117).
---
## DES-008: Log Safety (CWE-532, CWE-117)
**Date:** 2026-02-23
**Status:** SETTLED
**Topic:** Preventing sensitive data leakage in logs
**Supersedes:** Log Redaction (query string stripping in stdlib access logs)
### Design
Uvicorn's access log is disabled entirely (`access_log=False`), eliminating query string leakage at the source. The search handler logs only result count, never the raw query. WebSocket session keys sanitized before logging.
---
## DES-009: Single Table Design
**Date:** 2026-02-08
**Status:** SETTLED
**Topic:** LanceDB table structure
### Design
All chunks live in one LanceDB table (`chunks`). Document and collection boundaries are columns, not separate tables. Simplifies cross-document search and avoids table proliferation. Filtering uses LanceDB's built-in filter predicates on vector search.
---
## DES-010: Configurable CORS Origins
**Date:** 2026-02-23
**Status:** SETTLED
**Topic:** Cross-origin access control for the HTTP API
### Design
`--cors-origin` (repeatable) controls `Access-Control-Allow-Origin`. The server reflects the request's `Origin` header only when it matches the allow list, and adds `Vary: Origin`. Defaults to `http://localhost`.
---
## DES-011: Container Deployment (Fly.io)
**Date:** 2026-02-19
**Status:** SETTLED (relocated)
**Topic:** Production deployment
### Design
> Dockerfile, fly.toml, .dockerignore, and sync-chat-db.sh relocated to [punt-labs/public-website/infra/quarry/](https://github.com/punt-labs/public-website/tree/main/infra/quarry). The Dockerfile installs `punt-quarry` from PyPI.
Multi-stage build downloads the embedding model at build time for fast cold starts (~5s). LanceDB data on Fly persistent volume at `/data`. TLS terminated by Fly's proxy.
---
## DES-012: Documentation Consolidation into LaTeX
**Date:** 2026-03-25
**Status:** SETTLED
**Topic:** Reducing docs fragmentation
### Design
Created `docs/architecture.tex` — a single LaTeX architecture document absorbing content from `ADVANCED-CONFIG.md`, `SEARCH-TUNING.md`, `NON-FUNCTIONAL-DESIGN.md`, and the architecture sections of `DESIGN.md`. The LaTeX source is readable by agents; the compiled PDF is readable by humans.
### Why This Design
Docs were fragmented across 5+ markdown files with overlapping content. LaTeX provides consistent rendering, section numbering, and the ability to include tables, diagrams, and cross-references in one document. Follows the pattern established by lux's `docs/architecture.tex`.
### Alternatives Considered
**Keep markdown files** — simpler tooling but no cross-references, inconsistent formatting, content duplication across files. Rejected.
---
## DES-013: Agent Discovery — Three-Layer Strategy
**Date:** 2026-03-26
**Status:** SETTLED
**Topic:** How agents discover quarry's capabilities
### Design
Three complementary mechanisms:
1. **Researcher agent** (`plugin/agents/researcher.md`) — subagent that combines quarry local search with web research. Auto-ingests valuable web findings.
2. **CLAUDE.md injection** — `quarry install` appends a capabilities section to `~/.claude/CLAUDE.md` (idempotent, HTML comment sentinel).
3. **SessionStart context enrichment** — hook returns `additionalContext` with tool names, slash commands, and researcher agent mention.
### Why This Design
No single mechanism is reliable across all contexts. CLAUDE.md is always loaded but generic. SessionStart is session-specific but has known delivery issues. The researcher agent is discovered via the plugin system. Layering all three provides redundancy.
---
## DES-014: Thin Hook Standard Compliance
**Date:** 2026-03-26
**Status:** SETTLED
**Topic:** session-start.sh refactoring
### Design
Refactored `session-start.sh` from 88 lines of shell business logic to a 4-line thin gate that derives `PLUGIN_ROOT` from `$0` and delegates to `quarry-hook session-setup`. Command deployment and MCP permissions logic moved to `handle_session_setup()` in `_stdlib.py` (stdlib-only imports).
### Why This Design
Follows punt-kit/standards/hooks.md: hooks are plumbing, not product. Business logic belongs in testable Python, not shell. The 4-line gate (not 3) is necessary because `session-setup` needs the plugin root to find the `commands/` directory, and `CLAUDE_PLUGIN_ROOT` is only a hooks.json template variable, not an env var.
---
## DES-015: Filesystem Standard Adoption
**Date:** 2026-03-26
**Status:** SETTLED
**Topic:** User data directory layout
### Design
Adopted punt-kit/standards/filesystem.md. User data moves from `~/.quarry/` to `~/.punt-labs/quarry/`. Per-project config moves from `.claude/quarry.local.md` to `.punt-labs/quarry/config.md`. Logs move from `data/quarry.log` to `logs/quarry.log`.
### Why This Design
One namespace, one root. All Punt Labs tools under `~/.punt-labs/<tool>/`. Per-repo config owned by quarry, not by Claude Code's `.claude/` directory. No automatic migration per the standard — clean break with manual `mv` command documented.
---
## DES-016: Execution Provider Strategy for Embedding
**Date:** 2026-03-27
**Status:** SETTLED
**Topic:** Which ONNX execution provider and model precision to use per platform
### Design
Two deployment profiles:
| Profile | Model | Provider | Throughput | Use case |
|---------|-------|----------|-----------|----------|
| **Local (CPU)** | int8 (~120 MB) | CPUExecutionProvider | 9.4 texts/s (M2 Air), 134 texts/s (AMD) | Laptops, default install |
| **Server (GPU)** | FP16 (~218 MB) | CUDAExecutionProvider | 3,042 texts/s (RTX 5080) | Central quarry, bulk ingestion |
The model is selected based on available hardware. The embedding dimension (768) is identical across these precision variants of the same model, so embeddings remain comparable while the underlying architecture, checkpoint, and tokenizer are unchanged. If any model artifact changes (even with the same dimension), all existing vectors must be re-embedded.
### Why This Design
Benchmarked 6 configurations on two machines (M2 Air, AMD + RTX 5080):
| Config | M2 Air | AMD host (w/ RTX 5080) | Notes |
|--------|--------|------------------------|-------|
| int8 + CPU | 9.4 texts/s | 134 texts/s | Production default (CPU only, GPU unused) |
| int8 + CUDA | — | 158 texts/s | 168 memcpy nodes, barely faster than CPU |
| FP32 + CUDA | — | 1,639 texts/s | Full precision, no warnings |
| FP16 + CUDA | — | 3,042 texts/s | Half precision, fastest, 9ms/batch |
| FP32 + CPU | — | 79 texts/s | Slower than int8, larger model |
| FP32 + CoreML | 0.8 texts/s | — | 99 graph partitions, 8.9 GB RAM, 12x slower |
| FP16 + CoreML | — | — | Not tested (CoreML dead end) |
### Alternatives Considered
1. **CoreML EP on Apple Silicon** — Rejected. The Neural Engine cannot efficiently run this transformer architecture. 463 of 636 nodes supported, requiring 99 partitions. Result: 0.8 texts/s (12x slower than CPU), 8.9 GB RAM.
2. **int8 on CUDA** — Rejected for GPU deployment. The int8 quantized operators require 168 CPU↔GPU memory copies per inference, negating GPU acceleration. Only 18% faster than CPU despite having the GPU.
3. **FP32 on CUDA** — Valid but suboptimal. FP16 is 1.9x faster with identical embedding quality for retrieval.
4. **AWS SageMaker** — Previously rejected (DES-004 era). Network round-trip dominated; local CPU was faster.
---
## DES-017: Hybrid Search via BM25 + Vector + RRF
**Date:** 2026-03-28
**Status:** SETTLED
**Topic:** How to retrieve agent memories and documents
**Implemented by:** `retrieval/hybrid.py` (`HybridRetriever`) behind the DES-037
seam; scoring uses normalized embeddings + cosine (DES-038).
### Design
Dual-channel retrieval: vector similarity (existing ANN search) + BM25 full-text search (Tantivy via LanceDB native FTS). Results fused with Reciprocal Rank Fusion: `score[id] = Σ(1/(60 + rank))` across channels, weighted by temporal decay for agent-scoped memories.
Hybrid search is used for all `find` calls regardless of whether `agent_handle` is provided. FTS failures gracefully fall back to vector-only with a WARNING log.
### Why This Design
Vector search misses exact terms (proper nouns, code identifiers, jargon). BM25 catches keyword matches that embeddings miss. RRF fuses rankings without requiring score normalization across channels (~30 lines). Temporal decay (`exp(-decay_rate * hours)`) keeps recent working memories relevant without losing stable reference material. Decay applies only to chunks with `memory_type` in {fact, observation, opinion, procedure}. Documents and seeded expertise (empty `memory_type`) are exempt — even when tagged with `agent_handle` for ownership.
### Alternatives Considered
1. **Vector-only with reranking** — Rejected. Reranking improves ordering but cannot surface documents the vector search missed entirely.
2. **BM25-only** — Rejected. Loses semantic similarity for paraphrased queries.
3. **Weighted linear combination** — Rejected. Requires normalizing scores across channels (cosine similarity vs BM25 are on different scales). RRF avoids this by using rank positions only.
4. **Graph-based retrieval (Neo4j)** — Rejected. Overkill for local-only. Entity extraction + metadata filtering gives 80% of the benefit at 20% complexity.
---
## DES-018: Agent Memory Metadata Schema
**Date:** 2026-03-28
**Status:** SETTLED
**Topic:** How to scope memories to agents and classify memory types
### Design
Three new columns on the LanceDB chunks table:
| Column | Type | Default | Purpose |
|--------|------|---------|---------|
| `agent_handle` | utf8 | `""` | Which agent owns this memory (empty = unscoped) |
| `memory_type` | utf8 | `""` | fact, observation, opinion, procedure (empty = document) |
| `summary` | utf8 | `""` | One-line summary for lightweight search |
Migration via `table.add_columns()` with empty-string defaults. Idempotent `_migrate_schema()` runs on every table open. Memory type taxonomy from the Hindsight architecture (91.4% on LongMemEval).
### Why This Design
Agent memories need scoping (rmh's memories shouldn't pollute bwk's searches) and classification (a "how do I deploy?" query should prefer procedures over facts). Empty defaults preserve backwards compatibility — existing unscoped documents gain the columns with no behavior change. Per-agent collections (`memory-claude`, `bwk-books`) provide isolation; the columns enable filtering within shared collections.
### Alternatives Considered
1. **Separate tables per agent** — Rejected. Complicates cross-agent search and increases table management overhead.
2. **JSON metadata column** — Rejected. No SQL filtering, no type safety, harder to index.
3. **Full entity extraction at ingestion** — Deferred to v2. Requires LLM pass per chunk (~1s each). The current schema supports it when added later.
---
## DES-019: Ethos Extension Session Context Setup
**Date:** 2026-03-29
**Status:** SETTLED
**Topic:** How `quarry install` writes memory instructions into ethos identity extension files
### Design
`quarry install` step 7/7 scans `~/.punt-labs/ethos/identities/*.ext/quarry.yaml` and appends a `session_context` YAML literal block scalar to any file that has `memory_collection` but no `session_context`. The template is parameterized by agent handle and collection name.
Key design choices:
1. **Raw file append, not YAML round-trip.** The function reads the raw text to detect existing keys, then appends the `session_context: |` block directly. `yaml.safe_load` is used only to extract the `memory_collection` value — the file is never re-serialized through `yaml.dump`.
2. **Per-identity exception handling.** The scan loop wraps each identity in a broad `try/except Exception`. A malformed file for one identity does not abort processing of the others. Failed identities are reported in the result message.
3. **Three-way classification.** `EthosExtDiagnostics.write_session_context` returns `"updated"`, `"already_set"`, or `"no_collection"` — not a boolean. The `no_collection` case is surfaced in the install output so users know their config is incomplete.
### Why This Design
Ethos v2.4.1 removed `BuildMemorySection` (hardcoded quarry knowledge in Go code, a DES-008 violation) and replaced it with generic `BuildExtensionContext` (DES-022). Ethos now emits whatever is in the `session_context` key of any extension YAML verbatim at session start and compaction. Quarry owns the content of its own instructions — ethos just delivers them.
Without this install step, agents with existing `quarry.yaml` ext files (containing `memory_collection` but no `session_context`) silently lose their memory instructions after upgrading ethos. The install step closes this gap idempotently.
### Alternatives Considered
1. **YAML round-trip via `yaml.safe_load` + `yaml.dump`** — Rejected. Destroys comments, blank lines, and key ordering in the user's file. Silent data corruption on the happy path.
2. **Ethos writes quarry's instructions** — Rejected. Violates the one-way dependency: quarry depends on ethos for identity, but ethos has zero knowledge of quarry's internals (DES-008).
3. **Require users to manually add `session_context`** — Rejected. Silent failure with no error message. Users would not know their memory stopped working until they noticed missing recall.
---
## DES-020: TLS Everywhere with TOFU Certificate Pinning
**Date:** 2026-04-01
**Status:** SETTLED
**Topic:** How quarry secures remote connections
### Design
The installed quarry service always runs with TLS enabled — including on localhost. `quarry serve` without `--tls` is supported for local development but must not be used for production. The security model is TOFU (Trust On First Use) with self-signed CA certificate pinning.
**Certificate generation:** `quarry install` generates a self-signed EC P-256 CA and server certificate with full x509 extension set. Certs are written atomically to `~/.punt-labs/quarry/tls/` with 0600/0644 permissions. The CA cert CN is `"Quarry CA"` (not hostname-scoped). The server cert SAN includes the configured hostname, localhost, and loopback addresses.
**TOFU login flow:** `quarry login <host>` fetches the server's CA cert over HTTPS with verification disabled (bootstrap), displays the SHA256 fingerprint for out-of-band confirmation, then pins the CA cert locally at `~/.punt-labs/mcp-proxy/quarry-ca.crt`. All subsequent connections verify against the pinned CA only — system roots are excluded.
**mcp-proxy integration:** The `quarry.toml` profile includes a `ca_cert` field pointing to the pinned CA cert. mcp-proxy builds a custom TLS config with a cert pool containing only this CA, enforcing TLS 1.3 minimum.
### Why This Design
Quarry servers hold the user's entire document corpus. Plaintext connections expose both the content and the API key to any network observer. TLS on localhost adds zero user friction (handled by `quarry install`) and eliminates the "it's just localhost" exception that leads to split security models. TOFU is appropriate because there is no PKI — quarry servers are personal infrastructure, not public services.
### Alternatives Considered
1. **Let's Encrypt / ACME** — Rejected. Requires a public domain name and port 80/443 access. Personal home servers often have neither.
2. **System trust store** — Rejected. Adding a self-signed CA to the system trust store requires root and varies by OS. TOFU pinning works in userspace.
3. **Optional TLS (`--insecure` flag)** — Rejected. Two code paths, two security models. Users who start with `--insecure` never switch. One path, always encrypted.
---
## DES-021: Remote CLI Routing — No Split Horizon
**Date:** 2026-04-01
**Status:** SETTLED
**Topic:** How CLI commands route when a remote server is configured
### Design
When a remote quarry server is configured (via `quarry login`), **every data command routes to the remote server**. There is no split horizon where some commands go remote and others silently fall back to the local database. The only local-only commands are authentication and administration:
**Local-only:** `login`, `logout`, `remote list`, `install`, `uninstall`, `serve`, `mcp`, `version`
**Everything else routes remotely:** `find`, `status`, `list` (all subcommands), `show`, `ingest`, `remember`, `delete`, `register`, `deregister`, `sync`, `use`, `doctor`
The CLI detects remote configuration via `_safe_proxy_config()` and routes through `_remote_https_get()`, `_remote_https_post()`, and `_remote_https_delete()` helpers with the pinned CA cert and Bearer token.
**Current state (v1.16.0):** All data commands route remotely: `find` (/search), `show` (/show), `status` (/status), `ingest` (/ingest POST), `remember` (/remember POST), `delete document` (DELETE /documents), `delete collection` (DELETE /collections), `sync` (POST /sync), `list documents` (GET /documents), `list collections` (GET /collections), `list registrations` (GET /registrations), `list databases` (GET /databases), `register` (POST /registrations), `deregister` (DELETE /registrations). Register and deregister use fire-and-forget (return task_id, exit 0) since v1.15.0.
**Remaining endpoint surface:**
| Endpoint | Method | Purpose |
|----------|--------|---------|
| `/use` | POST | Switch active database (needs endpoint) |
### Why This Design
When a user is connected to a remote server, they expect every command to operate on the remote data. A user who runs `quarry list collections` after `quarry login` expects to see the remote server's collections, not whatever happens to be on their local machine. Silent local fallback gives wrong data with no warning — the worst kind of bug.
The "local-only" classification applies only to commands that genuinely operate on local configuration (authentication, service management). There is no data operation that makes sense locally when the user has chosen to work with a remote server.
### Alternatives Considered
1. **Split horizon with explicit `--local`/`--remote` flags** — Rejected. Adds complexity to every command. Users shouldn't have to think about where their data lives after logging in.
2. **Only route read operations remotely** — Rejected. A user who can search remotely but can't ingest remotely has a broken workflow. Remote means remote for everything.
---
## DES-022: Single Install Script with --network Flag
**Date:** 2026-04-11
**Status:** SETTLED
**Topic:** How the installer handles local-only vs network-accessible daemon modes
### Design
One `install.sh` with a single optional flag: `--network`. Default (no flags) installs everything with the daemon on localhost. `--network` binds the daemon to `0.0.0.0` and requires `QUARRY_API_KEY`. Plugin installation is opportunistic — if the `claude` CLI exists on PATH, the plugin and marketplace are installed; if not, they're skipped with a warning.
All install steps always run in both modes: CLI, model download, GPU swap, daemon registration, TLS certificates, local `quarry login localhost`. The only difference is the bind address.
### Why This Design
The prior four-script split (`install.sh`, `install-server.sh`, `install-client.sh`, `install-both.sh`) caused the same drift bug three times in one session: VERSION constants not bumped across all scripts (quarry-tu0w), GPU swap block deleted from split scripts (quarry-e4c2), and step numbering divergence. Every release that didn't update all four scripts created a broken install path for some users.
The original `--server`/`--client` split was based on a false premise — that "server" and "client" were different install types. In practice every machine needs the full install (CLI, model, daemon, GPU swap). The only real axis is whether the daemon should be reachable from the network.
### Alternatives Considered
1. **Four separate scripts** — Rejected. Caused three drift bugs in one day. Shared code across four files with no enforcement mechanism.
2. **Three modes: default, --server, --client** — Implemented first (PR #220), then simplified. `--client` skipped model download and daemon, but clients that happen to have GPUs still want local quarry operations. The mode distinction added complexity without matching real use cases.
3. **Shared sourced fragment** (`.bin/install-gpu-swap.sh`) — Rejected. The `curl | sh` flow has no `$script_dir` to resolve sourced files. Would require a two-file download or inline the fragment via heredoc.
4. **Environment variable instead of flag** (`QUARRY_INSTALL_MODE=network`) — Rejected. `sh -s -- --network` is POSIX and more discoverable than env vars in documentation.
---
## DES-023: FTS Index Rebuild After Optimize
**Date:** 2026-04-11
**Status:** SETTLED
**Topic:** Keeping the Tantivy full-text index consistent after LanceDB compaction
### Design
`optimize_table()` now calls `table.create_fts_index("text", replace=True)` after `table.optimize()`. This rebuilds the Tantivy FTS index against the new fragment layout. The call also passes `cleanup_older_than=timedelta(days=7)` to prune old manifest versions during compaction.
### Why This Design
`table.optimize()` compacts data fragments — merging small fragments and removing deleted rows. The Tantivy FTS index stores row references by fragment ID. After compaction, those fragment IDs no longer exist. Every subsequent FTS query hit `RuntimeError: lance error: ... fragment id N but this fragment does not exist` and fell back to vector-only search via the existing exception handler in the FTS search path (now `db/chunk_search.py`, after `database.py` was decomposed into the `db/` package).
This meant the BM25 leg of hybrid search (DES-017) was dead after every `sync_all` cycle since the feature shipped. The RRF fusion that hybrid search was designed to provide — catching keyword matches that vector search misses — never worked in production. The fallback masked the failure: search returned results, but only from the vector channel.
The rebuild is O(n) in table size but only runs after bulk sync operations (via `sync_all` → `optimize_table`), not on every query. On a 33GB dataset with 59K fragments, the rebuild adds seconds, not minutes.
The 7-day version pruning addresses a secondary issue: 118K manifest files consuming 11GB in `_versions/`. LanceDB's `optimize()` merges fragments but doesn't prune old manifests by default.
### Alternatives Considered
1. **Rebuild FTS on every query** — Rejected. O(n) per query is unacceptable.
2. **Rebuild FTS on daemon startup** — Rejected. Doesn't help when the daemon runs for days between restarts. The stale index reappears after the first sync.
3. **Use `replace=False` and rely on LanceDB incremental FTS updates** — This is what was in place. LanceDB does not incrementally update the FTS index after `optimize()` changes fragment IDs. The index must be fully rebuilt.
4. **Catch the RuntimeError and retry with a fresh table handle** — Rejected. Treats the symptom. The FTS index is structurally stale after compaction; retrying with the same stale index doesn't help.
---
## DES-024: File-Based Claude Code Plugin Check in Doctor
**Date:** 2026-04-11
**Status:** SETTLED
**Topic:** How `quarry doctor` verifies the Claude Code MCP plugin is configured
### Design
`_check_claude_code_mcp()` reads `~/.claude/plugins/installed_plugins.json` directly and checks for the `quarry@punt-labs` key. It validates the install path exists and the plugin manifest contains an `mcpServers.quarry` entry. No subprocess calls. Exception handler catches `JSONDecodeError`, `OSError`, `KeyError`, `TypeError`, and `AttributeError` for graceful degradation on corrupted or changed registry formats.
### Why This Design
The prior implementation shelled out to `claude mcp list` with a 10-second timeout. That command spawns every configured MCP server for health checks — sequentially. With 10 plugins installed (quarry, biff, lux, vox, beadle, ethos, dungeon, z-spec, plus others), the total exceeded 15 seconds. The doctor check timed out on every run, on both Linux and macOS.
The check only needs to know whether quarry is configured, not whether every MCP server is healthy. Reading the JSON registry file answers that question in <1ms with zero side effects.
The Claude Desktop check (`_check_claude_desktop_mcp`) already used this file-based pattern. The Claude Code check now matches.
### Alternatives Considered
1. **Raise the timeout to 30s or 60s** — Rejected. Moves the goalpost. Adding more plugins would hit the new timeout. Users shouldn't wait 30 seconds for a doctor check.
2. **Background the probe and report results asynchronously** — Rejected. Doctor checks are synchronous by design — the output is a pass/fail table. An async probe that reports "pending" defeats the purpose.
3. **Skip the check when running under a Claude Code parent process** — Rejected. Detection is fragile (check `CLAUDE_SESSION_ID` env var?) and doesn't help when running `quarry doctor` from a plain terminal.
4. **Read the MCP config that `claude mcp add` writes to** — Considered as an alternative data source. `_configure_claude_code()` writes via `claude mcp add`, which uses a different store than the plugin registry. In practice both stores have the quarry entry because quarry is always installed as a plugin. Noted as a known limitation in a code comment.
---
## DES-025: Agent Tool Access — disallowedTools vs tools
**Date:** 2026-04-12
**Status:** SETTLED
**Topic:** How to grant sub-agents MCP tool access in agent definitions
### Design
Agent definitions in `.claude/agents/*.md` use `disallowedTools` (denylist) instead of `tools` (allowlist) when the agent needs access to MCP tools. Claude Code sub-agents inherit all tools from the main session by default, including MCP tools. The `tools` field is an allowlist — specifying it restricts the agent to exactly those tools and nothing else. Since MCP tool names are dynamic (e.g., `mcp__plugin_quarry_quarry__find`) and vary across projects and sessions, an allowlist cannot enumerate them portably.
**Prior pattern (all existing agents):**
```yaml
tools:
- Read
- Write
- Edit
- Bash
- Grep
- Glob
```
This gives the agent 6 tools. All MCP tools are excluded. The agent cannot call `mcp__plugin_quarry_quarry__find`, `mcp__plugin_biff_tty__write`, or any other MCP tool.
**New pattern (agents that need MCP access):**
```yaml
disallowedTools:
- Write
- Edit
```
This gives the agent everything the main session has — all internal tools plus all MCP tools — except Write and Edit. The agent can call quarry MCP tools, biff MCP tools, etc.
### Why This Design
The QA agent (`qae`) needs to run smoke tests that exercise quarry's MCP tools (`find`, `remember`, `ingest`, `delete`, `status`, `list`, `show`, `use`). With the `tools` allowlist, these are excluded. With `disallowedTools`, they're inherited. The QA agent is restricted from Write and Edit because smoke tests should be read-only — test data is created via CLI (`quarry remember`) and MCP tools, not by writing files.
This pattern applies to any agent that needs to interact with MCP servers: QA agents testing MCP tools, agents that need to search quarry, agents that need to send biff messages, agents that need to display via lux.
### When to Use Which
| Pattern | When |
|---------|------|
| `tools: [Read, Write, Edit, Bash, Grep, Glob]` | Agent only needs file operations and shell. Most implementation agents (rmh, bwk, adb, etc.). |
| `disallowedTools: [Write, Edit]` | Agent needs MCP tools but shouldn't modify files. QA agents, research agents. |
| `disallowedTools: []` (or omit both fields) | Agent needs full access to everything. Use sparingly. |
| `tools: [Read, Grep, Glob, Bash]` + specific MCP tools | Agent needs a specific subset. Fragile — MCP tool names change across projects. |
### Alternatives Considered
1. **Add MCP tool names to the allowlist** — Rejected. MCP tool names include plugin and server prefixes (`mcp__plugin_quarry_quarry__find`) that are project-specific. An allowlist that works in quarry won't work in biff or lux. Not portable.
2. **Use wildcard patterns in tools** — Not supported. The `tools` field takes exact tool names only.
3. **Give all agents full access** — Rejected. Implementation agents should not have MCP tool access by default. A sub-agent that accidentally calls `mcp__plugin_quarry_quarry__delete` during a code fix is a data loss risk. Least privilege applies.
## DES-026: Sync Concurrency Control and Batch Writes
**Date:** 2026-04-18
**Status:** SETTLED
**Topic:** Preventing LanceDB compaction death spiral from concurrent sync
### Problem
The quarry serve process accumulated 133K LanceDB fragments (83 GB)
and burned 13 CPU cores sustained for 5 days. Root cause: every
SessionStart hook spawns `quarry sync`, which routes through the
HTTP `/sync` endpoint to run `sync_all()` inside the serve process.
No concurrency guard existed, so multiple syncs ran simultaneously.
Each sync creates 2 LanceDB transactions per document (delete + add),
each creating a new fragment. Compaction (`optimize_table()`) could
not keep up, creating a self-reinforcing death spiral.
### Design
Five changes:
1. **Server-side sync lock** — `SyncTaskState` on `_QuarryContext`
tracks whether a sync is running. `POST /sync` returns 409 when
one is already in progress. Simpler than `asyncio.Lock` because
the state check is synchronous in the async handler.
2. **Registration subsumption** — `register_directory()` enforces
that parent directories subsume children (deregisters them) and
child directories are rejected when a parent is already registered.
Prevents duplicate scanning.
3. **Batch LanceDB writes** — **SUPERSEDED by DES-034.** `prepare_document()`
chunked and embedded a whole document without writing to LanceDB;
`sync_collection()` accumulated every document's chunks and did a single
`batch_insert_chunks()` at the end. This reduced write transactions to 1
per collection sync but made nothing searchable until the sync finished, lost
all embedding work on a crash, and scaled peak memory with data size (a single
large file materialized all its vectors before any write). DES-034 replaces it
with bounded progressive commit: a streaming embed producer plus a
`ProgressiveIndexer` that flushes every `sync_flush_mb` and commits one
registry transaction per flush. `prepare_document()` and `batch_insert` are
removed.
4. **optimize_table() guard** — Skips when fragment count exceeds
10,000 to prevent the death spiral. `quarry optimize --force`
CLI command for manual recovery.
5. **Async sync endpoint** — `POST /sync` returns 202 with task_id.
`sync_all()` runs as a background `asyncio.Task`. `GET /sync/{task_id}`
returns status. CLI becomes fire-and-forget.
### Alternatives Considered
1. **Queue concurrent sync requests** — Rejected. Queuing hides the
problem. The caller should know sync is already running.
2. **Batch deletes via IN clause** — Rejected. LanceDB's delete
predicate may not handle large IN clauses efficiently.
3. **Partial compaction** — LanceDB 0.30 does not support it.
4. **WebSocket streaming of sync progress** — Over-engineered for a
background batch operation. Polling is sufficient.
### Amendment 2026-07-01 — Deregister is not async (quarry-noiw, quarry-xsz3)
Change #5 (async endpoint, 202 + task_id) targets `/sync`: long, multi-file
embedding that would block the event loop. `DELETE /registrations` does not
share that profile — its existence check and registry-row mutation are trivial
metadata operations that must complete before the response so the caller gets
correct feedback. Deregister is therefore split:
- **Synchronous (before the HTTP response):** validate the collection exists
and delete its registry rows, executed via `run_in_threadpool` so the event
loop is never blocked. An unknown collection returns **404**
(`No registration found for '<collection>'`), which the CLI maps to exit 1 —
parity with the local path (`__main__.py`). A registry failure returns 500.
- **Asynchronous (202 + task_id):** only the chunk purge (`delete_document`
over the removed documents), which has real latency. `GET /tasks/<id>`
reports its terminal status; the CLI polls after the 202 and maps a `failed`
status to a non-zero exit with the server's error text. The CLI never reports
success for an operation that then dies.
`SyncRegistry` now sets `PRAGMA busy_timeout=5000` on every connection, so a
concurrent sync writer causes a bounded wait instead of an instant
`database is locked` (the quarry-xsz3 mechanism). The async model of change #5
is otherwise retained for `/sync`.
The **MCP** surface (`deregister_directory`) is now synchronous end-to-end as
well — existence check, registry delete, and chunk purge run before the tool
returns. MCP has no poll channel to expose to agents, so full synchronous
execution is the correct shape there; an unknown collection returns
`No registration found for '<collection>'` and a registry or purge failure
surfaces as an error rather than a false success. All three surfaces (CLI,
HTTP, MCP) report the same fields — `collection`, `removed`, `deleted_chunks`.
Adding the CLI deregister poll grew `__main__.py` past its OO `module_size`
baseline, so the remote HTTP client machinery moved out of the CLI entrypoint
into a new module, `remote_client.py`. `RemoteClient` (a frozen dataclass bound
to one `[quarry]` config) owns the transport — `request`, `get`, `find`, and
the `await_task` poller — and `RemoteError` carries the HTTP status. This is a
behavior-preserving extraction: the CLI constructs `RemoteClient(config)` at
each remote branch and the local/remote routing (`_safe_proxy_config`) stays in
`__main__.py` where it decides which path a command takes. `remote_client.py`
is CLI-tier (it may raise `typer.Exit` and print to stderr for poll
failures/timeouts); it deliberately does **not** live in `remote.py`, which owns
a different concern — proxy config and TLS (`read/write_proxy_config`,
`fetch_ca_cert`, `validate_connection`).
## DES-027: jemalloc Memory Tuning for LanceDB Arrow Buffers
**Date:** 2026-04-18
**Status:** SETTLED
**Topic:** Daemon RSS growth from jemalloc arena retention
### Problem
The quarry daemon's RSS grew monotonically — +178 MB per sync cycle,
reaching 5.4 GB after one day of operation. `gc.collect()` did not
reclaim the memory. `tracemalloc` (Python-only) showed low Python
allocation growth. The anonymous heap (4.35 GB) was not file-backed
(not LanceDB mmap) and not attributable to the ONNX model (~110 MB).
### Root Cause
LanceDB's Rust core links jemalloc as its global allocator.
`batch_insert_chunks` calls `table.add(records)`, which creates Arrow
RecordBatches in Rust. After the write completes and the RecordBatch
is dropped, jemalloc retains the freed memory arenas for potential
reuse rather than returning them to the OS. With the default decay
settings (effectively infinite on Linux), freed pages accumulate
indefinitely.
### Evidence
Profiling the running daemon (PID 2370838, 5.4 GB RSS):
| Metric | Value |
|--------|-------|
| VmRSS | 5,548 MB |
| Pss_Anon | 4,349 MB (not mmap) |
| ONNX model | ~110 MB |
| Database on disk | 2.7 GB (62K chunks) |
| Unexplained heap | ~4.2 GB |
Empirical testing across 4 MALLOC_CONF variants (62K chunks, 3K docs):
| Config | Post sync 1 | Post sync 2 | Growth/sync |
|--------|-------------|-------------|-------------|
| No MALLOC_CONF | 5,400 MB (1 day) | cumulative | +178 MB/cycle |
| decay:1000 | 2,543 MB | 2,543 MB | stable |
| + narenas:1 | 1,760 MB | 2,292 MB | +532 MB |
| + tcache:false | 1,144 MB | 1,323 MB | +179 MB |
| + muzzy:0 | 1,147 MB | 1,356 MB | +209 MB |
### Design
Set `MALLOC_CONF=dirty_decay_ms:1000,muzzy_decay_ms:0,narenas:1,tcache:false`
in the daemon's environment:
- **Linux**: Written to `~/.punt-labs/quarry/quarry.env` by
`_write_env_file()` in `service.py`. Systemd reads it via
`EnvironmentFile=`.
- **macOS**: Set in the launchd plist `EnvironmentVariables` dict.
The four settings:
- `dirty_decay_ms:1000` — return dirty pages (freed, not yet
returned to OS) within 1 second. Amortizes madvise syscall cost.
- `muzzy_decay_ms:0` — return muzzy pages (lazily purged) immediately.
These are already MADV_FREE'd; the zero cost makes immediate
return optimal.
- `narenas:1` — single allocation arena. Quarry's daemon does bulk
writes in batches, not high-concurrency per-request allocation.
Fewer arenas = less fragmentation = less retained memory.
- `tcache:false` — disable thread-local caching. Each thread's tcache
holds freed objects for fast reallocation, but with batch workloads
these caches hoard memory. No sync performance regression observed
(4.92s vs 6.62s baseline).
### Rejected Alternatives
1. **`dirty_decay_ms:0`** — Immediate return for dirty pages too.
Rejected: causes excessive `madvise` syscalls during bulk Arrow
writes. 1-second decay amortizes the cost.
2. **Replace jemalloc with glibc malloc** — Not feasible. LanceDB's
Rust binary links jemalloc at compile time. We don't control the
LanceDB build.
3. **Use Arrow-native API instead of Python dicts** — Would reduce
intermediate allocation pressure but doesn't address jemalloc
retention of Arrow's own buffers. A complementary optimization,
not a replacement.
4. **Periodic daemon restart** — Operational workaround, not a fix.
5. **decay:1000 only (without narenas/tcache)** — Tested: reduces to
2.5 GB but still retains ~1.4 GB of fragmented arena memory.
narenas:1 + tcache:false eliminate the fragmentation.
The daemon should be able to run indefinitely.
## DES-028: CLI --verbose Maps to INFO, Not DEBUG
**Date:** 2026-04-18
**Status:** SETTLED
**Topic:** --verbose stderr level deviates from org CLI standard
### Problem
The org CLI standard (`punt-kit/standards/cli.md` §Global Flags) says
`--verbose` enables "debug logging." The logging standard
(`punt-kit/standards/logging.md` §Levels) specifies
`configure_logging(stderr_level="DEBUG")` for `--verbose`.
Quarry maps `--verbose` to INFO instead of DEBUG.
### Rationale
40 `logger.debug()` calls across 12 modules, many in hot loops:
database.py (11 per-query RRF stats), hooks.py (12 hook lifecycle),
embeddings.py (2 per-batch timing), text_extractor.py (2 per-page).
For a 100-file sync with 500 pages, DEBUG produces 1000+ lines
dominated by per-query fusion stats and per-page character counts.
INFO (80 calls across 16 modules) contains the operational messages
users want: sync plans, embedding throughput, batch-insert timing.
### Design
- `--verbose` sets `stderr_level="INFO"` in `configure_logging`
- `QUARRY_LOG_LEVEL=DEBUG` env var provides the developer escape hatch
- Third-party loggers (lancedb, onnxruntime, httpx) pinned at WARNING
in the dictConfig `loggers` block to prevent noise at any level
### Rejected Alternative
Map `--verbose` to DEBUG per org standard. Rejected: 1000+ lines of
per-query RRF fusion stats and per-page char counts overwhelm the
terminal. The org standard's "debug logging" description fits tools
with fewer, coarser DEBUG calls — quarry's DEBUG is developer-trace
granularity.
---
## DES-029: quarry enable — Unified Project Activation
**Date:** 2026-05-11
**Status:** SETTLED
**Topic:** How projects activate all three knowledge capture types
### Design
`quarry enable <directory>` sets up three scoped collections in one command:
1. **File sync** — registers the directory for incremental sync (existing mechanism). Collection name derived from directory basename or `--collection` override.
2. **Passive captures** — web fetches and session transcripts route to `<name>-captures` (separate from the file-sync collection). Falls back to global `web-captures` / `session-notes` when no registration covers the cwd.
3. **Agent memory** — bootstraps ethos identity extensions (`quarry.yaml` with `memory_collection: memory-<handle>`) for all agent identities in `~/.punt-labs/ethos/identities/`.
`quarry disable <directory>` reverses: deregisters, optionally deletes indexed data (`--keep-data` to preserve), removes `.punt-labs/quarry/config.md`.
Session-start hook uses walk-up matching (child directories inherit the parent's collection) with a descendant guard (won't auto-register a parent that would subsume existing child registrations).
### Why This Design
Before `quarry enable`, the three capture types were configured through disconnected mechanisms: SessionStart auto-registration (crashed on child directories), hardcoded fallback collections (mixed content across projects), and manual ethos extension setup (nobody did it). One command replaces all three.
The captures separation (`<name>-captures` vs `<name>`) prevents web research and session transcripts from polluting the file-sync collection. Users searching their codebase get code, not browser tabs.
### Alternatives Considered
1. **Separate commands for each capture type** — Rejected. Three commands to set up what is conceptually one thing. Users forget the ethos step.
2. **Auto-enable on first session** — Rejected. CEO constraint: auto-registration stays for basic functionality, but `quarry enable` is for explicit configuration with captures separation and ethos bootstrap.
3. **Per-project config file only (no CLI)** — Rejected. The config file is written by `quarry enable`, not hand-authored. The command is the interface.
## DES-030: Session Transcript Lifecycle and Capture Strategy
**Status**: SETTLED (May 2026)
### Context
Claude Code stores session transcripts as `<session-uuid>.jsonl` files under `~/.claude/projects/<encoded-project-dir>/`. These are the primary source material for quarry's knowledge capture — each transcript contains the full conversation: research, debugging, design decisions, tool use.
### The Cleanup Problem
Claude Code deletes transcript `.jsonl` files older than `cleanupPeriodDays` (default: **30 days**) at startup. This is non-configurable per-project — it's a global setting in `~/.claude/settings.json`. The cleanup also covers `tool-results/`, `plans/`, `debug/`, and other session artifacts.
Known issues (as of May 2026):
- Auto-updates have silently deleted `.jsonl` files ([#41591](https://github.com/anthropics/claude-code/issues/41591))
- `--setting-sources local` ignores the setting and uses the 30-day default ([#45903](https://github.com/anthropics/claude-code/issues/45903))
- No warning or notification before deletion ([#46175](https://github.com/anthropics/claude-code/issues/46175))
### Decision
**PreCompact hook is the primary capture mechanism.** The hook fires before context compaction, while the transcript still exists in memory. It extracts conversation text and ingests it into quarry before Claude Code can clean it up. This is reliable regardless of `cleanupPeriodDays`.
**`quarry backfill-sessions` is a secondary, bounded recovery tool.** It ingests `.jsonl` files that still exist on disk — typically only sessions from the last 30 days. It cannot recover transcripts that have already been cleaned up. Its value is: (a) capturing the recent window when quarry is first installed, and (b) re-ingesting after a quarry database reset.
**The two mechanisms are complementary, not redundant.** PreCompact captures going forward; backfill captures the recent past. Neither can capture sessions older than `cleanupPeriodDays` unless the user has extended that setting.
### Filesystem Structure
```text
~/.claude/projects/<encoded-project-dir>/
<session-uuid>.jsonl # Main transcript (cleaned up after 30 days)
<session-uuid>/
subagents/ # Subagent transcripts
agent-<id>.jsonl
tool-results/ # Tool output artifacts
```
The encoded project dir replaces `/` with `-` and preserves the leading dash (e.g., `/Users/jm/code` → `-Users-jm-code`).
### Implications
1. Users who want full history should set `cleanupPeriodDays` to a high value (e.g., 365) in `~/.claude/settings.json` before sessions are lost.
2. `quarry enable` should advise users about the cleanup window.
3. Subagent transcripts (`subagents/agent-<id>.jsonl`) are not currently ingested by either mechanism — they are a future opportunity but not the primary knowledge source.
---
## DES-031: Daemon-First Architecture — One Engine, Formal Wire Protocol, Pure Clients
**Date:** 2026-05-16 (v1 PROPOSED); revised and **ACCEPTED** 2026-07-12 (v2.1); amended 2026-07-14 (v2.2)
**Status:** ACCEPTED
**Topic:** Separation of the compute/storage engine from access interfaces (CLI, MCP, Library) behind a formal wire protocol
**Supersedes:** the v1 "Engine-First Architecture with Thin Interfaces" framing (this entry replaces it). Full design + the 5-agent audit that grounds it: `docs/des-031v2-daemon-first.md`. The v2.2 amendment (below) is backed by `docs/des-client-architecture.md`. Epic `quarry-ma6f` / `quarry-ynvs`.
### Revision v2.2 (2026-07-14) — MCP is a client concern; `quarryd` binary; loopback `serve.token`
The operator ratified a reversal of two v2.1 decisions plus two additions.
Backing design: `docs/des-client-architecture.md`. The v1/v2.1 body below
otherwise stands; where they conflict, v2.2 governs.
- **Reverses §3.5 + §4.2 (MCP served BY the daemon).** MCP is a **client**
concern, modeled on vox (`vox mcp` → `server.py` → `VoxClientSync`). The daemon
serves the **REST API only** — the in-daemon `/mcp` route is deleted (landed in
PR-1, #356). `quarry mcp` becomes a FastMCP **stdio subcommand** whose tools
reach the daemon through `QuarryClient`; the Claude Code plugin runs
`quarry mcp` directly. The "MCP via the daemon over `/v1/mcp`" wording in the
§3.3 table and §3.5/§3.6 no longer applies.
- **DES-021 routing change (the tool is unaffected).** Quarry's plugin no longer
routes its MCP path through **mcp-proxy** (`plugin.json` runs `quarry mcp`
directly; the auth/reconnect the proxy handled for quarry moves into
`QuarryClient`). mcp-proxy itself remains a live, supported tool used by other
consumers — this is a change to quarry's own routing, not a supersession or
retirement of the tool.
- **Adds `quarryd`** — a dedicated daemon entry point (like `voxd`/`luxd`) that is
the sole engine process. This makes I1 a hard **process** boundary and removes
the v2.1 §3.1 lazy-import carve-out for `serve`/`mcp`: no `quarry` /
`quarry mcp` / `quarry-hook` process imports the engine; only `quarryd` (via
`quarry/daemon/`) does.
- **Adds loopback `serve.token`.** The daemon writes a mode-0600 token beside
`serve.port`; a `ClientConfig` resolves port+token (env → remote-login config →
loopback files), so loopback requests are authenticated — closing the
multi-user-host exposure §7 only aspired to. Vox-parity with `voxd`'s
`serve.port`/`serve.token`.
- **Confirms** the §3.4 seam (`QuarryClient` + injectable REST transport; no
vox-style gateway Protocol and no biff-style commands layer) and §3.1 hook
routing (hooks call `QuarryClient`, not `Database` — faster than the cold
in-process load and I1-correct).
- **Defers the richer error envelope.** The wire error stays `{"error": ...}` in
v2-2 (a faithful contract *extraction*, not a wire redesign — `ErrorBody` is a
superset via `extra="allow"`, preserving bug-class-3 parity). The
`{code, message, detail}` envelope from §3.3 is a **breaking** wire change,
deferred to land WITH `QuarryClient`'s typed `QuarryError` hierarchy in v2-4 so
the codes and the client that consumes them move together (`RemoteClient` is
already gone by then).
Sequencing under v2.2 (epic `quarry-ynvs`): PR-1 (#356) removed daemon MCP and
added pyright to CI; PR-2 introduces `quarry/api` + FastAPI + `/v1` (+ the missing
`/v1/optimize`, `/v1/backfill-sessions`); later PRs add `quarryd` / supervision /
`serve.token`, `QuarryClient` + the CLI thin-client (deleting `RemoteClient`), and
`quarry mcp`-as-client.
**PR-6 (`quarry-7ftj`) locked the boundary — the epic (`quarry-ma6f`/`quarry-ynvs`)
is complete.** An **import-linter** contract (`.importlinter`, run by
`make check-imports` in the `make check` chain and in CI) forbids every client
module (`quarry.__main__`, `quarry.hooks`, `quarry.mcp_server`, `quarry.client`,
`quarry.api`) from importing any engine package (`quarry.db`, `quarry.embeddings`,
`quarry.ingestion`, `quarry.retrieval`, `quarry.sync`, `quarry.daemon`) along any
*transitive* chain — a violating import fails CI, not review. A **full-client-surface
runtime sabotage test** (`tests/test_init.py`, each client module imported in a
subprocess with `lancedb`/`onnxruntime`/`pyarrow` poisoned) is the companion guard
that no engine reaches a client module's *module* scope. A hermetic **in-process
ASGI fixture** (`httpx.ASGITransport` over `build_app`) lets daemon-dependent tests
hit the real handlers without a socket, and one **real-loopback-TLS smoke** (slow
tier, out of the fast CI suite) covers the pinned-CA wire contract end-to-end;
**install health-gates** on `/health` `ready` before exit 0. Net: one engine in
`quarryd`, four thin client surfaces (CLI, hooks, library, MCP), the boundary
enforced structurally so it cannot regress.
**Correction to §3.1 enforcement-#2 of `docs/des-031v2-daemon-first.md`.** That
text assumed import-linter *cannot see* function-body lazy imports, so the
host-local engine-owning commands would be invisible to the static graph. With the
current toolchain (grimp 3.15) that is **false** — grimp resolves lazy imports.
The boundary therefore does **not** rest on lazy-import invisibility: the engine's
only entry point (`quarry.daemon.launcher`, the `quarryd` script) lives *inside*
`quarry.daemon` (engine-side, never a contract source), and the one sanctioned
exception — the host-admin diagnostics (`doctor`/`install`/`uninstall`) that probe
the local engine environment to report *why* a daemon is unhealthy — is an
**explicit, enumerated `ignore_imports` list** of the host-admin diagnostic lazy
edges (`doctor`/`doctor_captures`), not an invisibility trick. A new engine import from any other client-reachable module
still fails the contract; the sabotage test proves those diagnostics stay off the
module-scope hot path. (`quarry mcp` is no longer in the exception at all — PR-4
made it a pure client.)
#### PR-3a as landed (`quarryd` + `serve.token` + `ClientConfig`)
PR-3a implements the daemon/supervision/loopback-auth half of v2.2. What shipped:
- **`quarryd` entry point** (`quarry.daemon.launcher:entrypoint`) — the sole
engine process. A `DaemonLauncher` refuses a remote-reachable bind that has no
operator key, mints a 256-bit loopback token when none is given, and hands a
`ServeConfig` to `DaemonServer`. The `quarry serve` subcommand is **deleted**
(PL-PP-1, no shim); the supervised unit execs `quarryd` (launchd `KeepAlive` /
systemd `Restart=always`). Because only `quarryd` (via `quarry/daemon/`)
imports the engine, the v2.1 §3.1 lazy-import carve-out for `serve`/`mcp`
is gone: I1 is now a hard **process** boundary, not a within-process rule.
- **`serve.token` (mode-0600).** `DaemonServer` writes the token beside
`serve.port` (shared `RunDir`) **after a successful bind** and removes only
what this instance wrote (guarded by a bound flag). Writing post-bind is
what keeps a second `quarryd` that fails to bind (port in use) from
clobbering a running peer's live token on the shared per-db path — a failed
bind never writes or removes it. The token lands microseconds after the
serve loop starts accepting; a client that races that window fails closed
and retries, an acceptable trade for not nuking a live daemon. The write is
atomic (`os.open(0o600)` + tmp-rename) so no world-readable or partial-file
window exists. The token is written whenever
the daemon has an effective key — **including** when the operator sets
`--api-key` (the file holds that key), so a local client on a `--network`
server can authenticate on loopback without re-typing it. This closes the
multi-user-host exposure: before it, any local UID could reach the
unauthenticated daemon on `127.0.0.1`.
- **`LoopbackPolicy`** classifies a host with `ipaddress`, and splits into two
gates so one predicate never governs both a bind and a secret. **Bind gate**
(`is_loopback`, daemon/installer): name-tolerant —
`localhost`/`::1`/`127.0.0.0/8`/`::ffff:127.x` are loopback; `0.0.0.0`/`::` and
any unresolved name are remote (fail closed — require a key). **Token-
presentation gate** (`is_literal_loopback`, client): a LITERAL loopback IP
only — a NAME is never a presentation target (see `ClientConfig`).
- **`ClientConfig`** (`quarry/client`) resolves a login config into (URL, pinned
CA, bearer). The live `serve.token` is presented **only to a LITERAL loopback
IP** (`is_literal_loopback`), never to a name: a name like `localhost` is
resolver-controlled and on a dual-stack host can resolve to a co-tenant's
`::1`, so presenting the secret to it would leak the token in transit. The
managed path canonicalizes `quarry login localhost` → `127.0.0.1` at write
time (a policy mapping to the IPv4 literal the daemon binds, not an OS-resolver
lookup), so the normal case pins the un-hijackable `127.0.0.1:8420`. For a
remote target the stored bearer is kept. It fails closed (`OSError` → typed
`ClientConfigError`) rather than sending an empty bearer. **Residual:** a
manual `quarry login ::1` while a co-tenant squats `[::1]:8420` and the real
daemon is IPv4-only could still present the token to the squatter — non-default
and operator-initiated; the managed install never stores `::1`. An
endpoint-bound token (record the bind host beside `serve.port`, present on
exact match) would erase it and is held pending an operator decision.
The 13 `RemoteClient` construction sites route through `ClientConfig`, so any
loopback CLI session now presents the token. `ClientConfig` resolves the
token from the run dir of the process's **active database** (the
`DatabaseSelection` override the CLI records from `--db`, else the
persisted default), since `serve.token` lives under the daemon's startup-db
run dir.
**Limitation:** a loopback client cannot learn a non-default daemon's
database from the URL, so loopback `--db` auth requires the operator to run
a matching `--db` on both `quarry` and `quarryd`; the default,
service-managed case (default database) is unaffected.
- **install `/health` gate** requires `state=="ready"`, not a bare HTTP 200 (a
warming daemon returns 200 with `state=="starting"`).
- **Deferred (PR-4) — install success misconfigures the plugin's MCP transport
(`quarry-lejv`, P1).** PR-1 removed the daemon's `/mcp` WebSocket, so the
interim MCP transport is stdio `quarry mcp`, which loads the engine in-process
until PR-4's mcp-as-client lands. But `install.sh`'s successful `quarry login`
still writes `quarry.toml`, and `plugin.json` prefers `mcp-proxy` →
`wss://…/mcp` — a now-dead route — so a fresh or re-install misconfigures
Claude Code's MCP on the **success** path (a *failed* login instead leaves the
working stdio `quarry mcp` fallback in place). PR-3a's loopback login also
stores no bearer. Repointing the install MCP transport is PR-4's domain
(mcp-as-client; `mcp-proxy` out of quarry's path), so it is deferred here
rather than fixed in PR-3a, tracked as `quarry-lejv`. **Re-install guard until
PR-4:** after re-installing, keep (or point) the Claude Code plugin at stdio
`quarry mcp`; do **not** rely on the written `mcp-proxy` `quarry.toml` config.
**Rejected alternatives (operator/leader rulings):**
- **Route the local in-process CLI onto the daemon now** — rejected. It would
break CLI usage when no daemon is running; that cutover (and its daemon-down
nudge/fallback handling) belongs to `QuarryClient` in v2-3. PR-3a re-sources
only sessions already in remote mode; the no-login local path is unchanged.
- **Trust a stored token for a loopback target** — rejected. `quarryd` mints a
fresh token every restart (respawned under KeepAlive/`Restart=always`), so a
stored token is stale immediately; loopback bearers are read live.
- **Auto-generate a token for a non-loopback bind** — rejected as false
security: an auto-token in a local 0600 file is unreadable by the remote
clients that need it. A non-loopback bind still requires an operator-set key.
- **Place the token-injecting construction seam in `__main__` (a free helper) or
on `RemoteClient` (a classmethod)** — rejected: the first regresses the CLI
god module, the second grows `remote_client.py` (deleted in v2-3). The seam
lives in the new, absolute-gated `client` tier (`ClientConfig.remote_client`)
so each CLI change is a call-site swap.
- **Defer the loopback gate-flip to a later PR** (keep loopback unauthenticated
for now) — rejected. The gate flips in v2-3a; the interim `ClientConfig`
wiring keeps every loopback client (the only clients that hit the daemon
today) authenticated, so nothing is locked out and the exposure closes now
rather than one PR later.
- **Write `serve.token` only when the token is auto-generated** (skip the file
when `--api-key` is set) — rejected. Then a local client on a `--network`
server would have no token to read on loopback and would fail closed even
though a valid key exists. The file always holds the effective key.
### Context (audited reality)
A 5-agent audit (2026-07-12) established ground truth and corrected the v1 framing:
- **"Five engines per host" is a latent ceiling, not steady state.** The steady state is **one** resident engine — the `quarry serve` daemon (~1.6 GB RSS). `mcp-proxy` (~5 MB) and the menubar load no engine. The five ONNX-loading rows in the v1 table are per-invocation *possibilities*, not coexisting processes. The real driver is not duplication avoidance but **always-on incremental indexing**, which a per-invocation process cannot provide.
- **The transition v1 named never happened** — only the v1 ADR shipped. The CLI still builds the engine in-process for 18 data commands (a routing seam to `RemoteClient` exists from DES-021, but the local-engine branch was never removed). The library's public names (`Database`, `get_db`, `ingest_*`) *are* the engine. `mcp_server.py`'s stdio path loads the engine; `RemoteClient` is a de-facto thin client but CLI-coupled (`typer.Exit`/`SystemExit`). The HTTP surface has ~20 routes, **zero** Pydantic schemas, and no endpoint for `optimize`/`backfill-sessions`.
- **DES-037 already unified the *search* path** across surfaces via a shared `SearchService` library seam — bug-class-3 drift is closed for search, but every non-search command still has divergent dual paths. This design reuses `SearchService` as the daemon's internal `/v1/search` impl; it does not re-solve it. `mcp-proxy` (DES-021) already provides a thin MCP transport.
### The three invariants (operator-set)
- **I1 — Hard client/engine boundary.** CLI, MCP, and the library are **pure clients**; none may import or construct the engine (`Database`, `embeddings`, `ingestion.pipeline`, `retrieval`, `SyncRegistry`) in-process. The engine lives only in `quarry/daemon/`.
- **I2 — Daemon assumed always present.** One supervised, always-on engine per machine; clients assume it is there. Rationale: the daemon is the **one resident indexing engine** every trigger reuses instead of cold-starting ~1.6 GB per run — the precondition for always-on incremental indexing.
- **I3 — Well-specified wire protocol.** A formal, versioned REST contract is the single source of truth: shared Pydantic models generating an OpenAPI document, one `QuarryClient` conforming to it, used by CLI, MCP-thin, and library.
### Design
**Boundary (I1).** Three layers, enforced by *package membership*, not convention: `quarry/api` (Pydantic models + errors; zero heavy deps) ← `quarry/client` (`QuarryClient` + typed errors) ← client processes (`__main__`, hooks); `quarry/daemon` holds the engine and is never imported by a client. Enforced by an **import-linter** CI contract plus a **runtime-sabotage test** (import the client packages with `lancedb`/`onnxruntime` poisoned; assert success). The two host-local commands that legitimately need the engine (`serve`; `mcp` until v2-4 drops it) pull it in via a **lazy in-body import**, guarded by the sabotage test rather than import-linter. `disable` is a *hybrid*: its hook-config removal is host-local, but its chunk purge becomes a daemon call.
**Daemon-assumed (I2).** The daemon is the single engine, supervised (launchd `KeepAlive` / systemd `Restart=always`), loopback by default. It owns `SyncRegistry` and exposes `/v1/sync`; register/deregister/sync are thin mutations. `/health` reports `warming` vs `ready`. When the daemon is not running, the client **nudges the service manager** (kickstart / `systemctl --user restart`), polls `/health` under a ~30 s warm budget, then **fails fast** with a typed error distinguishing *not-installed* (→ `quarry install`) from *installed-but-down* (→ `quarry doctor`). There is **no in-process fallback** — a fallback engine would resurrect the dual-path drift I2 exists to remove. The always-on indexing *triggers* are decoupled to separate beads: interim scheduled sync (`quarry-tqdq`/`quarry-uae`) now, the event-driven watch/index loop with a serialized per-collection queue (`quarry-lxrk`) later.
**Wire protocol (I3) — FastAPI.** The daemon adopts **FastAPI** (Starlette-compatible: the WS `/mcp` route, the asyncio task system, lifespan startup, and bearer-auth middleware pass through unchanged), so request validation *and* the OpenAPI document derive from the same Pydantic models — no hand-rolled emitter or route registry. `quarry/api` holds one request/response model per operation (drift → import-time type error, the structural kill of bug-class-3 for non-search commands) and a uniform `ErrorBody` envelope via a FastAPI exception handler. All engine operations move under **`/v1`** (health/openapi unversioned); `/health` advertises `api_version`; a major mismatch raises `QuarryVersionError`. Long operations return `202 + TaskAccepted`, polled at `/v1/tasks/{id}`. The two missing operations (`/v1/optimize`, `/v1/backfill-sessions`) are added, closing split-horizon. `IngestRequest` is dual-mode (daemon-local *path* vs uploaded *bytes*, chosen by transport) so remote ingest cannot silently diverge.
**QuarryClient (I3).** One library-safe client (`quarry/client`): typed `QuarryError` hierarchy, Pydantic responses, **no `typer`/`SystemExit`/console**. `RemoteClient` is deleted; the CLI re-adds exit/print via a single boundary decorator at `__main__`. quarry-menubar (`../quarry-menubar`) is prior-art proof: a Swift `QuarryClient` over `URLSession` with pinned-CA TLS, zero Python-engine coupling — the OpenAPI doc is the shared contract both the Python and Swift clients conform to.
**MCP.** The in-process `quarry mcp` engine path is **dropped**; MCP is served by the daemon over `/v1/mcp` via `mcp-proxy`, reusing the warmed `ctx.embedder`/`ctx.database` (fixing an incidental third-ONNX-session defect). `SearchService` (DES-037) stays as the daemon's internal `/v1/search` implementation.
### Key decisions and rejected alternatives
- **FastAPI over Starlette + a hand-rolled OpenAPI emitter** (operator ruling: correctness over migration-avoidance). FastAPI *is* Starlette underneath, so the "risky migration" premise was false; the hand-rolled registry/emitter were gold-plating.
- **No in-process fallback** when the daemon is down — it would reintroduce bug-class-3.
- **MCP via `mcp-proxy` → daemon**, not an embedded thin MCP server (reuse over rebuild).
- **Watcher decoupled** from this epic into `quarry-lxrk`/`quarry-tqdq`.
- Keeping dual paths, dropping the daemon, or keeping `RemoteClient` CLI-coupled were all rejected (see `docs/des-031v2-daemon-first.md` §4).
### Scope and sequencing (supersedes the v1 6-PR table)
| PR | Bead | Scope |
|----|------|-------|
| v2-1 | `quarry-p8dq` | This ADR + `architecture.tex`/`CLAUDE.md`/README doc corrections (doc-only) |
| v2-2 | `quarry-qyrm` | FastAPI + `quarry/api` contract: schemas, `/v1`, OpenAPI, `optimize`/`backfill` endpoints, alias-route removal, menubar lockstep |
| v2-3a | `quarry-ufjt` | `quarryd` entry point + `DaemonLauncher`; `serve.token` (mode-0600) + `LoopbackPolicy` detection fix; `ClientConfig` loopback-token resolver + re-source the `RemoteClient` sites; supervised units exec `quarryd` (`Restart=always`); delete `quarry serve`; install `/health` `state==ready` gate |
| v2-3 | `quarry-veb0` | `QuarryClient` + typed errors; CLI thin-client (delete `RemoteClient` + engine branches); `disable` chunk-purge → daemon |
| v2-4 | `quarry-ydz5` | MCP over `/v1/mcp`; drop `quarry mcp`; reuse warmed `ctx` |
| v2-5 | `quarry-5e5t` | Library API = `QuarryClient` (pure in-repo removal of engine exports) |
| v2-6 | `quarry-7ftj` | Install `/health` gate + in-process ASGI fixture + import-linter boundary + one real-loopback-TLS smoke test |
The contract (v2-2) precedes clients dropping their engine paths; supervision (v2-3a) precedes fallback removal. No in-code shims (PL-PP-1): `RemoteClient`, `quarry mcp`, and the engine library exports are deleted, not aliased. The only cross-repo touch is the Swift client's `/v1` bump, coordinated in v2-2.
### Risks (summary; full table in the design doc)
Framework swap breaking the WS/task/auth path (mitigated: FastAPI inherits Starlette; the ASGI fixture exercises them); contract rewrite regressing a route (OpenAPI diff + structural equivalence); cold-start/wedged daemon (warm-budget poll + `restart` + typed not-installed-vs-down errors); remote-ingest divergence (dual-mode `IngestRequest`); the ASGI fixture not exercising TLS (one real-loopback-TLS smoke test rides the wheel gate). Default binding is loopback; remote is opt-in over the existing TLS + pinned-CA path; nothing weakens the trust model.
### Supersession
This ACCEPTED v2.1 replaces the v1 PROPOSED "Engine-First Architecture" ADR: it drops the "five engines" framing, names always-on indexing as the driver, reflects DES-037 and `mcp-proxy`, adopts FastAPI, decouples the watcher, and folds in the audited incidental defects (the `/mcp` third-ONNX-session, the stale `architecture.tex`/`CLAUDE.md` docs, the `/ca.crt` naming + task alias routes, the hardcoded `await_task` "Deregister failed" bug). Full design and audit: `docs/des-031v2-daemon-first.md`.
## DES-032: Daemon Resource Management — Thread Limits and OS Scheduling
**Date:** 2026-05-27 (revised 2026-05-29)
**Status:** SETTLED
**Topic:** Auto-detecting thread counts and OS scheduling hints for the quarry daemon
**Supersedes:** Original DES-032 (ONNX Thread Limits only)
### Problem
Three concurrent quarry processes (serve + ingest-background + CLI) each spin up ncpu ONNX threads + ncpu rayon threads + ncpu OMP threads. On 8 cores: 3×(8+8+8) ≈ 48-72 runnable threads → system load 148. Stack samples show rayon thread-pool workers and ONNX `RunInParallel` as hot frames.
Additionally, macOS App Nap throttles the windowless quarry daemon 5-10x without `ProcessType=Interactive`, because launchd classifies it as a background process eligible for power-saving throttling.
DES-027 sets `MALLOC_CONF=narenas:1,tcache:false` in the daemon environment. More threads × single arena = worse contention. Under sustained load the arena lock serialises all ONNX threads and throughput drops from 44 texts/s to 0.1 texts/s (440x slower).
### Evidence
| Scenario | Throughput |
|----------|-----------|
| Clean process, no MALLOC_CONF | 67.4 texts/s |
| Clean process, with MALLOC_CONF | 23.5 texts/s |
| Daemon after 20 min sync | 0.1 texts/s |
### Design
Auto-detect and cap all thread pools at construction time. Zero user configuration required.
| Parameter | CPU provider | GPU provider | Set where |
|---|---|---|---|
| `intra_op_num_threads` | `min(2, ncpu)` | `1` | `OnnxEmbeddingBackend.__new__` |
| `inter_op_num_threads` | `1` | `1` | `OnnxEmbeddingBackend.__new__` |
| `TOKENIZERS_PARALLELISM` | `false` | `false` | `os.environ.setdefault` |
| `OMP_NUM_THREADS` | `min(2, ncpu)` | `min(2, ncpu)` | `os.environ.setdefault` |
| `MALLOC_CONF` | DES-027 value | DES-027 value | plist/systemd env |
| macOS QoS | `ProcessType=Interactive` | `ProcessType=Interactive` | plist |
| Linux QoS | `Nice=-5` | `Nice=-5` | systemd unit |
Why each parameter has exactly one right answer:
- **`intra_op_num_threads`**: GPU offloads GEMM to CUDA so one CPU feeder thread suffices; CPU needs parallelism but more than 2 threads causes arena lock serialisation under `narenas:1`.
- **`inter_op_num_threads`**: Quarry runs a single model with sequential ops — no inter-op parallelism to exploit.
- **`TOKENIZERS_PARALLELISM`**: Disables rayon thread pool inside HuggingFace tokenizers — redundant when texts are already batched by `embed_texts`.
- **`OMP_NUM_THREADS`**: Caps OpenMP threads to match ONNX intra-op limit — without this, OpenMP spawns ncpu threads independently.
- **`ProcessType=Interactive`**: Asks macOS to exempt the windowless daemon from App Nap throttling (5-10x).
- **`Nice=-5`**: Asks systemd to keep the daemon's CPU priority high under load — search latency is user-facing.
**The OS scheduling hints are best-effort, not guarantees.** A negative `Nice` value requires privilege (`CAP_SYS_NICE` / `RLIMIT_NICE`). Under `systemctl --user`, the per-user manager generally cannot grant a negative nice, so systemd silently clamps the request to `0` — the daemon then runs at default priority and DES-032's latency guarantee does not strictly hold. `ProcessType=Interactive` is likewise an advisory QoS hint that the kernel may or may not honour. These hints improve latency where the platform permits and are harmless where it does not; the correctness of the daemon never depends on them. The thread-pool caps above are the load-bearing mitigation and are unconditional.
Note that DES-027's `MALLOC_CONF` interacts with thread count: more threads × `narenas:1` = worse contention. The thread limits here are specifically tuned for `narenas:1`.
**Session isolation** — the HTTP server's `_QuarryContext.embedder` creates a dedicated `OnnxEmbeddingBackend()` instance instead of sharing the `get_embedding_backend()` singleton with the sync pipeline. ONNX `session.run()` serialises callers via an internal mutex. With a shared session, a search query must wait for the sync's current embedding batch to complete — up to 70 seconds under `narenas:1,tcache:false`. Separate sessions eliminate this blocking. Memory cost: one extra ONNX model (~120 MB).
### Dependency
This decision depends on DES-027's `narenas:1,tcache:false`. If `MALLOC_CONF` changes to use multiple arenas or re-enables tcache, the thread limits should be revisited.
### Rejected Alternatives
1. **Increase `narenas` in MALLOC_CONF** — Tested with `narenas:4`. Reduced ONNX contention but increased jemalloc memory retention. With 4 arenas, RSS grows faster than DES-027's target. Also insufficient: queries still took 10+ seconds because session mutex contention is the primary bottleneck, not arena contention.
2. **Remove `tcache:false`** — Tested with decay-only MALLOC_CONF. RSS was 785 MB after 1 hour (DES-027 showed 2.5 GB/day without narenas+tcache). Unacceptable memory growth for a daemon that runs indefinitely.
3. **Async search route** — Making `_search_route` async and using explicit `run_in_threadpool` for blocking calls. Did not help: the bottleneck is the ONNX session mutex, not Starlette's thread dispatch.
4. **Process isolation** — Separate ONNX worker process. Eliminates all contention but adds IPC complexity. Disproportionate when session isolation achieves the same result in-process.
5. **User-configurable thread count** — Adding an `embedding_threads` field to `Settings`. Rejected for this PR because auto-detection covers all known hardware configurations correctly. Future work (DES-033) if edge cases emerge.
### Amendment 2026-07-26 — LanceDB compute-pool cap + compaction throttle (quarry-exz9)
The original DES-032 caps governed ONNX/OMP but **not LanceDB's own tokio
compute runtime**, which the Rust core sizes to `num_cpus`. With the always-on
watch loop (DES-045), each per-collection finalize — `optimize()` plus a full
`create_fts_index("text", replace=True)` rebuild — ran on that uncapped pool and
seized every core, spiking the daemon to ~291% CPU whenever a finalize fired.
The trigger is the daemon's **own** periodic maintenance (the finalize behind an
index update and the `watch_safety_scan_s` reconcile re-finalizing collections),
not developer activity — an earlier draft of this entry wrongly attributed the
spike to `make check` file churn; in fact a `make check` writes only
gitignored/excluded paths and does not touch the index. Whatever the trigger, a
background service must stay bounded, so the CPU envelope is capped by
construction rather than left to the compaction cadence.
Two structural mitigations:
1. **Cap LanceDB's compute + IO pools.** `ThreadConfig.apply_env_limits()` now
clamps `LANCE_CPU_THREADS`/`LANCE_IO_THREADS` (default 2) alongside `OMP_*`,
applied in `DaemonServer.run()` immediately before the first `lancedb.connect`
in `warm()`. The clamp is **fail-closed with a per-variable floor**: an
inherited/operator value is honored only within `[floor, cap]`; a higher or
non-numeric value is forced down to the cap (a `setdefault` would yield upward
to a stale `LANCE_CPU_THREADS=32` and void the ceiling), and a below-floor
value is raised to the floor. For the LANCE pools floor == cap == 2
(`LANCE_CPU_THREADS=1` stalls lance's runtime), so a `1` is raised to `2`,
never honored as a "tighter bound"; `OMP_NUM_THREADS` keeps floor 1. Measured
(8-core): the tokio/FTS pool tracks the cap, not
`num_cpus` (33→14 workers as the cap goes 8→2), so the ceiling holds
core-count-independently. The FTS rebuild runs on LanceDB's **native**
inverted index (`use_tantivy=False`, the default) — there is no separate
uncapped Tantivy `IndexWriter` pool, so no search-engine change was needed.
2. **Throttle the finalize.** `FinalizeThrottle` coalesces the per-batch
finalize to at most one per `watch_optimize_min_interval_s` (default 30 s)
**per database**, with a trailing timer so FTS still catches up after churn
settles. Freshness is therefore database-granular, not per-collection; the
`watch_safety_scan_s` reconcile (300 s) is the unthrottled backstop that
re-finalizes every registered collection.
Trade-off (accepted): the cap is process-global, so it collapses DES-032's
query/sync thread isolation onto one 2-thread lance runtime. Measured search
latency under a concurrent finalize is comparable-to-better than uncapped (p95
214 ms vs 264 ms — the cap tames the finalize's own flooding); the cost is on
baseline flat-scan throughput (p50 70 ms vs 45 ms). Bulk compaction is ~1.65×
slower (not the naive 4×; it is partly IO-bound). Acceptable for a background
daemon whose ingest is not latency-critical (DES-034 bounds the working set).
This also corrects a stale reference: DES-017 and DES-023 name the FTS backend
"Tantivy," but the shipped index is LanceDB's native FTS (`use_tantivy=False`).
The name is legacy; the engine is native and rides the capped runtime.
## DES-033: User-Configurable Embedding Thread Count — RESERVED
**Status:** RESERVED (not adopted)
Reserved by DES-032 (rejected alternative #5) for a possible `embedding_threads`
`Settings` knob. Not written: ONNX thread auto-detection covers all known
hardware. Open this entry only if a hardware edge case forces a manual override.
The number is held so DES cross-references stay stable — do not reuse it.
## DES-034: Bounded Progressive Commit for Sync Ingestion
**Date:** 2026-07-03
**Status:** SETTLED
**Supersedes:** DES-026 change #3 (batch-write-at-end)
**Topic:** Making ingestion incremental, crash-resilient, and memory-bounded
### Problem
DES-026 change #3 accumulated every `(chunks, vectors)` for a collection in
memory and did one `batch_insert` at the end. Consequences: nothing searchable
until the whole sync finished; a crash lost all embedding work; peak memory
scaled with collection size *and* with a single large file's chunk count
(`prepare_document` embedded a whole file at once). LanceDB (append-only, MVCC)
is fully capable of concurrent read/write and large data — the bottleneck was
quarry's batch-at-end usage, not the engine. Very large single files are a
primary use case, so the whole-file embed path was the load-bearing failure.
### Design
1. **Streaming producer** — a `DocumentStreamer` chunks a document once
(assigning a document-global, contiguous `chunk_index`) and embeds it in
bounded windows (`embed_window_chunks`, default 512), yielding
`(chunks, vectors)` sub-batches so a single large document never materializes
all its vectors. The window size is an embedding-throughput tuning seam
(kpz per the CLAUDE.md pairing table).
2. **Bounded progressive commit** — a `ProgressiveIndexer` buffers windows and
flushes to LanceDB when the buffered vector bytes reach `sync_flush_mb`
(default 32 MB, **can fire mid-file**) and at end-of-collection. Each flush is
Lance-add → **one** registry transaction covering every file the flush
touched: completion rows for files whose final window is durable *and*
advanced watermarks for files still mid-flight. All registry mutations for one
flush commit atomically (single `conn.commit()`), so the registry can never be
partially consistent with the one durable Lance version. Peak resident vectors
are bounded by `N + queue_capacity × window`, independent of file or
collection size. Writes serialize because exactly one consumer thread performs
them (not `_table_lock`, which guards only `create_table`); a `_write_lock`
additionally serializes producer overwrite-deletes against the consumer's adds.
3. **Fragment budget** — flush count is `O(total_vectors / N)`, independent of
file count, keeping fragments two-plus orders of magnitude below the 10K guard
so the single post-sync `optimize()` runs and compacts. A *file completion* is
a registry checkpoint, not necessarily a Lance flush — small files coalesce
into a shared flush, so a many-tiny-files collection cannot reopen the DES-026
spiral.
4. **MVCC progressive visibility** — each flush commits a new manifest version;
readers `open_table` per query (`chunk_search.py`) and snapshot-isolate from
the writer, so concurrent search returns partial results without blocking. The
vector channel sees fresh chunks immediately; FTS coverage lags until the
post-sync FTS rebuild (graceful vector-only fallback). LanceDB MVCC provides
blue/green visibility for incremental updates for free — no manual 2× shadow
swap; a shadow swap is reserved for a full re-index (embedding-model change),
scoped separately.
5. **Crash-resume (within-file, v1)** — committed flushes are durable. The
`files` table carries a `chunks_committed` watermark and a `partial_hash`
column; each flush advances the watermark for files it touched. On resume, a
file with a partial watermark `w` re-enters ingestion and resumes *within* the
file, re-embedding only `[w, total)`. Two rules keep this correct:
- **Delete-tail-on-resume (required).** Before re-embedding, delete every
chunk with `chunk_index >= w`. A crash between a mid-file `add` and the
watermark commit can leave durable chunks `[w, K)` in Lance with the
watermark unadvanced; without delete-tail, resume would re-add `[w, end)` and
duplicate `[w, K)`. Delete-tail makes the document's Lance contents exactly
`[0, w)` before re-embedding, so resume is idempotent under repeated crashes.
- **Determinism precondition + fallback.** The watermark is valid only when
re-chunking the same bytes yields byte-identical boundaries (holds for the
deterministic text/PDF loaders). If `partial_hash != content_hash` (file
changed) or the loader is non-deterministic (OCR/rapidocr), the watermark is
discarded and the file is fully overwrite-deleted and re-embedded from
`chunk_index 0`.
### Relationship to DES-027 / DES-032
Progressive flush bounds the resident **vector** working set to
`N + queue_capacity × window` in-process (source-text residency stays O(file)
until the loaders stream — a separate follow-on) and produces a bounded sawtooth
that DES-027's `dirty_decay_ms:1000` reclaims between flushes. Because the vector
working set is bounded in-process and DES-032's thread caps + session isolation
already protect query latency, **process isolation of the ingest worker is
deferred** (revisit only if benchmarks show query p99 regressing during large
syncs; the future path is an ingest subprocess writing Lance directly, read via
MVCC — candidate DES-035).
### Rejected / deferred
1. **Literal per-file flush** — reopens the fragment spiral on >10K-file
collections (fragments decouple from data size). Rejected in favor of
size-gated flush + per-file registry checkpoint.
2. **Add-without-delete-tail on resume** — not idempotent; a crash between a
mid-file `add` and the watermark commit would duplicate the post-watermark
chunks. Delete-tail-on-resume is required.
3. **Trusting the watermark under non-deterministic extraction** — an OCR re-run
can produce different boundaries, so the watermark is discarded when
`partial_hash` mismatches or the loader is non-deterministic.
4. **Streaming the loaders (bound source-text residency)** — deferred to a
follow-on bead; v1 bounds the vector working set, not the extracted-text
residency (O(file) via `_extract_pages`).
5. **`pyarrow.RecordBatch` build (drop the ~6× `.tolist()` transient)** — deferred
to a follow-on bead (DES-027 rejected-alt #3).
6. **Process isolation** — deferred (see above).
## DES-035: Ingest-Worker Process Isolation — RESERVED (DEFERRED)
**Status:** RESERVED (deferred)
Reserved by DES-034 for a possible ingest subprocess that writes Lance directly
and is read via MVCC. Deferred: DES-034's in-process progressive commit already
bounds the vector working set, and DES-032's thread caps + session isolation
protect query latency, so a separate process is unnecessary. Revisit only if
benchmarks show query p99 regressing during large syncs. The number is held so
DES cross-references stay stable — do not reuse it.
## DES-036: Capture PII Redaction — Placeholder Emails, Bounded Local Hostname, Single Write Choke Point
**Date:** 2026-07-11
**Status:** SETTLED
**Topic:** Write-time PII redaction of captures (paths, emails, hostname, URL metadata)
**Relates:** DES-030 (capture lifecycle); quarry-ow3k (private shadow-repo sync, depends on this)
Captures (session transcripts and WebFetch auto-captures) previously scrubbed
secrets and profanity at write time but not PII. vox reported ~598
`/Users/<user>/` path findings in its captures. Because captures are the input
to DES-030's lifecycle and to the planned private shadow-repo sync (quarry-ow3k),
un-redacted PII in a capture is a leak into a git-tracked (and soon pushed)
surface. This ADR records the settled redaction invariant that ow3k depends on.
### Decision
Three write-time PII passes were added to `scrub.py`, composed as a `Scrubber`
class (secrets → paths → emails → hostname → profanity), and a single
`CaptureWriter` choke point (`capture.py`) now serves both `.md` producers
(PreCompact via `hooks.py`, backfill via `backfill.py`), replacing two duplicated
writers. WebFetch DB-ingest is scrubbed via an opt-in `content_scrubber`
parameter on `ingest_url`.
1. **Emails → `[REDACTED:email]` placeholder, not ethos-handle mapping.**
Redaction is a security property: completeness (no false negatives) beats
attribution. A placeholder redacts every address — team, third-party, pasted
git-log authors — while handle-mapping would leak unknown emails and pull
ethos identity resolution (YAML + filesystem walks) into a leaf text
transform. Idempotent and zero-coupling.
2. **Hostnames → bounded to the local machine name** (`socket.gethostname()` +
`.local` + short leaf ≥4, case-insensitive), not arbitrary dotted-token
detection. Arbitrary detection has a catastrophic false-positive rate
(`github.com`, `config.yaml`, `quarry.db.facade`, version strings). The actual
PII is the operator's machine name; `gethostname()` targets exactly that.
3. **Ordering: email before hostname (hard constraint).** A hostname inside an
email domain must be subsumed by the whole-email redaction; if hostname ran
first it would produce `jim@[REDACTED:hostname]`, which the email regex then
fails to match, leaking the local part `jim`.
4. **Redaction is at write time and fail-closed.** Scrub runs to completion
before any atomic write; on scrub failure no file is written. Every pass emits
a marker no pass can re-match, so `scrub(scrub(x)) == scrub(x)` (backfill may
re-run). Both WebFetch ingress branches (primary + `ingest_url` re-fetch
fallback) scrub before content reaches the pushable `web-captures` collection.
5. **URL metadata redaction via `CaptureUrl` (`capture_url.py`), not just the
body.** The page body scrub (decisions 1–4) does not protect the URL itself,
which is persisted as a capture's `document_name`/`document_path`. A fetched
URL like `…/reset?email=a@b.com&token=xyz` would leak PII/secrets into the
pushable `web-captures` collection even with the body scrubbed. `CaptureUrl`
strips userinfo, query, and fragment and runs the bare `scheme://host/path`
through the scrubber before it is stored, on **both** WebFetch branches
(primary `hooks.py`, fallback `pipeline.py::ingest_url`) and for the dedup key.
IPv6 host literals keep their `[...]` brackets so the netloc stays valid
(`CaptureUrl._bracketed`). This is defence-in-depth distinct from body
scrubbing, and it is the specific property `quarry-ow3k` relies on.
### Path pass scope
The path pass rewrites `/Users|/home/<user>/` → `~/` at a genuine path boundary,
including `file:///Users/…` and protocol-relative `//Users/…`, but a lookbehind
(`(?<!:/)`) excludes a URL scheme authority so `http://home/dashboard`,
`https://example.com/home/…`, and nested `/var/home/…` are left intact (they are
not a user home directory). This distinguishes a real home path from a hostname
or URL path segment.
### Scope boundary
`content_scrubber` on `ingest_url` defaults to `None` (byte-unchanged), so
user-initiated `quarry ingest <url>`, sitemap/bulk, and directory sync are NOT
scrubbed — only the WebFetch auto-capture path opts in. Redaction is a captures
concern, not a general-ingestion concern; deliberately-ingested documents keep
their content searchable.
### Rejected / deferred
1. **Ethos-handle email mapping** — false negatives on unknown addresses +
layering/coupling cost; rejected for a security property (operator-ratified).
2. **Arbitrary hostname detection** — catastrophic false positives corrupt
capture usefulness and code snippets; rejected (operator-ratified).
3. **Scrubbing globally in `_chunk_embed_store`** — would corrupt user-initiated
ingests; rejected in favor of the opt-in WebFetch-only parameter.
4. **Git-history scrub of already-committed captures** — separate concern
(quarry-mr0l). vox's public history verified clean (0 committed captures);
forward redaction (this ADR) + shadow-repo sync (ow3k) are the go-forward fix.
5. **Unicode/IDN emails and SSH-remote (`git@host`) matching** — documented
accepted limits of the ASCII email regex (over-match on SSH remotes is
over-redaction, not a leak; IDN under-match is low-frequency).
## DES-037: Retrieval Seam — `RetrievalConfig` + `SearchService` for Single-Path, Reproducible Search
**Date:** 2026-07-05
**Status:** SETTLED
**Topic:** One production retrieval path shared by every surface and by the eval harness
**Relates:** DES-017 (hybrid algorithm), DES-031 (engine-first single path)
### Problem
Hybrid search lived in `search.py` and was invoked directly by each surface
(CLI, HTTP, MCP). This is the origin of bug class 3 (remote/local divergence):
the HTTP `/search` route once ran vector-only while the CLI ran hybrid, and query
params drifted between paths. There was also no seam to hold retrieval tunables
constant while measuring quality, which the eval work (DES tie-in below) requires.
### Decision
Introduce a `src/quarry/retrieval/` package that makes retrieval one object with
one entry point:
- **`RetrievalConfig`** (`config.py`) — a frozen config carrying the committed
production baseline: `rrf_k=60`, `fetch_multiplier=3` (3× over-fetch),
`metric="cosine"`, `embedding_strategy="baseline"`. This is the single place a
tunable is defined, so an eval run can vary one lever and hold the rest fixed.
- **`SearchService`** (`service.py`) — the one `retrieve()` path. All three
surfaces (`http_server.py`, `__main__.py`, `mcp_server.py`) now construct a
`SearchService` and call it; none reach into the ranking internals. This kills
the divergence bug class *by construction* — there is no second path to drift.
- **`HybridRetriever`** (`hybrid.py`), RRF **`fusion.py`**, **`reranker.py`**,
and structural **`protocols.py`** (`Retriever`/`Reranker`) — the algorithm
(DES-017) extracted out of `search.py` behind protocols so a lever
(reranker, embedding strategy) can be swapped without touching callers.
The same seam is what the eval harness drives: `make eval` (PR #344) constructs a
`SearchService` from a `RetrievalConfig` and measures ranx metrics (per-bucket
MRR/success@k + a metadata-pollution diagnostic) against a known-item baseline.
Production and evaluation therefore exercise identical retrieval code.
### Rejected / deferred
1. **Leaving hybrid in `search.py` and adding an eval shim** — the shim would be
a second path, reintroducing the divergence risk; rejected in favour of one
`SearchService` both production and eval share.
2. **Baking lever choices in now** — `embedding_strategy` and reranker are behind
protocols but the production default stays `"baseline"` until a lever wins on
the numbers (a lever bake-off is future eval work; see
`docs/eval-harness-design.md`).
## DES-038: Normalized Embeddings + Cosine Metric, with Honest FTS-Only Scores
**Date:** 2026-07-04
**Status:** SETTLED
**Topic:** Bounded, comparable similarity scores across both retrieval channels
**Relates:** DES-017 (hybrid), DES-037 (retrieval seam)
### Problem
Vector search ran LanceDB's default (unbounded) distance and reported
`similarity` without normalization, so scores were not bounded to `[0, 1]` and
were not comparable across queries. Separately, FTS-only result rows (matched by
BM25 but absent from the vector channel) reported a synthetic `1.00` similarity —
a fake perfect score that mis-ranked keyword-only hits and polluted any
score-based diagnostics.
### Decision
- **L2-normalize embeddings and search with the cosine metric** (PR #336). Every
vector is unit-normalized at embed time and vector search uses
`metric("cosine")` (`db/chunk_search.py`); `RetrievalConfig.metric` defaults to
`"cosine"` (DES-037). Similarity is now bounded and comparable — a precondition
for meaningful RRF fusion and for eval metrics.
- **FTS-only rows report their true cosine** (PR #339), computed against the
query embedding, instead of the fake `1.00`. A `SearchResult` value type
carries the real score so keyword-only hits rank honestly.
### Rejected / deferred
1. **Keeping unbounded distance and post-hoc rescaling** — rescaling a
per-query, unbounded distance is not stable across queries; normalization +
cosine is bounded by construction.
2. **Dropping FTS-only rows that lack a vector score** — they are legitimate
keyword hits; computing their true cosine keeps them and ranks them correctly.
## DES-039: Private Capture Shadow-Repo Sync — `<repo>` → `<repo>-quarry`
**Date:** 2026-07-11
**Status:** SETTLED
**Topic:** Move redacted captures off the public repo into a per-project private
shadow; commit + push from `quarry sync`
**Relates:** DES-030 (capture lifecycle), DES-036 (write-time PII redaction — the
dependency this builds on)
### Problem
Redacted session captures land as `.md` files in the project's captures dir. The
public repo gitignores that dir, but the files have no durable home — they live
only on the operator's disk and are lost on a clean checkout. DES-036 redacts
only *new* captures at write time; captures already on disk predate it and are
un-redacted for PII. A naive "push whatever is there" first sync would leak PII
to a git remote on day one.
### Decision
1. **Per-repo private shadow `<repo>-quarry`** (not one org-wide captures repo).
The gitignored captures dir is a standalone nested git working tree, never a
submodule of the public repo, with a fail-closed **allowlist** `.gitignore`
(`*` / `!.gitignore` / `!session-*.md`) so only quarry's own capture files can
be staged — a stray non-`.md` file can never bypass the `.md`-scoped
re-scrubber. Two fail-closed bootstrap gates: refuse if the captures dir is
not gitignored by the parent, and refuse if the parent public repo already
**tracks** any capture file (`git ls-files`). `.gitignore` does not untrack, so
already-committed captures must be `git rm --cached`'d first; and an
already-**pushed** capture additionally needs a history purge (`git
filter-repo`/BFG + force-push, coordinated with the repo owner) since
`rm --cached` does not rewrite history.
2. **Opt-in `shadow:` block** in `.punt-labs/quarry/config.md` (default
`enabled: false`); remote derived as `<origin>-quarry` when unset.
3. **Push runs at the end of `quarry sync`** (fail-open) and via explicit `quarry
captures push`; auth reuses the user's git credentials (no new secret storage).
4. **SECURITY (re-scrub only + trust boundary):** before each **COMMIT** (not
just the push — `git push` ships all unpushed commits, so the gate is at
commit time), re-scrub the staged `.md` bytes with the DES-036 `Scrubber`
(idempotent: redacted files are no-ops; pre-fpc5 files get redacted), then run
an I/O-race guard asserting the staged bytes are a `scrub` fixed point and
ABORT-before-commit on any mismatch. `push` is never in a `finally`. The guard
is **not** an independent PII oracle — it shares the scrubber's rules, so it
catches stage/commit races, not scrubber blind spots. Residual PII classes the
scrubber cannot catch (IDN/unicode email, non-`/Users`/`/home` paths,
**cross-host** hostnames) are backstopped **solely** by the private-remote
trust boundary — so visibility enforcement is **load-bearing**: verifiably
public remotes are refused, and unverifiable visibility (no `gh`) requires an
explicit `acknowledge_unverified`.
5. **Offline/failure:** git's local commit log is the durable, resumable queue;
the next sync pushes accumulated commits.
### Rejected / deferred
1. **One org-wide captures repo** — breaks per-repo access control.
2. **Push on capture write** — chatty, risks blocking sessions; the push runs at
sync time instead.
3. **A second, independent PII detector beyond the shared `Scrubber`** — it would
only reproduce the scrubber's blind spots at double maintenance cost; the
operator ruled re-scrub-only + trust boundary instead.
4. **Gating the push instead of the commit** — a poisoned commit leaks on the
next push, so the gate must protect the commit.
5. **quarry auto-creating the private repo by default** — leak risk on a wrong
owner or an accidental `--public`; configure-only by default, opt-in
`--create` with `gh` private-verification.
---
## DES-040: Merge-Base OO/Coupling/Suppression Ratchet (adopt vox's tooling)
**Status:** Accepted (2026-07-12) · **Bead:** quarry-05mb
**Context.** quarry's ratchet was a 1,238-line `tools/oo_score.py` that compared
touched files against the **in-tree** `.oo-baseline.json` (`git diff HEAD~1..HEAD`
for the touched set). That model drifts and invites rebaseline-gaming: an author
can hand-edit the baseline inside a PR to lower the comparison floor. The sibling
repo `vox` rebuilt the ratchet as merge-base-scored packages, which closes that
gap; the operator ruled vox authoritative and directed adoption before further
DES-031 v2 work.
**Decision.** Adopt vox's tooling **verbatim** — `tools/oo_ratchet/` (13 modules),
`tools/coupling/` (17), `tools/suppression/` (9) — reducing the three old monoliths
to thin shims. Scoring compares the working tree against the baseline **committed
at `git merge-base origin/main HEAD`** (read via `git show <base>:<baseline>`), not
the in-tree file. `check-coupling` and `check-suppressions` join `check-oo` in the
`make check` chain; CI enforces with `--base-ref <merge-base> --require-base` and
`fetch-depth: 0` on PRs plus a `HEAD~1` push tripwire. This is a **hybrid** model:
the committed baselines stay (per-commit integrity lock + `--relax` audit anchor),
but the comparison floor is the immutable base-commit blob, so gaming is blocked at
every merge hop. Metric thresholds are unchanged (vox's tables were byte-identical
to quarry's); `.oo-baseline.json` values are byte-identical after cutover (zero
positional-only params in `src/quarry/`, so vox's PEP-570-aware `_avg_params`
yields the same numbers).
**The one quarry-origin divergence.** `tools/suppression/patterns.py` retains
quarry's `tokenize`-based suppression counter instead of vox's regex + AST
heuristic (which has documented blind spots: `# noqa` after `async def`, after
`obj.attr =`, on tuple targets, and single-line docstrings containing `noqa`). It
keeps vox's public interface so the rest of `tools/suppression/` stays verbatim.
This fix is to be upstreamed back into vox (quarry-njmr).
**Rejected / deferred.**
1. **Keep quarry's monolith** — retains the drift/gaming problem the move is meant
to fix.
2. **Full-delete / recompute-from-source baseline** — kills the per-commit
"commit `.oo-baseline.json`" chore, but it is an untested re-architecture that
drops the audit log and vox's fail-closed ladder; net ratchet-strength
regression. Operator ruled hybrid.
3. **Carry forward quarry's four baseline-era features** (`--verify` phantom guard,
`--correct/--reason`, ratio-tolerance band, asymmetric `module_size` headroom) —
operator directed taking vox's stricter behavior verbatim; only the tokenize
suppression counter is retained.
**Consequence.** In-flight branches (esp. `qyrm` and the rest of the DES-031 v2
chain) must rebase onto post-adoption main and reseat their per-file baselines;
once `check-coupling` is a gate, a branch whose base predates adoption fails coupling
with "base commit predates baseline adoption — rebase onto current main" (correct
fail-closed behavior). Vox's stricter gate (no ratio tolerance, no asymmetric
`module_size` headroom) may block growth that previously passed.
---
## DES-041: One scrubbing content-ingest path — remember and capture collapse; the capture hook goes thin
**Status:** Accepted (2026-07-19) · **Bead:** quarry-en68 · **Full design:** `docs/des-capture-ingest.md`
**Context.** Two accidental gaps shared one fix. (1) The inline-content DB write did
not scrub: `RememberJob` called `ingest_content` with no scrubber, and the
PreCompact transcript capture rode the same path — so `memory-<agent>` and
`<repo>-captures` collections stored secrets/PII in cleartext. Only the git-tracked
`.md` capture file was scrubbed (DES-036 covered that surface only). (2) The
capture/PreCompact hook was the last fat client of the DES-031 daemon-first cutover:
it spawned a detached subprocess that built a full ~1.6 GB engine per compaction.
With no concurrency cap, ~14 cold engines oversubscribed an 8-core host to load
77-97 (quarry-lnog). An earlier draft assumed `remember` was *deliberately*
unscrubbed and proposed a separate scrubbing capture endpoint; the operator
established that assumption was false — remember simply never scrubbed, and should.
**Decision.** One scrubbing inline-content core, shared by two correctly-named entry
points, with the collection as a parameter (reuse the logic, never the name):
`remember` (`POST /v1/remember` → `memory-<agent>`) and `capture`
(`POST /v1/capture` → `<repo>-captures`, one door for both the transcript and
web-fetch triggers). Both build one `ScrubbedIngestJob` that always scrubs
server-side, on the worker thread, **before** embed/store (fail-closed: a raised
scrub writes zero chunks and — after DES-041's fix — deletes nothing on overwrite).
The capture hook becomes thin: it writes its durable local `.md` + raw archive, then
POSTs content to the resident daemon via `QuarryClient` fire-and-forget (202), and
returns — no per-hook engine (kills quarry-lnog), no engine import (unblocks the PR-6
boundary, gated by a runtime engine-sabotage test, not import-linter, since the
hook's imports were already lazy). Directory `sync` fills the `<repo>` core
collection from files on disk and stays **unscrubbed** (scrubbing source corrupts
it); an unregistered cwd falls back to `default-captures` via the standard
`<repo>-captures` pattern (the one-off `session-notes`/`web-captures` names are
retired). Captures are kept out of the core collection **structurally** —
`.punt-labs/quarry/captures/` is added to sync-discovery's `_DEFAULT_IGNORE_PATTERNS`
rather than relying on an ambient `.gitignore` line. As a security completion under
"ship means secure," a non-loopback bind now requires TLS (a key authenticates but
does not encrypt), and the client resolver refuses a plaintext non-loopback target
regardless of token presence — so raw pre-scrub content only ever crosses loopback
or TLS. Legacy cleartext already in the DB is **not** swept (operator-ruled
forward-only; a future full purge covers it).
**Rejected alternatives.** A dedicated capture endpoint justified by scrub semantics
(the false "remember must stay raw"); reusing the `/remember` name/`RememberJob` for
captures (overloads the name — a reader can't tell a capture from a remember); a
`scrub: bool` flag (a mode a caller sets wrong silently corrupts); a discriminated
`IngestRequest{url|content}` (revives the scoped-out user file-ingest surface); a
daemon path-read mode (arbitrary-local-file-read surface — content-over-path avoids
it); scrubbing source on sync (corrupts fixtures/paths/changelogs); relying on
`.gitignore` to keep captures out of `<repo>` (per-repo, fragile); scrubbing on the
event loop (blocks every other request); a one-time legacy-cleartext sweep
(forward-only ruling).
**Consequence.** `pipeline.py` decomposed 1,475→~1,090 (`ImagePreparer` consolidated
to one implementation, `TextLikeFormat`); `background_ingest.py` and `BackgroundIngest`
deleted (PL-PP-1, no
shim). Scoped ratchet relaxations were taken only for irreducible wire-contract /
security-gate growth on pure-schema/re-export files that carry no offsettable metric
(real principal paid on `pipeline.py`/`ingestion`), each justified in the audit logs
and verified against the merge-base. quarry-lxrk (the serialized per-collection queue)
remains the follow-on that bounds bursty concurrent captures; quarry-czf3 decides
whether an unregistered cwd should nudge-to-register rather than fall back at all.
## DES-042: Daemon-owned serialized capture/index queue
**Status:** Accepted (2026-07-20) · **Bead:** quarry-lxrk (queue portion) · **Full design:** `docs/des-capture-queue.md` (PR #371)
**Context.** DES-041 made captures fire-and-forget to the daemon, but bursty
concurrent captures (a compaction and a web-fetch landing together, several
sessions compacting at once) fired unbounded `asyncio.create_task` ingests that
oversubscribed the box — the load-90 starvation (quarry-lnog). The per-collection
LanceDB writer assumes a single caller; concurrent overwrite delete-then-adds
interleave so both chunk sets survive (lost update).
**Decision.** A daemon-owned serialized queue: one resident FIFO worker per
collection — one in-flight `progressive_insert` per LanceDB table, the
daemon-scope generalization of DES-034's single-writer invariant (not a fork;
`_write_lock`/`ProgressiveIndexer` untouched) — plus a global embed `Semaphore`
hard-clamped to 1 (DES-032: >1 buys no matmul parallelism behind the shared ONNX
session mutex and re-adds arena contention), and non-blocking bounded admission
(→ 503 on full, never a silent drop). The worker map is reaped when idle and
hard-capped so client-controlled collection keys can't accrue resident workers.
Drain-on-shutdown is bounded; an aborted un-durable job (remember/ingest) is
spooled to a private `0600` file with a truthful task status. No wire change.
**Rejected alternatives.** Reusing sync's `CollectionIngestor` (per-sync lifetime and
registry-coupled); a coroutine-level per-job timeout (inert against the
non-cancellable `run_in_threadpool` ingest — the hang is instead bounded at the
fetch's own socket timeout); `abandon_on_cancel=True` (detaches a thread that can
re-open the delete-then-add lost-update).
**Consequence.** Bursty-capture starvation is structurally closed. Follow-ons
filed as beads: remember-durability under drain-abort, total-deadline fetch
hardening, plain-ingest collection-key precision, abandon-safe interruptible
timeout, ingest-queue asyncio cleanup. The always-on watch/index loop
(quarry-lxrk residual) remains.
## DES-043: Fetch-safety — the SSRF gate runs on every fetch hop (redirects + sitemap crawl)
**Status:** Accepted (2026-07-21) · **Bead:** quarry-5pg1 · **Full design:** this entry
**Context.** The SSRF guard (`UrlSafetyCheck.reject_reason`) ran only on the
INITIAL source at the route boundary (`/v1/ingest`, `/v1/capture`) and was never
re-run downstream, so two categories of URLs reached the fetcher ungated.
(1) **Redirect targets** — `WebFetcher` delegated redirect-following to urllib's
auto-redirect, and the only post-redirect check (`_reject_non_html`) validated
the final URL's *scheme*, not its host/address, so a caller-controlled public
server could `302` to `169.254.169.254`, loopback, or a private range.
(2) **Sitemap-crawl URLs** — the crawl runs through `ultimate-sitemap-parser`
(USP), which fetched every sitemap-index, `robots.txt`, and sub-sitemap
server-side with its own `requests` client, recursing (to depth ~11) *before*
quarry saw any leaf entry; gating only the flattened leaf output missed all of
it. Both vectors are authenticated-only (a valid API key) and reachable
identically via CLI or MCP ingest — the daemon fetch is the shared choke point.
Distinct from, and a superset of, the DNS-rebind TOCTOU beads
(quarry-ljym/quarry-kmzo): entire URL categories never reached the gate, no DNS
trickery required.
**Decision.** Gate every fetch hop against the RESOLVED address, at the fetch
boundary, fail-closed — never after the fetch. `UrlSafetyCheck` relocates
daemon→core (`src/quarry/url_safety.py`) so the daemon route and the ingestion
fetch layer share one classifier without a package cycle. Two guarded seams,
both over a single module-level `GUARDED_OPENER`:
- **Redirects** — `SsrfGuardedRedirectHandler` (`ingestion/ssrf_redirect.py`)
re-runs the gate on every 30x `Location` before opening it; `WebFetcher`'s
final-URL check now validates the resolved host/address, not just the scheme.
- **Sitemap crawl** — `GatedSitemapWebClient` (`sitemap_web_client.py`)
implements USP's `AbstractWebClient` and runs the gate on the initial URL and
every redirect hop *before the socket opens*, threaded into BOTH USP entry
points (`sitemap_tree_for_homepage`, `SitemapFetcher`), so index recursion,
`robots.txt` `Sitemap:` lines, and nested sub-sitemaps are gated at every
depth; a blocked target returns a non-retryable error so USP skips it and
never connects. Discovered leaf entries are re-gated (`reject_unsafe`) as
defense-in-depth.
The classifier rejects link-local, loopback, RFC-1918, CGNAT, unspecified,
IPv4-mapped-IPv6 variants (normalized to the embedded IPv4), and the NAT64
well-known prefix; it loops all `getaddrinfo` records (multi-record DNS → any
internal record rejects) and fails closed on a resolution error. No
TLS/connection-layer change.
**Rejected alternatives.** Gating only the leaf sitemap output (USP has already
fetched the indexes/robots/redirects server-side — the trap a mocked test hid);
IP-pinning the resolved address here (that is the DNS-rebind TOCTOU fix,
quarry-kmzo/ljym — a connection-layer change; gate-every-hop is complementary and
closes the redirect/sitemap gap without it); trusting the scheme-only
`_reject_non_html` check as sufficient.
**Consequence.** The redirect and sitemap SSRF vectors are closed at the fetch
boundary for both CLI and MCP callers, with dedicated unit coverage for the
classifier and both guarded seams and sitemap tests that drive USP's real
recursion (no wholesale mock). A residual DNS-rebind TOCTOU (the gate resolves,
the socket re-resolves) and network-specific NAT64 prefixes remain the tracked
pinning follow-up (quarry-kmzo/ljym), to be unified as validate-then-pin every
hop.
## DES-044: Fetch IP-pinning — connect to a connect-time-validated address (DNS-rebind)
**Status:** Accepted (2026-07-22) · **Bead:** quarry-kmzo + quarry-ljym · **Full design:** `docs` design note (`.tmp/design/dns-rebind-pinning.md`) + this entry
**Context.** DES-043 gates every fetch hop against the resolved address, but the
gate (`UrlSafetyCheck.reject_reason`) and the socket connect resolved DNS
*independently*: `reject_reason` ran `getaddrinfo` and validated, then threw the
result away, and `http.client`'s `connect` re-resolved the hostname at socket
time. A party controlling both an attacker-reachable URL (an `/ingest` source, a
capture `source_url`, a sitemap `loc`, a redirect `Location`) and an
authoritative DNS server could return a safe public IP at gate time and a
blocked internal/metadata IP at connect time (classic DNS rebinding). The window
was documented at `url_safety.py:44-48`; the metadata-name denylist did not close
the address-rebind.
**Decision.** Pin AND re-validate as **one** connect, so the change is
*impossible by construction* rather than merely mitigated. `PinnedHTTPConnection`
/ `PinnedHTTPSConnection` (`src/quarry/ingestion/pinned_connection.py`) perform
exactly one `getaddrinfo` on the safety path — inside `connect`, via
`UrlSafetyCheck.validated_addresses(host)` — and connect the socket to a
validated IP **literal** drawn from that same result set. There is no second,
independent resolution. The seam is the stdlib's own instance attribute
`HTTPConnection._create_connection` (set to `socket.create_connection` and called
by `connect`); the pinned classes override `connect()` to rebind it to a
validating connector, then call `super().connect()` — no `__init__` override
(PY-CC-1 clean), and via the MRO `PinnedHTTPSConnection(HTTPSConnection,
PinnedHTTPConnection)` the HTTPS `connect` still runs
`wrap_socket(server_hostname=self.host)` after the pinned TCP connect. One shared
`GUARDED_OPENER` composes the per-hop `SsrfGuardedRedirectHandler` + the pinned
HTTP(S) handlers + `ProxyHandler({})`; both attacker-reachable surfaces
(`WebFetcher.fetch`, `GatedSitemapWebClient.get`/USP crawl) use it, and each
redirect hop opens a fresh pinned connection that re-resolves-and-re-validates
the new host. `UrlSafetyCheck` gains `validated_addresses` (raises
`UrlRejectedError`, returns the validated address set, fail-closed,
all-records-reject-if-any-blocked) — the single seam the pinned connection calls
inside `connect`; `reject_reason` is retained as the None-means-safe
route-admission wrapper (it calls `validated_addresses` directly and is now pure
defense-in-depth whose divergence from the connect-time result is irrelevant to
safety). The block predicates (metadata denylist, CGNAT, IPv4-mapped
normalization) are one policy shared by both the admission and connect paths.
**Trust-domain invariant.** The pin narrows the *address*, not the *trust*.
`self.host` is never mutated → SNI = hostname, cert verified against the hostname
(never the pinned IP), `Host` header = hostname. The public-fetch context stays
`ssl.create_default_context()` (system trust store, `check_hostname=True`,
`CERT_REQUIRED`) — deliberately **not** the daemon-RPC pinned-CA context in
`client/`/`tls.py`, which is untouched.
**Rejected alternatives.** `reject_reason` returning `(reason, addrs)` and
rebuilding a per-fetch opener with those pinned addresses (P2) — reintroduces two
resolutions that must be proven equal and adds per-request plumbing to a
module-global opener for no security gain; switching the fetch path to `httpx`
with a custom resolver (the fetch path is entirely `urllib`; httpx lives only in
the daemon-RPC client — the switch would rewrite the redirect gate, size cap,
wall-clock deadline, and the USP `AbstractWebClient` adapter for nothing);
overriding `__init__` to rebind the seam (the `connect()` rebind is PY-CC-1
clean and needs no `__init__`); deleting `reject_reason` (churns route admission
and the final-URL checks for no safety gain).
**Consequence.** The DNS-rebind TOCTOU is eliminated on every attacker-reachable
fetch path (ingest, capture, sitemap crawl, redirect hops) with no TLS-trust
weakening, closing quarry-kmzo + quarry-ljym and completing the "validate-then-
pin every hop" mechanism DES-043 deferred. Covered by a headline rebinding
simulation (safe-at-gate / blocked-at-connect → `socket.create_connection` never
reached with a blocked literal) plus the five recurring bug classes (TLS
semantics, socket-leak failure injection, remote/local equivalence, exception
boundaries). `_create_connection` is a semi-private stdlib seam; a pin-target
assertion test fails loudly if a future CPython refactor bypasses it. `proxy.py`
(hardcoded GitHub hosts, install-time) is out of scope — not attacker-supplied.
## DES-045: Always-on filesystem watch/index loop (daemon-owned, all databases)
**Status:** Accepted (2026-07-22) · **Bead:** quarry-lxrk (watch-loop residual; the
DES-042 queue portion merged earlier) · **Full design:**
`.tmp/design/watch-index-loop.md` + this entry
**Context.** DES-031 named always-on incremental indexing as *the* first-class
rationale for the daemon: a per-invocation CLI cannot keep an index fresh as
files change. DES-042 landed the concurrency-critical half — the serialized
per-collection queue — but the daemon still only indexed on an explicit
`quarry sync`. The residual was the watch loop itself, and a prior draft was
non-ratifiable because it never reconciled its serialization against the
existing DES-042 / DES-034 / DES-026 layers.
**Decision.** The daemon runs one `WatchLoop` that is a **producer only**: it
watches the registered directories of **every** database in the on-disk roster,
debounces edit bursts, and submits `IngestUnit`s to the **existing DES-042
queue**. It invents no second queue and writes no LanceDB table directly, so it
inherits the whole stack unchanged — DES-042 per-table FIFO → DES-034
`ProgressiveIndexer`/`_write_lock` → DES-026 WAL + `busy_timeout`. The queue's
routing key becomes `(database, collection)` (`RouteKey`), extending
single-writer-per-table across the entire roster. Three producers (initial scan
on start, live watch, explicit `quarry sync`) all feed the one queue and thus
cannot race or double-write.
The fragment-budget-vs-fairness reconciliation (the crux the draft missed): a
small delta re-indexes as per-file `FileIndexJob`/`DocumentDeleteJob` (fragment
cost negligible, embed gate released between files for cross-collection
fairness); a burst above `watch_bulk_threshold` (default 50) for one collection
collapses to a single `CollectionSyncJob` running the unchanged
`CollectionIngestor` (DES-034's size-gated flush preserves the fragment budget).
The per-file DES-034 core is extracted as `SingleFileIndexer`, shared by both the
bulk path and `FileIndexJob` (a real paydown: `sync_ingest.py` module_size
285→237, efferent 8→5).
**FTS-rebuild coalescing is a hard resource constraint, not an option.** All
index writes use the daemon's persistent LanceDB connection, and
`create_fts_index(replace=True)` pins deleted-file readers the LanceDB Rust core
never evicts — so a per-file FTS rebuild would reopen the quarry-0dss fd leak.
`FileIndexJob` therefore never rebuilds FTS; a lone `CollectionFinalizeJob`
(`SyncFinalizer`) runs post-quiescence per `(database, collection)`, FIFO behind
the file jobs. `test_watch_session_does_not_leak_descriptors` proves the fd count
plateaus over hundreds of edits across two databases — the 0dss shape.
**All-databases (operator ruling, 2026-07-22).** The operator chose to watch
every registered database, not only the startup-active one. Each database gets
its own persistent connection + observer set (macOS FSEvents stream / Linux
inotify with a `PollingObserver` fallback on `ENOSPC`); the fd-plateau invariant
holds across all of them. **Trust invariant:** the only path to opening a
non-active database is enumerating the on-disk roster (`quarry_root` subdirs
holding a `registry.db`); no network/registry-request field can steer a DB-root
open — verified, no remotely-steerable path. This retires `quarry-uae`'s
rationale (continuous freshness without a timer) for every roster database.
**Rejected alternatives.** A parallel `asyncio.Queue` for the same tables (the
draft's sin — two writers, race hazard); a resident cross-job `ProgressiveIndexer`
per collection (a `delete_document` on a document whose windows are still
buffered resurrects stale chunks — needs a drain-before-foreign-job coupling not
worth it); one coarse `CollectionSyncJob` per fs-event (full rescan per keystroke,
holds the embed gate, breaks fairness); native inotify/FSEvents hand-rolling (two
platform paths watchdog already solves); polling-only as the primary (poll
latency + CPU walking large trees — the interim behavior being retired).
**Consequences.** Continuous freshness with no human action, bounded by
construction: embed-gate=1 (CPU), DES-034 windows (memory), coalesced FTS (fds),
per-`(database,collection)` FIFO (single writer). **This "bounded by
construction" claim proved incomplete — see DES-045b (an unrestricted watch
*scope* let a `/private/tmp` registration storm the system temp dir) and
DES-045c (LanceDB's own compute pool was uncapped); both surfaced only when the
installed daemon was measured under real load, not in the idle path this claim
was reasoned from.** The explicit-sync 409 is
dropped in favor of transparent enqueue (202 + poll; the 409 stays for
`optimize`/`backfill`). `watch_enabled` defaults on (installing `quarryd` is
already the opt-in to a background engine; reverses the prior uae opt-in
posture). **`quarry-uae` fully retired:** a periodic roster reconcile
(`watch_safety_scan_s`, default 300 s) re-enumerates the roster each interval —
it begins watching databases created after `start()` (the mid-run-creation edge)
and re-submits any bulk scan the queue shed under load, so no registered
directory is left unindexed and no timer-based periodic sync is needed. The
reconcile runs synchronously on the loop between intervals (no interleave with
register/deregister) and is the single backstop for both the new-database and
shed-scan cases. (The reconcile's blocking reads — the chunk-table scan and the
registry read — are offloaded to a threadpool; only the roster enumeration and
the teardown/drain decisions stay on the loop thread, and removal actions run
only on a fully-successful enumeration so a partial read never tears down a live
watch.)
### DES-045a: Removal lifecycle, orphan sweep, and retained-collection identity
**Status:** Accepted (2026-07-24) · **Bead:** quarry-lxrk (follow-up) · **Formal
model:** `docs/spec/watch_lifecycle.tex` (fuzz-clean, ProB-checked, I1–I7)
The watch loop's *removal* half — deregister, subsumption, and orphan cleanup —
proved to be the concurrency-critical part and was hardened under an explicit Z
model rather than through review iteration. The state (`registered`, `watched`,
`pending`, `chunks`, `retained`, and a per-collection `dir` identity map) carries
ten invariants (I1–I10), all model-checked across the full reachable space. The
ProB model includes **non-directory collections** — captures, agent memories,
`remember` targets, URL-host collections — as a first-class class (`dircoll` is
the directory-registration subset); the earlier *absence* of that class is the
abstraction error that produced the I6 erratum below.
- **I1** `registered ∩ pending = ∅` — a live collection carries no deferred purge.
- **I2** `(chunks ∩ dircoll) ∖ (registered ∪ retained) ⊆ pending` — every
*directory-collection* orphan is tracked for purge. (The unrestricted
`chunks ∖ (registered ∪ retained)` was the closed-world falsehood — see the I6
erratum: a capture/memory has chunks yet is never registered/retained/marked, so
it violated the unrestricted form.) The model-check found the code deferred a
shed purge on the *subsume* path but not the symmetric *deregister* path — a
reachable untracked orphan; fixed by making both paths defer
(`registration_lifecycle.py`).
- **I3** `registered ⊆ watched` · **I4** reconcile preserves I3 · **I5**
`watched ∩ pending = ∅` (single writer per table).
- **I6** the orphan sweep deletes only `(pending ∩ chunks) ∖ (registered ∪
retained)` — a **closed set** of collections the registry has *explicitly marked
for purge* (the durable `pending_purge_collections` table), never the open-world
complement. **Erratum — the catastrophic miss.** An earlier revision of this ADR,
and the Z model it was checked against, defined the sweep as the *open-world*
`chunks ∖ (registered ∪ retained)` and the model validated it — because the model
wrongly assumed every collection is a directory registration. In the real system
captures (`*-captures`), agent memories (`memory-*`), the `default` `remember`
target, and URL-host collections have chunks but are never directory-registered,
so the open-world sweep would have purged **all captures and all agent memories
~5 minutes after every daemon start** (`watch_safety_scan_s` default 300).
Cursor/Copilot caught it; djb confirmed it default-on catastrophic. **Fix:** a
collection is marked in `pending_purge_collections` (in the same transaction as
the `directories` delete) *only* on a non-keep-data deregister or a subsume
eviction; the sweep purges only marked collections still holding chunks, so
non-directory collections are **structurally unreachable** (I9). A crash between
mark and purge leaves the marker → next sweep drains it (the cross-restart
durability the sweep exists for). `pending`/`registered`/`retained` are read
under **one `BEGIN … commit` snapshot**, and `CollectionPurgeJob` re-checks the
same three under its own transaction at execution time — skipping the delete if
the collection was re-registered/kept/cleared after the sweep's snapshot, which
closes a live-collection race on a disable→re-enable toggle. The sweep read is
fail-closed (`except Exception` → skip the cycle; `# noqa: BLE001`
operator-approved for backstop liveness) so a transient DB error cannot kill the
reconcile loop.
- **I7** a retained collection's chunks are re-adopted **only by the directory
that archived them**. `retained_collections` records the `original_directory`;
`register_directory` clears the marker (adopts the chunks) iff the registering
directory matches, else refuses before any mutation. A legacy NULL origin
matches no directory and is refused (never-merge on unverifiable origin). This
closes a cross-project data-merge bug: an unrelated directory reusing an
archived collection's leaf name (`backend`, `docs`, …) would otherwise silently
inherit the first project's chunks. Client and daemon name-pickers avoid
archived names, and the retained set (with `original_directory`) is surfaced on
the HTTP `/registrations` endpoint for remote/local parity.
- **I8** a re-adopt (same-directory re-enable) leaves no stale document — the
re-adopt reconciles the collection's stored documents against disk and prunes
those whose file is gone, keying on the authoritative absolute `document_path`
and pruning only on *definite* absence, so a nested/basename-stored or
transiently-unreadable file is never wrongly deleted.
- **I9** the sweep never deletes a collection that was never marked for purge —
in particular a non-directory collection (capture, memory, `remember` target,
URL host) is structurally unreachable by the sweep. This is the invariant whose
absence (an unmodeled collection class) produced the I6 erratum; its negative
control reproduces exactly that capture/memory data loss.
- **I10** a different directory is never *auto-assigned* a collection name that
already holds another project's chunks. The fresh-name picker avoids
`registered ∪ retained ∪ chunk_collections` on both surfaces (the client
`Registrations`/`CollectionNamer` and the local hooks `CollectionResolver`,
including the `{leaf}-{hash}` fallback tier). A merge can only arise by claiming
a chunk-bearing name, so avoiding *every* chunk-bearing name is
necessary-and-sufficient to make the auto path merge-proof — this closes a
family of four cross-project-merge findings (subsumed-child name reuse; and the
captures/`memory-*`/`remember`/URL-host auto-merge). The picker **fails closed**:
when the daemon is unreachable the chunk set is unverifiable, so session-start
skips the auto-registration and nudges rather than arm a latent merge that would
materialize on daemon return (registration is local/un-gated, but indexing is
daemon-gated, so deferring costs nothing). Same-directory keep-data re-adopt is
checked first (local retained markers) and is unaffected. Non-vacuous — the
negative control (picker avoiding only `registered ∪ retained`, dropping
`chunk_collections`) reproduces the merge in a 4-step trace.
**keep-data re-enable semantics — re-adopt + auto-freshen (operator ruling,
2026-07-24).** When a directory disabled with `--keep-data` is re-enabled, the
*same* directory re-adopts its archived collection (reusing the kept index) and
the re-adopt triggers a full reconcile that prunes chunks for files deleted while
it was disabled — so a resumed collection reflects current directory contents with
no stale entries. A *different* directory can never adopt (I7); it gets a fresh
distinct collection.
**Rejected alternatives (re-enable).** *Fresh collection always* — re-enabling
makes a new collection and the kept index becomes a permanent unused archive;
rejected because it guts keep-data's pause-and-resume value. *Re-adopt without
freshen* — reuses the kept index but leaves stale chunks for files deleted while
disabled, so search returns entries for files that no longer exist; rejected as a
silent correctness gap.
**Method note.** The removal lifecycle is the project's first use of the
model-first discipline for stateful-protocol work: model in Z, model-check the
invariants (a violation is a design bug found before any test), and adversarially
verify the data-deleting paths. A series of data-safety defects were caught before
merge that review-by-iteration had not: the I2 deferral asymmetry (model-check);
the sweep's two-SELECT read-skew, the retained cross-project merge, the pending
re-register purge race, the force-purge-vs-keep-data race, and the whole
name-picker auto-merge family — a different directory being auto-assigned a
chunk-bearing name — closed structurally by the I10 picker invariant (adversarial
review + model-check); and — most seriously — the **open-world sweep that would
have wiped all captures and agent memories** on a 5-minute default timer
(Cursor/Copilot, djb-confirmed). The last is the method's sharpest lesson, and a
caution against false confidence: the original model *proved the sweep safe by
omitting the non-directory collection class entirely* — a model is only as sound
as its abstraction. The corrected model
adds that class, and I9 (with a negative control) now pins it.
### DES-045b: Watch scope — refuse temp/scratch/VCS/gitignored roots
**Status:** Accepted (2026-07-26) · **Bead:** quarry-dpww · **PR:** #396
**Context.** DES-045 watches "the registered directories of *every* database in
the on-disk roster." Its trust invariant covered *how* a root is opened (only via
the roster; no network-steerable path), but placed no floor on *what* a root may
be. The operator's default registry held a `/private/tmp` registration (collection
`tmp`) — a legitimate roster entry by construction — so the loop watched the macOS
system temp directory, which every process on the box writes to continuously. The
watcher never reached quiescence: it re-indexed whatever landed in `/tmp`, pinning
1.5-2 cores for hours (283 CPU-minutes observed before the operator killed it).
DES-045's "bounded by construction" bounded fds, memory, and the write-serializer,
but a watch *root* that is itself churning scratch defeats every downstream bound
because the producer never stops.
**Decision.** One `ScratchGuard` predicate (`scratch_paths.py`), consulted at every
producer seam — initial scan (`FileDiscovery`), live watch
(`WatchLoop._begin_collection`), the periodic reconcile
(`WatchReconciler._sync_enumerated`), and explicit `quarry sync`
(`_submit_all_scans`) — refuses a root that is (a) an OS temp root (`/tmp`,
`/private/tmp`, `/var/tmp`, `/private/var/tmp`, `/var/folders`), compared casefolded
so a case variant on case-insensitive APFS (`/private/TMP`) cannot slip past, or
(b) a repo's own scratch, anchored on `<gitroot>/.tmp` for **any** ancestor git
repo (not the innermost `.git`, which a nested repo under `<outer>/.tmp` would
otherwise shadow); and, below a permitted root, skips gitignored paths plus an
always-skip set (`.git`, `.beads`, `.venv`, `node_modules`, `__pycache__`, `dist`,
`build`, caches). Resolution is fail-closed **per root** — a `resolve()` that raises
(symlink loop, permission) skips that one registration and logs, never aborting the
scan. The guard fires at *watch* time, not only at register time, so an
already-registered temp root — the operator's stale `/private/tmp` — is refused on
daemon restart without the registry being edited first.
**Consequence.** The watch loop can never index scratch/temp/VCS/gitignored content
on any surface, whether the temp root was registered before or after the guard
shipped. Verified on the live daemon: it logs `watch: skipping temp/scratch root
/private/tmp (systmp)` and stays idle where it previously stormed. The stale
registration is left in place (harmless — refused) and cleared by a one-time manual
step, not migration code (there is one user; a migration path protects users who do
not exist).
**Rejected alternatives.** *Killing the daemon on a CPU timer* (the first instinct)
— treats the symptom, discards the evidence, and would also kill a legitimate
catch-up index; a stack dump of the hot process is the diagnosis, a threshold is
not. *Deregistering `/private/tmp` and calling it fixed* — leaves the class open
(any future temp-root registration re-storms) and relies on remembering. *A bare
`.tmp`-segment match anywhere in the path* — false-refuses a legitimate checkout
that merely lives under a `.tmp` ancestor (an agent worktree at `<repo>/.tmp/wt`);
the git-anchored form is location-independent. *Trusting the observer's
hidden-segment reject alone* — it excludes dot-dirs below a root but does not refuse
a temp/scratch *root*, the actual hole.
### DES-045c: CPU/compaction envelope under continuous load
**Status:** Accepted (2026-07-26) · **Bead:** quarry-exz9 · **PR:** #397
**Context.** With DES-045b closing the scope hole, the daemon still spiked to
291-403% CPU. A stack sample of the live process — not a guess from a CPU number —
put the CPU in LanceDB's own `LanceDBBackgroundEventLoop` and tokio compute workers
running a finalize (`optimize()` plus a native `create_fts_index` rebuild).
LanceDB's Rust core sizes that tokio compute runtime to `num_cpus`, and DES-032's
caps never governed it (they bounded ONNX/OMP only), so a single finalize seized
every core. DES-045's "coalesced FTS" bounded *fds* (the quarry-0dss reader-eviction
leak) but not *CPU*. The trigger is the daemon's **own** periodic maintenance — the
finalize behind an index update, and the `watch_safety_scan_s` reconcile
re-finalizing collections — not developer activity. (An earlier account of this
work blamed `make check` churn; that was wrong — a `make check` writes only
gitignored/excluded paths and does not touch the index. The trigger is internal;
the defect is that the internal work was unbounded.)
**Decision.** Bound the CPU envelope structurally, independent of trigger.
*Cap LanceDB's pools:* `ThreadConfig.apply_env_limits()` (DES-032's home) sets
`LANCE_CPU_THREADS`/`LANCE_IO_THREADS` before the first `lancedb.connect` in
`DaemonServer.run()`, via a **fail-closed clamp with a per-variable floor** — an
inherited/operator value is honored only within `[floor, cap]`, a higher or
non-numeric value is forced down to the cap (a `setdefault` would yield upward to a
stale `LANCE_CPU_THREADS=32` and void the ceiling), and a below-floor value is
*raised* to the floor. The LANCE floor is 2 because `=1` stalls lance's runtime at
0% progress (floor equals ceiling at 2); OMP keeps floor 1. Measured (8-core): the
tokio/FTS pool tracks the cap, not `num_cpus` (33→14 workers as the cap goes 8→2),
so the ceiling holds core-count-independently; the FTS index is LanceDB native (no
separate uncapped Tantivy `IndexWriter` pool), so no search-engine change was
needed. *Throttle the finalize:* `FinalizeThrottle` coalesces the finalize to at
most one per `watch_optimize_min_interval_s` (default 30 s) **per database**,
covering the bulk path (a branch switch/rebase trips the bulk threshold and would
otherwise finalize every debounce window, defeating the cap), with a trailing timer
that is *cancelled*, not merely dropped, on the immediate path so a due timer cannot
double-fire. Freshness is database-granular; the vector channel is never stale
(files index immediately) and the reconcile is the unthrottled backstop.
**Consequence.** The daemon stays bounded under continuous load, not only when idle.
Trade-off (accepted): the cap is process-global, collapsing DES-032's query/sync
thread isolation onto one runtime — measured search latency under a concurrent
finalize is comparable-to-better than uncapped (the cap tames the finalize's own
flooding), and bulk compaction is ~1.65× slower (not the naive 4×; it is partly
IO-bound), acceptable for a background daemon whose ingest is not latency-critical
(DES-034 bounds the working set). The load-bearing verification is
install→restart→**sample the installed daemon under real load** — not the idle case
and not a green test suite: DES-045's original "bounded by construction" was
asserted from the idle path and held two holes (DES-045b and this entry) that only
surfaced when the running daemon was watched on a loaded machine. Read that claim as
"bounded where measured," not "bounded everywhere."
**Rejected alternatives.** *Blaming the developer's `make check`* (see Context) —
factually wrong and a category error; the daemon must bound its own maintenance
regardless of what runs alongside it. *Exempting the query connection from the cap*
— `LANCE_CPU_THREADS` is process-global, not per-connection, so it is not
expressible, and the latency measurement shows it is not warranted. *Throttling the
compaction cadence without the thread cap* — leaves a single finalize able to seize
every core; the cap is the ceiling, the throttle is the duty-cycle, and both are
needed. *Raising `narenas` to reduce contention instead of capping threads* —
addressed in DES-032; increases jemalloc retention (DES-027) and does not bound CPU.
### DES-045d: One ignore seam — watch, scan, and filter all consult IgnoreRules
**Status:** Accepted (2026-09-02) · **Beads:** quarry-0bej, quarry-ndrj
**Context.** The watch layer and the scan layer disagreed about what is ignored.
`FileDiscovery` pruned its bulk walk with full gitignore semantics (`pathspec`:
`.gitignore` at every level, `.quarryignore` at the root, scratch defaults,
hidden-dir skip), but the fs watchdog handed the raw registered root to
`observer.schedule(..., recursive=True)` — inotify allocated one watch descriptor
per directory, junk included. The operator's workspace root holds 206k
directories (117k of them accumulated pytest scratch under one repo's `.tmp`)
against the default 65,536-descriptor budget, so scheduling failed with ENOSPC
after consuming ~55k descriptors, the partial acquisition was never released,
and even 300-directory trees in the same daemon could not be watched. Ignore
filtering ran only on *events*, after the descriptors already existed — paying
the kernel cost for directories whose events would then be discarded.
**Decision.** Extract the ignore decision into one component, `IgnoreRules`
(`ignore_spec.py`), composed by `FileDiscovery`, and route every code path that
walks, watches, or filters registered-directory contents through it: the bulk
scan (`discover()`), the watch layer (`iter_watchable_dirs()` /
`is_watchable_dir()`), the post-debounce submitter filter, and file-level
matching (`keeps_file` — `FileDiscovery` never calls `pathspec` directly).
Compiled specs are cached; the observer-thread hot path never re-reads ignore
files per event. Paths that walk fixed, non-document directories (doctor's
existence stat, shadow rescrub's captures glob, backfill/ethos-memory/transcript
readers) are documented non-consumers, audited per mission m-2026-09-02-003.
**Consequence.** Watch-time pruning and scan-time pruning cannot drift: a
directory is ignored for descriptors, events, and bulk scans by the same rules,
matching git's own semantics (`.gitignore` + `.quarryignore`). The operator's
four registered roots prune from ~206k directories to ~1.2k watchable ones.
**Rejected alternatives.** *Raising `fs.inotify.max_user_watches`* — operator-
refused; treats the symptom and scales with junk accumulation rather than
content. *Event-level filtering only* (the status quo) — the kernel cost is paid
at watch establishment, so filtering events cannot protect the descriptor
budget. *A second pattern list for the watcher* — the exact drift bug class
(remote/local divergence, bug class 3) transplanted into the ignore layer.
### DES-045e: Watch mechanism — one pruned recursive inotify instance per root
**Status:** Accepted (2026-09-02) · **Beads:** quarry-0bej, quarry-ndrj
**Context.** The first implementation of DES-045d scheduled one *non-recursive*
`observer.schedule()` per surviving directory. Review rejected it: watchdog
gives every `ObservedWatch` its own emitter, and on Linux each emitter
constructs its own `Inotify` object calling `inotify_init()` — one kernel
*instance* plus two threads and three fds per directory, against
`fs.inotify.max_user_instances` = 128 (per-user, shared with IDEs). It
exhausted at ~120 directories — structurally worse than the 65,536-descriptor
problem it replaced. It also introduced a cross-thread watch-bookkeeping dict
with no lock (evaluator blocker) and aborted whole trees on ordinary mid-walk
ENOENT churn (evaluator major).
**Decision.** Operator-ratified: return to ONE recursive watch per registered
root — one emitter, one kernel instance, one thread-pair — and prune the
*descriptor* walk instead. On Linux, `PrunedInotify` (`daemon/inotify_prune.py`)
subclasses watchdog's private `Inotify`, overriding `_add_dir_watch` (the
initial recursive walk: prunes via the DES-045d seam; ordinary per-directory
`OSError` skips that directory, only `ENOSPC`/`EMFILE` aborts the tree) and
`_add_watch` (the auto-add primitive for directories created later: rejects an
ignored path via an `OSError` subclass every vanilla call site already
tolerates, with sentinel bookkeeping so a file directly inside a pruned
directory cannot `KeyError` watchdog's event simulation). A
`PrunedInotifyBuffer`/`Emitter`/`Observer` chain (`inotify_prune_chain.py`)
wires it into watchdog's construction pipeline. macOS keeps watchdog's standard
recursive FSEvents observer — one stream per root, no per-directory kernel
cost — with ignore filtering at the event/scan seam. Platform selection is a
module-scope import of the pruned chain; on failure the `PollingObserver` and
default recursive observer remain as fallbacks. Each collection's live watch
state (`watched`/`degraded`/`scan-only`) surfaces through `/registrations` and
`quarry list registrations`, so a silently degraded watch is observable.
**Consequence.** The daemon fits the operator's unraised kernel budgets: four
roots cost four inotify instances and ~1.2k descriptors. Watch bookkeeping
lives inside watchdog's own single-threaded buffer, eliminating the cross-
thread dict the first attempt needed. The cost accepted: `PrunedInotify`
depends on watchdog private internals, pinned by tests that fail loudly if the
subclassed surface moves.
**Rejected alternatives.** *Per-directory non-recursive scheduling* (the first
attempt) — one inotify instance per directory; dies at
`max_user_instances` = 128. *A bespoke ctypes inotify layer* — full control,
but reimplements rename cookies, queue overflow, and thread lifecycle — the
hairiest, most defect-prone part of the space watchdog already handles.
*Raising either sysctl* — operator-refused, and instance exhaustion is
per-user, shared with every editor and IDE on the box.
## DES-046: File-descriptor envelope — raise RLIMIT_NOFILE at daemon start
**Context.** The resident `quarryd` exhausted file descriptors over long uptime: an
18h41m run hit `OSError: [Errno 24] Too many open files` (EMFILE) 65 times across
~16h, with 305 tracebacks and cascading failures in `sync_ingest` (`Flush failed`),
`daemon.job_spool`, `shadow.sync`, and the fd-telemetry sampler itself; `quarry
doctor` showed ~9/256 fd headroom. Two facts compound. First, **nothing raises the
limit**: the daemon runs at whatever soft `RLIMIT_NOFILE` launchd/systemd hands it
(256 on macOS by `launchctl limit maxfiles`), and the fd measurers (`fd_headroom`,
`fd_telemetry`, and the doctor fd check — see the amendment) only *measured* it —
DES-032's `ThreadConfig`
enveloped CPU/threads but the fd side was never enveloped. Second, **the ceiling was
sized for two connections**: the per-connection reader-recycler (DES-023/quarry-0dss)
bounds *one* connection's descriptor growth, but post-DES-045 the daemon holds one
persistent LanceDB connection *per roster database* (`watch_roster`), so at a
~21-collection roster the aggregate of N bounded per-connection plateaus, plus the
ONNX/socket/watcher/sqlite baseline, sits above 256. This is not an unbounded leak —
it is a plateau summed over a roster that grows with uptime, against a limit meant
for two connections.
**Decision.** A new `FdEnvelope` (`fd_config.py`) — the fd sibling of DES-032's
`ThreadConfig` — raises the process soft `RLIMIT_NOFILE` toward a configured target
at daemon start, applied at the one process-resource seam in `DaemonServer.run()`
immediately before the first `lancedb.connect`. The target is a `Settings` field,
`QUARRY_FD_LIMIT` (default 8192), coerced fail-safe (a malformed or non-positive
override degrades to 8192 with a logged warning, never crashing construction). The
raise is fail-safe at every boundary (Bug-class-2): a missing `resource` module, a
raising `getrlimit`/`setrlimit`, or a target above the hard limit all degrade to a
single logged line and the inherited limit — the point is to survive, so a platform
quirk must never crash the daemon at start. It **clamps to the hard limit** and
**never lowers** an already-higher inherited soft limit, so an operator's systemd
`LimitNOFILE=65536` is honored, not clamped to 8192. The **reader-recycler is
unchanged**: the operator ruled prove-then-fix, and a daemon-scale plateau invariant
(N≈21 in-process connections × repeated optimize+FTS) is the arbiter — it shows the
working recycler holds the aggregate flat (growth 0.4, peak 243) while a
never-swapping recycler climbs (growth 189) and fails the guard, so accumulation is
already bounded and the fix is the higher ceiling, not a new mechanism. The stale
two-connection ceiling rationale in `connection.py` is re-derived to the correct
`(N+1)·recycle_after·fds_per_rebuild + baseline` aggregate (uncompressed — comments
are never a ratchet lever).
**Consequence.** The daemon always gets at least 8192 descriptors independent of what
launchd/systemd hands it — ~4–8× the bounded working set at 21 collections (~90
before nearing the limit), with `FdTelemetry` + doctor warning at 80% and the target
a one-line/env bump. Verified against *real* `setrlimit`, not just mocks: under a
256 soft limit the installed code raised 256→8192 (default), 256→65536
(`QUARRY_FD_LIMIT=65536`), and 256→8192 with a warning (`QUARRY_FD_LIMIT=nope`, no
crash); the live daemon, which currently inherits a 1,048,576 soft limit, correctly
no-ops (`unchanged … never lower`). The envelope is **daemon-only by construction** —
the CLI and stdio-MCP are short-lived thin clients holding no roster connections, so
every-surface-or-none is satisfied by explicit scoping. Sibling to DES-032
(CPU/thread envelope) and DES-023 (per-connection reader-pinning).
**Rejected alternatives.** *A fraction-of-hard target* — the hard limit is
`RLIM_INFINITY` on macOS launchd, so a fraction is undefined; a fixed target with
clamp-to-hard is well-defined on both platforms. *A fixed non-configurable constant*
— the operator ruled the budget configurable (`QUARRY_FD_LIMIT`) so a pathological
roster can be accommodated without a release. *Mandating a recycler mechanism change
(forced `gc.collect()` / lower `recycle_after`) up front* — speculative against the
evidence; the plateau invariant proves the recycler already bounds each connection,
so prove-then-fix keeps `connection.py` a comment re-derivation. *Setting
`LimitNOFILE` in the launchd plist / systemd unit only* — launchd's plist has no
clean soft-limit knob beyond the 256 default and it would not be cross-platform; the
in-process raise works identically on both and its never-lower floor still honors a
higher unit-file value where one is set. **(This rejection was wrong — see the
amendment below.)**
### DES-046 amendment (quarry-fnzh): the in-process raise is not enough on launchd, and doctor was measuring the wrong process
Two defects surfaced when 2.0.1 was smoke-tested on a fresh `quarry install`, and
both are fixed here.
**The in-process raise silently no-ops on a fresh launchd install.** DES-046's
`FdEnvelope` is correct in isolation, but a freshly *bootstrapped* launchd user
agent inherits a **hard** `RLIMIT_NOFILE` of 256 — and a non-root process cannot
raise its own hard limit. So `FdEnvelope` clamps the soft limit to the 256 hard and
lifts nothing; the daemon comes up exactly as descriptor-starved as before. (A
`launchctl kickstart` of an *already-loaded* service inherits the user domain's high
limit and masks this entirely, which is how the DES-046 demo was fooled into passing.)
The rejected-alternative above was therefore wrong on two counts: the launchd plist
*does* have a clean knob — `SoftResourceLimits`/`HardResourceLimits` with
`NumberOfFiles` — and the service manager is the *only* actor that can raise the hard
ceiling, because it sets limits before the process spawns.
**Fix:** `service.py` now bakes the ceiling into the service definition — launchd
`SoftResourceLimits`=`Settings.fd_limit` (default 8192, floored to `DEFAULT_FD_LIMIT`
so `QUARRY_FD_LIMIT` can only *raise*, never lower into the EMFILE range) and
`HardResourceLimits`=65536; systemd `LimitNOFILE=8192:65536`. `FdEnvelope` stays as
the cross-platform floor and its never-lower guarantee. `QUARRY_FD_LIMIT` is applied
at **install** time (baked into the service limits); changing it requires re-running
`quarry install` — it is deliberately *not* in the daemon's environment, so a running
daemon's `FdEnvelope` never reads a stale override.
**`quarry doctor`'s fd-headroom check measured the CLI, never the daemon** — this is
why the fd limit was unprovable for three rounds. `ResourceDiagnostics.fd_headroom`
sampled `getrlimit`/`/proc/self/fd` of *the process running doctor* (the short-lived
CLI, whose limit is just the invoking shell's `ulimit`). The "9/256" that drove this
round was the operator's fresh shell, not the daemon. **Fix (Bug-class-3):** the
daemon self-samples its own `FdHeadroom` in the `/health` handler (`fd: FdHealth` on
`HealthResponse`), and doctor reads it through the *same* TLS-pinned `/health` probe
`reachability()` uses — one reader, no drift, and no silent fallback to a local sample
when the daemon is unreachable (it reports a clear degraded state instead). The dead
CLI-local sampler `doctor_resources.py` is deleted (PL-PP-1: no dead second path).
**Verification** is on the real fresh-bootstrap path — `quarry install` (regenerates
the plist and re-spawns the daemon under it), then `quarry doctor` reads
`FD headroom: daemon <open>/<limit>` sourced from the daemon, e.g. `daemon 232/10496`
— the daemon's own number, not the shell's — with the plist carrying
`SoftResourceLimits 8192` / `HardResourceLimits 65536`. Sibling to DES-032 (the
service manager, not the process, owns the pre-spawn envelope where the process is
powerless to set it).
## DES-047: Test-suite resource governance — hermetic, bounded, single-process
**Context.** An 8-core machine whose normal load average is under 5 reached
roughly 200 while several repositories' test suites ran concurrently.
Measurement of quarry's suite contradicted most of the suspected cause, and
found one real defect. Full record: `docs/test-resource-governance-design.md`.
What was suspected and is false: there are no parallel pytest workers (no
`pytest-xdist`, `addopts` has no `-n`); the log lines read as model loads are
`ThreadConfig` INFO records from one unit test file constructing 29 of them; a
LanceDB tokio runtime is process-global (13 threads once, not 8 per connection).
One full run consumes 0.69 average cores over 446.7 s — it is I/O-bound.
What is true: a full run appended **7,034 lines** to
`~/.punt-labs/quarry/logs/quarry.log`, roughly one 5 MB rotation every seven or
eight runs, discarding real daemon history each time.
**Decision 1 — the hermeticity contract.** A suite run writes nothing under the
operator's `~/.punt-labs/quarry/`. `HOME` is redirected to a session temporary
directory before anything imports quarry, with `QUARRY_ROOT` and
`QUARRY_LOG_DIR` set explicitly alongside it. This required a source change:
`LoggingConfig` pinned its destination in a class constant evaluated from
`Path.home()` at import time, so no fixture could move it; `configure()` now
resolves the directory per call, and `Settings` derives its config file from
`quarry_root`. That is target behavior, not a migration shim — the log
destination is configuration and was hardcoded.
Setting the environment variable rather than patching `Path.home` is required,
not stylistic: `os.path.expanduser` reads `$HOME` and ignores the patched
classmethod, so a `Path.home`-only patch leaves two of the three resolution
routes on production.
A per-test guard stats one *file* — `config.toml` — and fails the test that
moves it. A file, not a directory. A directory's
`mtime` changes when an entry is added to that directory and not when a file
below it is written, so a directory stat detects almost nothing; the obvious
repair, a recursive walk, is unavailable because the operator's tree is 15 GB
across ~1,600 files and `os.walk` with a stat per file did not finish inside two
minutes. The guard is a smoke check that the redirect is in force, not a tree
fingerprint. Do not reinstate the fingerprint.
The watch list is chosen for silence, not importance. `quarry.log` — the very
file whose leak opened this entry — is not on it, and neither is `registry.db`.
Each sits behind two independent redirects (`QUARRY_LOG_DIR` and `QUARRY_ROOT`,
with `HOME` behind both), so reaching either from a test requires an absolute
path written into source and the true-positive rate is about zero. The
false-positive rate is not: the daemon and every live MCP client write the log
continuously, and the DES-045 watch loop writes a registry row whenever anything
under a registered tree changes — which includes this repository while an agent
edits it during a run. Because the guard watches files rather than writers, both
reported other processes against whichever test was in flight. A check that
cannot catch what it exists for, and fires on unrelated processes, costs more
than it defends. `config.toml` is quiescent, so a firing there is likely to mean
something.
**Decision 2 — no `pytest-xdist`, ever.** The suite is dominated by filesystem
and LanceDB I/O. Workers would multiply the 180 MB base footprint and the tokio
runtime by the worker count for a modest wall-clock gain, and would create
exactly the per-worker engine multiplication the incident was wrongly believed
to involve. There is no CPU idleness for extra workers to fill. The correct
response to a slow suite here is to fix the slow tests.
**Decision 3 — no workspace-level test-concurrency mechanism.** Per-run caps
multiply by N and the incident was caused by N; an earlier draft proposed an
advisory slot lock at `~/.punt-labs/test-concurrency.lock` acquired by `make
test` and adopted by sibling repositories. Rejected, and the reasoning is
recorded because the temptation will recur. The lock path and the premise that
siblings cooperate are facts about how punt-labs is checked out, not about
quarry — shipping them ships our desk to every other contributor and every CI
runner, who would pay startup cost and a new failure mode for a contention they
do not have. How many suites run at once is a scheduling decision belonging to
whoever runs the machine. The work is rejected, not deferred: there is no
follow-on bead, no `Makefile` lock, and no `tools/test_lock.py`.
The suite's obligation is stated in terms of itself: **one quarry suite run is
bounded and hermetic on any machine, and assumes nothing about what else that
machine is running.** Per-run enforcement alone meets it.
**Enforcement.** Processes 1; OS threads ≤ 32 (`OMP_NUM_THREADS`,
`LANCE_CPU_THREADS`, `LANCE_IO_THREADS` pinned before lance builds its compute
runtime, which reads them once and ignores later assignment); real ONNX model
loads 0, by a fake installed at the `quarry.ingestion.backends` factory plus an
`InferenceSession` guard that fails an unmarked load, replacing ~40 ad-hoc
patches that each named one import site; writes under the real tree 0. A
session-scoped invariant asserts no non-daemon thread outlived the run —
session-scoped because LanceDB starts its pool lazily, so a per-test check would
fail on correct code.
**Amendment — network hermeticity covers subprocesses, not just this process.**
The original enforcement faked DNS by patching `socket.getaddrinfo`, which binds
only the interpreter running the tests. Anything the suite shells out to
resolves and connects on its own: the shadow tests' `git fetch` against a
deliberately unreachable remote spawned a real `ssh` on every run, passing only
because the lookup usually failed fast, and hanging for 30 seconds per call the
day it did not. The class is *subprocess network access*, of which git-over-ssh
is today's only instance; the pin is stated at that level — `GIT_ALLOW_PROTOCOL`
restricts git to local transports so any remote protocol is refused inside git
before a resolver or socket is reached, and `GIT_TERMINAL_PROMPT=0` closes the
other way a subprocess wedges, blocking on a credential prompt with no terminal.
Both are faithful rather than permissive: the operation still fails, which is
what those tests already assert, and it fails locally and instantly. A future
subprocess that can reach the network must be pinned the same way; a fake that
lives inside this interpreter will not cover it.
Sibling to DES-045 (the watch loop whose scratch guard constrains where a test's
project root may live) and DES-046 (the daemon's descriptor envelope).
## DES-048: Daemon operational logging — configured, separate, volume-governed
**Context.** For its entire life the daemon ran with no logging configuration
at all: `LoggingConfig.configure` was called only by the CLI's Typer root
callback and the MCP client tier, never on the `quarryd` path, so the root
logger had zero handlers and Python's `logging.lastResort` applied — INFO
discarded before any handler, warnings reaching launchd's stderr file stripped
of timestamp, level, and logger name. Three incidents in one week traced back
to this blindness: the load-200 post-mortem read a log that was mostly test
contamination, the ingest-churn fix could not observe its own
six-finalizes-per-sweep fingerprint at the demo gate, and a dead
`ModuleNotFoundError` traceback in the timestampless stderr file was mistaken
for a live outage.
**Decision.** The daemon configures logging at startup and writes a separate
`quarryd.log` beside `quarry.log`, same directory resolution (`QUARRY_LOG_DIR`,
falling back to the home path), same format, same rotation policy. Separate,
not shared: one long-lived writer versus many short-lived ones — interleaving
is what made attribution impossible. Uncaught exceptions reach the file through
three hooks (main-thread excepthook, `threading.excepthook`, and the event
loop's exception handler), each delegating onward so supervisor stderr remains
the backstop for a crash before logging is up.
**Volume policy (standing rule).** INFO is one line per user-visible operation
or coarser — a document indexed, a collection swept, a database optimized, the
daemon started; sub-operation detail is DEBUG. The rule exists because turning
on a file nobody had ever seen would have replaced silence with a firehose: the
per-flush chunk insert and the per-request search count were demoted to DEBUG
under it. The binding statement lives in `quarry.logging_config`'s module
docstring, where the next log line gets written; call sites cite it rather than
restating it.
**Invariant.** `quarryd.log` must never join the test suite's
`ProductionTreeGuard`: the live daemon writes it continuously, so watching it
reports other processes rather than breaches — the exact false-positive
mechanism DES-047's amendment removed the log and registry watches for.
Sibling to DES-047 (whose hermeticity redirects contain daemon-path logging in
tests) and DES-032/DES-045 (whose sweep and optimize operations are the INFO
lines the file exists to carry).
## DES-049: No database-size reporting — the walk was an outage, the cheap number is wrong
**Context.** The daemon's `/v1/databases` and `/status` routes, the CLI's
databases table and `Size:` line, and doctor's Storage check all reported
on-disk size by walking the data tree (`dir_size_bytes`). Measured at 10–19
seconds per request on a multi-GB store, payable by any authenticated client
and stacking across clients — and under load the daemon could not answer
inside the client's 15-second read timeout, turning a status query into a
client-triggerable outage.
**Decision.** Quarry reports no storage size on any surface. The walk and its
helpers (`dir_size_bytes`, `format_size`, `_fmt_size`, `DatabaseSummary`, the
already-dead `discover_databases`) are deleted, not cached and not flagged.
Operators who want the number ask the filesystem (`du -sh "${QUARRY_ROOT:-$HOME/.punt-labs/quarry/data}"`) and pay the walk knowingly.
**Rejected: LanceDB `table.stats().total_bytes` as a cheap replacement.** It
is genuinely O(1) (~0.3–13 ms) and it is the obvious future proposal — which
is why the rejection is recorded with its numbers. `stats()` measures the
**live dataset**, not the directory: on a freshly optimized 1.13 GB store it
under-reported the on-disk footprint by 43%; on a grown 3.5 GB store by 81%.
The gap's composition varies (superseded data-file versions, index files), so
no caller can correct for it — and it understates worst exactly when
optimize/FTS work has inflated the tree. A varying shortfall published as
"size" is a wrong number served fast. Do not resurrect the field on top of it.
Sibling to DES-032/DES-045 (whose optimize and sweep work is what grows the
directory beyond the live dataset) and DES-046 (the same file-descriptor
class of daemon-resource protection).
---
## DES-050: The shippable plugin surface lives in `plugin/` (git-subdir install)
**Context.** The marketplace entry for quarry used a `github` source, so
`claude plugin install quarry@punt-labs` cloned the **whole repository** into
`~/.claude/plugins/cache/punt-labs/quarry/<version>/`. A user who wanted six
slash commands and five hook scripts received `src/`, `tests/`, `tools/`,
`benchmarks/`, `docs/`, `prfaq.pdf`, the beads config, and the LaTeX sources —
tens of megabytes of material that is not the plugin and that no consumer can
act on. The clone also ran `--recurse-submodules`, which is why the org
identity registry had to be removed from this repo before it could be a
well-behaved plugin at all.
**Decision.** The four directories that constitute the plugin —
`.claude-plugin/`, `commands/`, `hooks/`, `agents/` — move under a single
top-level `plugin/` directory, and the marketplace entry switches to the
`git-subdir` source type with `"path": "plugin"`. A client then fetches that
subtree alone. `plugin/` is the plugin root: it is the directory Claude Code
names in `${CLAUDE_PLUGIN_ROOT}`, the directory `claude --plugin-dir` must be
pointed at, and the directory `installPath` refers to in
`installed_plugins.json`. Every path *inside* the surface is unchanged, so
`hooks.json`'s `${CLAUDE_PLUGIN_ROOT}/hooks/*.sh` entries and
`session-start.sh`'s `dirname $0/..` derivation stay correct without edits;
what breaks — and what this change fixes — is every reference from *outside*
the surface that named it by a repo-root path. See
`punt-kit/standards/plugins.md` for the org-wide layout this conforms to.
**The `agents/` directory moves to where it is actually read.** The researcher
agent (DES-013) sat at `.claude-plugin/agents/researcher.md` and was never
loaded by any session. Claude Code resolves a plugin's default agent directory
as `join(pluginRoot, "agents")` — a sibling of `.claude-plugin/`, not a child —
and quarry's `plugin.json` declares no `agents` override, so the file was dead
on arrival. It now lives at `plugin/agents/researcher.md`. Agent types are
namespaced per plugin (`quarry:researcher`), so this does not collide with the
`researcher` that prfaq ships.
**Cone-mode caveat, stated so nobody re-measures it as a defect.** A sparse
checkout in cone mode always materializes the repo-root *files* alongside the
requested directory. Fetching `plugin/` therefore also lands `README.md`,
`CLAUDE.md`, `DESIGN.md`, `prfaq.pdf`, and the other root files. The win comes
entirely from the directories left behind, which for quarry is the great
majority of the tree; a repo whose bulk sat in root files would gain little.
**Rejected: a separate publish-only repository.** Mirroring the plugin surface
into `punt-labs/quarry-plugin` on release would give a byte-exact minimal
clone, but it adds a second repository to keep in sync, a release step that can
silently skip, and a second review surface — to save a subtree that
`git-subdir` already excludes. Rejected as more machinery than the problem.
**Rejected: keeping the surface at the repo root and filtering at install
time.** There is no marketplace mechanism for it. `git-subdir` *is* the
mechanism, and it requires the content to be in a subdirectory. It also raises
the client floor to Claude Code ≥ v2.1.69, which is accepted as the punt-labs
baseline.
---
## DES-051: Homebrew distribution — Apple Silicon + Linux only, hand-pinned wheels for the sdist-hostile dependencies
**Context.** `Formula/quarry.rb` in `punt-labs/homebrew-tap` was an
unfinished scaffold (placeholder sha256, no resource blocks). Finishing it —
the second real-world test of `punt-kit/patterns/homebrew-pypi-formula.md`'s
recipe, after `punt-biff` — surfaced two problems the recipe didn't
anticipate, both stemming from Homebrew's `Language::Python::Virtualenv`
model: every dependency is built from its PyPI **sdist** inside the formula's
own sandboxed venv, with no `pip install <wheel>` fallback for anything not
explicitly pinned.
`lancedb` and `onnxruntime` — quarry's vector-index and embedding-inference
engines — publish **no sdist at all** on PyPI; they ship prebuilt wheels only,
and at the versions quarry pins, neither publishes a **macOS x86_64 (Intel)**
wheel. This is not a Homebrew-specific gap — `uv tool install`/`pip install`
hits the identical missing-wheel failure on Intel macOS, since pip has no
sdist to fall back to either. Homebrew's build-from-source model is simply
what surfaced it first. `pymupdf` does publish an sdist, but building it
means compiling a
full MuPDF + Tesseract + a dozen vendored C libraries from source — one of
the heaviest builds in the Python ecosystem, and not a path PyMuPDF's own
release process exercises. Two tree-sitter grammar packages
(`tree-sitter-c-sharp`, `tree-sitter-yaml`) build from sdist without error but
produce broken extensions: the build compiles `parser.c` but not the
grammar's external `scanner.c`, so the `.so` dlopens fine and crashes on the
first real parse — a defect only `quarry doctor` driven against the installed
binary caught, not `brew test` or the build step itself.
**Decision.** Two decisions, made together because the first caused the
second:
1. **Platform scope is Apple Silicon + Linux (both arches), not universal
macOS.** `on_macos { depends_on arch: :arm64 }` refuses Intel outright
rather than silently shipping a formula that fails to resolve
`lancedb`/`onnxruntime` on that one platform. Operator-ruled 2026-08-23
after presenting the fork (arm64-only / pin older deps / hand-roll wheel
install / drop Homebrew for quarry).
2. **Five dependencies are hand-pinned `resource` blocks with per-platform
(`on_macos` / `on_linux { on_arm / on_intel }`) wheel URLs and shas**,
pip-installed directly against the wheel file rather than run through
`virtualenv_install_with_resources`'s automatic sdist path: `lancedb`,
`onnxruntime`, `pymupdf`, `tree-sitter-c-sharp`, `tree-sitter-yaml`. The
other ~85 dependencies still regenerate mechanically via
`brew update-python-resources --ignore-main-package-cooldown
--ignore-errors` on every release bump; only these five need
hand-maintenance, and only because their upstream distribution or sdist
build is broken in a way no `depends_on` line can fix.
**`--ignore-errors`, not `--exclude-packages`, for the mechanical pass.** The
first attempt used `--exclude-packages=lancedb,onnxruntime,...` to let the
generator skip the five broken packages and still auto-generate the rest.
It silently cascade-dropped unrelated dependencies (`pyarrow`, `numpy`,
`pydantic`, `protobuf`, `typing-extensions`) whose *first* discovery path in
the resolver's graph traversal happened to run through an excluded package,
even though they're independently required. `--ignore-errors` instead
records every package it *can* resolve and only flags the genuinely
unresolvable ones with a `RESOURCE-ERROR` comment — the five above, and
nothing else. Silent scope-narrowing from a flag with a plausible-sounding
name is the failure mode to watch for here if this formula is touched again.
**Also fixed, not a design decision but recorded so it isn't rediscovered:**
the formula's `service do` block invoked `quarry serve --port 8420`, a
subcommand quarry has never had — the daemon is the separate `quarryd`
console-script entry point declared in `pyproject.toml`. And `tokenizers`
(pyo3 + tokio) didn't pick up the `-undefined,dynamic_lookup` linker flag
other pyo3 crates in the same build got automatically, requiring an explicit
`RUSTFLAGS` override on macOS only.
**Rejected: pin `lancedb`/`onnxruntime` to older releases with broader wheel
coverage.** Would diverge the Homebrew install from the version quarry
actually ships and tests against, and only defers the problem to the next
release bump.
**Rejected: hand-roll wheel installation for the whole formula**, abandoning
`virtualenv_install_with_resources` entirely. Throws away the mechanical
85-package regeneration path for a problem that only affects five packages.
**Rejected: drop Homebrew distribution for quarry.** Considered and rejected
by the operator — ML/Rust-heavy dependency trees don't fit
`Language::Python::Virtualenv` as cleanly as a pure-Python CLI like `biff`,
but the formula does build and pass its own tests plus a live `quarry doctor`
run against the installed binary; the extra hand-pinned resources are a
maintenance cost, not a blocker.
See `punt-labs/homebrew-tap` PR #34 and `punt-kit/patterns/homebrew-pypi-
formula.md`'s "Known failure mode" section for the full technical detail.
## DES-052: `POST /v1/captures/lookup` takes `cwd`, not a pre-resolved collection name
**Context.** The WebFetch loop-closer needs to ask the daemon "have I already
captured this URL?" before re-sending it. The ticket's illustrative route
shape was `/v1/captures/lookup?url=<url>&collection=<repo>-captures` — the
caller passing an already-suffixed captures collection name.
**Decision.** The route is `POST /v1/captures/lookup`, taking `cwd` in the
JSON body, not `collection`, mirroring `POST /capture`'s existing contract
exactly: the daemon derives the target
`<repo>-captures` collection server-side via
`CapturesCollection.for_registry_path(cwd, registry_path)`. `CaptureUrl.for_web_fetch`
also moved to run server-side, inside the route, rather than being computed by
the caller and passed as an opaque pre-normalized string.
**Why.** `CapturesCollection.for_registry_path`'s own docstring states the
constraint plainly: "the capture client cannot do this itself without
importing the engine" — resolving a directory to its registered collection
name requires opening the sync registry, and hooks are deliberately
engine-free (`TestHookImportsNoEngine` runs the capture paths with `lancedb`
and `onnxruntime` poisoned). A hook that had to spell `<repo>-captures` itself
would need either a registry import (breaking engine-free) or a naming
heuristic that can drift from what the daemon actually registered — exactly
the bug-class-3 local/remote divergence this repo's review history recurs on.
Passing raw `cwd` (as `POST /capture` already does) keeps the hook a pure
thin-client caller and the daemon the single owner of the naming rule.
**Rejected: client passes the literal `<repo>-captures` name.** Would require
the hook to either import `quarry.sync_registry` (violating the engine-free
hook boundary) or guess the name from `cwd` with a copy of the daemon's own
derivation logic — a second implementation of `CapturesCollection.for_cwd`
that could silently diverge from the one the daemon uses at write time.
## DES-053: `quarry learn` — a fourth capture verb with retrieval preference
**Context.** `remember` and `ingest` cover a durable fact and a URL, but
neither gives a caller a way to say "this rule should outrank ordinary
results" — the only lever is `memory_type`, which is descriptive metadata,
not a retrieval signal. The design considered and rejected a two-call shape
(`learn` then `set_config`): a crash between the two calls leaves a lesson
that never surfaces, and the shape freezes solid the moment a client exists.
Operator-ratified 2026-08-31 (three constraints, `docs/design/quarry-b6p-
quarry-learn.md`): **C1** atomic operation (one call writes the chunk and
registers the boost, on every surface); **C2** same verb/shape everywhere
(`learn` on CLI/MCP/slash/client, lesson text positional, `topic`/`name`
trailing options); **C3** slash is a thin door (`/quarry:learn` parses and
calls the MCP tool, no logic in the `.md`).
**Decision.**
1. **Collection: project-scoped `<repo>-lessons`, not agent-scoped.** A new
`LessonsCollection` (structurally identical to `CapturesCollection`) maps a
lesson's `cwd` to `<repo>-lessons` or `default-lessons`. The ratified wire
shape (`{lesson, topic?, name?}`) has no `agent_handle` field on any
surface — a distilled lesson is project knowledge, not personal memory.
2. **Retrieval preference is a fusion-time rank multiplier, not a second
write.** `RrfFusion` gains `lesson_boost` (default `1.5`, threaded from
`Settings.retrieval_lesson_boost`, `ge=1.0`): a row's RRF term is
multiplied by `lesson_boost` when `memory_type == "lesson"`. This is why
C1's atomicity is trivial — "retrieval preference" is a derived property of
the one field (`memory_type`) the single write already sets, not a second
row. A multiplier (not a hard filter/promotion) means an irrelevant lesson
still loses to a relevant plain result; `1.5` is derived from RRF's
`1/(k+rank)` term so a lesson ranked in the top ~30 of its own channel
outranks even the single best non-lesson hit, while rank 40+ still loses.
3. **Lessons are exempt from decay for free.** `learn` never sets
`agent_handle`, so every lesson row already fails the existing
`RrfFusion` decay gate (`decay_rate > 0 AND memory_type in _DECAYABLE_TYPES
AND agent_handle`) — no new gate needed. `_DECAYABLE_TYPES` deliberately
excludes `"lesson"` as belt-and-suspenders documentation against a future
change that adds `agent_handle` to `learn`'s shape.
4. **Document naming is always daemon-generated, never the caller's literal
name.** `LessonComposer.document_name` returns `lesson-<slug>-<8 hex>` —
the slug comes from `name`, else `topic`, else `"note"`. Forced by
`RrfFusion`'s dedup key `(document_name, chunk_index, page_number)`: two
lessons sharing a literal name would collide at `chunk_index=0,
page_number=0` (the common case for a short lesson), merging scores and
silently dropping one lesson's text. `learn`'s whole premise — quick,
low-ceremony capture — makes name reuse the expected case, so the
uniqueness guarantee is unconditional, not caller-opt-in; `overwrite` is
therefore never exposed as a `learn` parameter.
5. **`memory_type == "lesson"` is reserved.** `remember`/`ingest` both reject
a caller-supplied `memory_type: "lesson"` with `400` — without the guard, a
caller could get the retrieval boost while bypassing `learn`'s naming,
topic, and length-cap rules, defeating C1's atomicity guarantee.
6. **Length cap: 500 characters, server-side only.** `learn` rejects a lesson
over 500 characters, pointing the caller at `remember`. "Distilled" is the
operative word in the boundary sentence; without an enforced boundary,
`learn` would accept the same content `remember` does.
7. **`quarry doctor`'s memory-corpus check gains a handle-independent lesson
count.** The existing `_tally` excluded every empty-`agent_handle` row from
its type tally — meaning every lesson (empty handle by design) was
invisible to the one diagnostic built to answer "how much distilled
knowledge exists here." `_tally`'s counters were extracted into a
`_CorpusTally` accumulator (`add(row)` per row) so the lesson count is a
fourth, independently-gated field rather than a fifth tuple position.
8. **The boundary sentence propagates to `remember`/`ingest`, not just the
newcomer.** "remember = a specific durable fact, ingest = a URL, learn = a
distilled lesson that gets retrieval preference" appears verbatim in the
CLI, MCP, and slash-command descriptions of all three verbs — otherwise the
boundary the sentence exists to draw only holds for two of three verbs.
**One resolved deviation from C2's literal text.** C2's ratified prose spells
the client signature `topic=None, name=None`; the implementation uses
`topic: str = "", name: str = ""`. No `Optional`/`| None` field exists
anywhere in `quarry.api` today (`RememberRequest`, `IngestRequest` both use
the empty-string sentinel), and `None` here would be the only exception for no
different reason than the two adjacent methods that already use `""`. Read as
C2 specifying the *shape* (three params, one positional, two optional) and
implementing the *type* per the codebase's settled convention.
**Rejected: boost scoped by query-supplied `topic`.** Would require a new
`topic` filter on `find`/`SearchRequest`/`SearchFilter` — a much larger
surface-parity change (bug class 3) than a v1 retrieval-preference feature
justifies. A real, data-backed need for topic-scoped boosting is a separate
follow-on, not a speculative addition here.
**Rejected: agent-scoped lessons (route via `agent_handle` like
`remember`).** Would require smuggling an identity parameter onto a wire
shape (C2) that has no field for it on any of the four surfaces.
## DES-054: Vendored, locally-optimized ethos registry — pruned 8-identity copy, quarry team pin
**Decision.** Quarry commits a vendored identity registry at
`.punt-labs/ethos/` (org standard, the `lux`/`cryptd` pattern): the
8-member `quarry` team from the CLAUDE.md pairing tables (jfreeman,
claude, rmh, gvr, kpz, djb, mdm, adb), their personalities, writing
styles, talents, and roles, and `teams/quarry.yaml` only.
`.punt-labs/ethos.yaml` pins `agent: claude`, `team: quarry`, and
`resolution: repo-only` (identity resolution never falls back to the
global store; `ethos doctor` verifies the 8 identities resolve
repo-locally). The canonical registry (`punt-labs/team`) gained `teams/quarry.yaml`
claiming `punt-labs/quarry`, and `teams/engineering.yaml` stopped
claiming it, so exactly one team claims the repo. Runtime state
(`missions/`, `missions.jsonl`, `sessions/`, `.biff`) stays gitignored;
everything else under the path is tracked. There is deliberately no
`.vendor.yaml`: the copy is produced by `ethos vendor <8 seeds> --apply`
followed by a prune back to the quarry-team closure, because the tool's
closure is membership-connected — the 8 seeds belong to `engineering`,
so an unpruned vendor always plans the full 29-identity
engineering-connected roster
regardless of `repositories:` claims (verified empirically; `--team
quarry` seeding plans the same 29).
**Why.** Identity resolution previously fell back to the global
`~/.punt-labs/ethos/`, so a bare clone could not resolve identities, and
SessionStart injected the full 29-identity engineering roster (~155 KB
of persona and team context, measured on the injecting hook's output)
into every session. The `team: quarry` pin bounds the
injected team context to the specialists quarry actually delegates to;
the pruned copy makes the repo self-standing and keeps the marketplace
plugin clone lean.
**Rejected: unpruned `ethos vendor` snapshot (the `../ethos` repo
shape).** Tool-verifiable via `.vendor.yaml`, but carries the 29-identity
engineering-connected roster in every plugin clone — the payload bloat this change exists
to avoid — and its `.vendor.yaml` would be a lie the moment the copy is
pruned, so the manifest is dropped along with the excess identities.
**Rejected: `punt-labs/team` submodule.** Claude Code clones plugin
repos with `--recurse-submodules` (ships the whole roster to consumers),
and `ethos enable` v4.15.0+ refuses submodule mounts (`ethos-e29s`).
**Rejected: repo-local-only team file (lux's original shape).** A
`teams/quarry.yaml` existing only in the vendored copy leaves the org
registry unaware the team exists and leaves `engineering` claiming the
repo — two sources of truth for repo-to-team mapping. Canonical-first
matches the org's "edit `punt-labs/team` first, then sync" rule.