AGENTS.md · git:20261023.cff28ac · 2026-10-23 · sha256 63289debc72d8f75
AGENTS.md git:20261023.cff28acA
Immutable. This exact content never changes and is served at /api/v1/blob/63289debc72d8f75.
# AGENTS.md
Guidance for coding agents working in this repository. This is the single source
of truth; `CLAUDE.md` points here.
## Build and test
Rust 1.90+, edition 2024, 64-bit only (turbovec refuses 32-bit). Nothing has to
be installed first: turbovec 1.0.0 dropped BLAS, so there is no OpenBLAS step on
Linux any more, and the TLS stack is rustls rather than the system OpenSSL. The
packaged Linux binary must keep needing nothing but glibc and libstdc++ —
`release.yml` reads its `NEEDED` entries and fails on `libssl`, `libcrypto` or
`libopenblas`. First build compiles bundled SQLite and downloads ONNX Runtime —
minutes.
```sh
cargo fmt --all -- --check
cargo clippy --all-targets -- -D warnings
cargo test # offline default set — this is what CI runs
cargo test --test endpoint # the HTTP MCP endpoint; no model, so it is in that set
cargo build --release
```
Most integration tests are `#[ignore]`d because they download the ~52 MB
embedding model on first run:
```sh
cargo test -- --ignored # everything, incl. round trip
cargo test --test mcp -- --ignored # one file
cargo test --test mcp -- --ignored some_test_name # one test
cargo test --test image -- --ignored # images; downloads CLIP as well
cargo test --release --test measure -- --ignored --nocapture # perf claims; needs --release
cargo test --release --test shards -- --ignored --nocapture
cargo test --release --test retrieval -- --ignored --nocapture # retrieval quality
```
`tests/retrieval.rs` is the harness behind every retrieval claim this repository
makes. It runs a fixed set of 107 questions with ground-truth spans from
`tests/fixtures/retrieval/questions.yaml` — identifier-shaped, concept-shaped and
multi-hop — and prints hit@1, hit@3, hit@8, bytes per answer, and from 0.23.0 calls and tokens per answered question for
`brief` against the search-then-read path. It asserts two things: the wrong-yes
count for `path` is zero, and `tools/list` costs under 675 tokens (`mcp::tool_list_tokens`, bytes over four — the figure the Agents page prints too) — from 0.36.0
eight listed tools (`mcp::LISTED`; clients offer agents only listed tools,
so the other eight are the CLI's and portal's unless `SEMLITH_MCP_TOOLS=all`), 2 633 bytes, where sixteen were 7 330 and
cost an Opus request 3 073 tokens against 1 323 now, measured; every array
parameter carries an `items` type, which Copilot in VS Code requires. The
byte proxy in `mcp::tests::the_tool_list_stays_small` is the same number said
in bytes, so a list that would fail the criterion fails in seconds rather than
eight minutes into an indexing run.
The questions are split development/sealed by a recorded seed in
`tests/fixtures/retrieval/split.yaml`, and the sealed set is scored only when
`SEMLITH_RETRIEVAL_SEALED` is set. Development work sees the development set and
nothing else; a number tuned against is not a measurement. Change ranking, chunking or the
tool schemas and run it; a number moving is the report, and an adjective is not.
The question set's spans are line ranges in the 0.14.0 merge, and the `symbol`
field is the durable anchor when one moves — the file's header says how to
re-derive it.
`tests/endpoint.rs` is the exception in the other direction: it drives the
daemon's `/mcp` route and its credential, neither of which needs an embedding
model, so it runs in the default set and a break in the agent key's reach fails
CI rather than an `--ignored` run somebody remembered to do.
`tests/measure.rs` and `tests/shards.rs` are `#![cfg(unix)]` and assert real
thresholds — run them in `--release` or the numbers fail honestly.
Run `cargo test -- --ignored` (the `tests/roundtrip.rs` cycle) after touching
`lib.rs`, `store.rs`, or `chunk.rs`. It is the only test that catches chunk ids
in the vector index drifting out of sync with their rows in SQLite.
Document fixtures under `tests/fixtures/` are generated by real libraries via
`tests/fixtures/generate.py`, not hand-written XML. Regenerate rather than edit.
## Architecture
A store directory holds two pieces of state that must agree:
- **`index/`** — turbovec (TurboQuant) shards of 65536 quantized vectors, each
file named for the first chunk id it holds, zero-padded so sorted-by-name is
sorted-by-id. **The directory listing *is* the manifest** — nothing else
records shard boundaries, so nothing else can disagree with it. Format 1
stores (pre-0.7.0) keep a single `index.tv` and are never migrated.
- **`store.db`** — SQLite: chunk text, file path, line span, content hash.
- **`exact.f32`** — from 0.23.0, the full-precision copy of what the codes
approximate. Fixed-size `[id][dim f32]` records appended in id order, read back
by binary search for the handful of candidates a query is about to reorder. A
store written before 0.23.0 has none and is ranked by the codes alone, which
`semlith stats` says out loud. Only `compact` rewrites it (0.31.0), to the
records whose chunk still exists.
A store that has indexed an image holds a third, `images/`, in the same shard
layout at CLIP's 512 dimensions. It is opened lazily, so a store that never
meets an image never grows the directory.
`chunks.id` is `AUTOINCREMENT` specifically so SQLite cannot reissue a deleted
row's id — a reused id would fall inside two shards' ranges at once. Do not
change that.
Search is hybrid: the query embedding goes to the vector index and the bare
terms go to FTS5, each searched 32 deep (`FUSE_DEPTH`) whatever `k` is, then
fused by reciprocal rank (`1 / (60 + rank)`) and cut to `k`. Scores are fusion
scores, not cosines — meaningful for ordering within one result set,
meaningless across queries. Query text reaches FTS5 as bare terms, never as
typed, because `MATCH` is a query language. **`k` cuts the answer and never
changes it (0.42.0).** A depth of `4 * k` scored and ordered the first results
of k=5 differently from k=8 (audit G12); do not make the depth a function of
`k` again.
`index.rs` reads nothing from disk until a caller asks something only vectors
can answer, so `stats`, `files`, and an idle MCP server cost no vectors. Peak
RSS is ~600 MB regardless of corpus size; keep it that way. A store that holds
images loads CLIP's two encoders on top of that, on the first image it indexes
or is searched for — measured per store rather than folded into the number
above, which is about the text path.
Module responsibilities:
| File | Responsibility |
|---|---|
| `src/lib.rs` | `Semlith`: open, index, search, persist |
| `src/brief.rs` | `semlith brief` / `semlith_brief`: the ranked spans, their text and one-hop edges, composed from the primitives and fitted to a token budget. No engine of its own |
| `src/store.rs` | Every SQL statement. Nothing else touches the database |
| `src/index.rs` | Vector side: `Single` vs `Sharded` layouts, memory budget |
| `src/compact.rs` | `Semlith::compact` and `Semlith::footprint`: the sidecar rewritten to live records, shards packed from it and swapped in, history past the retention dropped, the database vacuumed; the one call the CLI, the daemon, the portal and Semlith Cloud make |
| `src/chunk.rs` | File bytes → text → chunks. Cut at definitions where tree-sitter found them and at headings in Markdown; 800 chars and 2 overlap lines are the fallback and the budget; 8 MiB cap |
| `src/formats.rs` | Readers for the thirteen non-plain-text formats. Private on purpose |
| `src/embed.rs` | Model selection/loading, incl. the hand-assembled granite default; the one place granite's pinned files are fetched and checked (`granite_paths`) |
| `src/pipeline.rs` | An index pass as stages: the prepare pool (read, hash, extract, scan, parse, chunk, tokenize, cache lookup), the embed scheduler that hands token-budget batches to every lane and returns windows in id order, the soundness guard, and the stage timings |
| `src/session.rs` | Embedding from token ids: the index pass's own CPU session on ONNX Runtime (spinning off), padding and CLS pooling shared by every ORT lane |
| `src/cache.rs` | The machine-wide vector cache under the semlith home: keys, the LRU cap, lookups and inserts, and its stats line |
| `src/accel.rs` | The lanes: which are on, the lane policy, the per-lane dispatcher and its worker process (`semlith __embed-worker`), the length-framed id protocol, background start, fault injection, and the known-answer check behind `doctor --gpu` |
| `src/gpu.rs` | GPU detection per platform, the software-renderer refusal, the WebGPU adapter choice, the pinned WebGPU plugin and fp16 downloads, and the session enum every worker runs |
| `src/cuda.rs` | NVML detection, the pinned CUDA pack (Linux x86_64) and the CUDA session a worker runs |
| `src/coreml.rs` | The Neural Engine and Core ML GPU lanes on Apple silicon: the pack's manifest, bucket planning, and the Core ML calls |
| `src/packs.rs` | Every downloaded component pinned by URL and digest, fetched, checked and unpacked into the model cache |
| `src/trt.rs`, `src/openvino.rs`, `src/llama.rs` | The experimental lanes: NVIDIA's TensorRT for RTX and Intel's OpenVINO plugin providers, and llama.cpp's server on loopback |
| `src/priority.rs` | The daemon's priority following its work: background while idle, normal while anything embeds or a request is served |
| `src/filter.rs` | `--path`/`--ext`/`--lang` → GLOB patterns → one chunk id set; the deny-list and the credential shapes |
| `src/keyscan.rs` | The verdict on each secret-shaped match: live or declared test dummy, a 0-100 % confidence with its signals, the mask, the salted fingerprint, redaction, and the one accept/refuse decision every pass applies |
| `src/tree.rs` | `semlith files --tree` / `semlith_files {tree: true}`: directories with counts, files with symbols, and what is on disk but not indexed |
| `src/fleet.rs` | Several stores, one query, merged ranking |
| `src/graph.rs` | tree-sitter extraction, the bounded traversals over the edges, and the ranked walk search expands through |
| `src/pattern.rs` | `semlith pattern`: one tree-sitter query over the indexed files of one language |
| `src/ledger.rs` | The one place a retrieval is recorded, whichever surface answered it |
| `src/report/` | The five reports: one typed structure (`mod.rs`, which also gathers each report's figures), Markdown/CSV/JSON/HTML (`render.rs`) and the natively typeset PDF (`pdf.rs`, `fonts/`) over it, so the formats cannot disagree. No engine of its own |
| `src/schedule.rs` | A report the daemon writes on a cadence: the record (every option the builder set), `~/.semlith/schedules.json`, one file per schedule replaced in place, and the one thread that runs them |
| `src/replay.rs` | What an agent did after an answer, read from this machine's Claude Code transcripts unless the Privacy page's toggle is off (on by default from 0.35.0; a file that wrote off stays off) |
| `src/image.rs` | Image support: the five extensions, and the CLIP pair that makes a picture comparable with a sentence |
| `src/lock.rs` | One writer per store, OS advisory lock (not file existence) |
| `src/watch.rs` | Event source in front of the same indexer `index` runs |
| `src/mcp.rs` | Hand-rolled stdio JSON-RPC MCP server |
| `src/add.rs` | `semlith add`: the one outbound fetch, its refusals, and where a fetched file lands |
| `src/home.rs` | The store home, `registry.json`, and which store a command resolves to |
| `src/http.rs` | Hand-rolled synchronous HTTP/1.1: the token, `Host` and CSP rules live here |
| `src/daemon.rs` | `semlith start`: the locks, the watcher threads, the write queue |
| `src/routes.rs` | The daemon's routes, as adapters over `Fleet` and the queue |
| `src/portal/` | The page itself, `include_bytes!`d — HTML, CSS, JS, IBM Plex. No `style` attribute anywhere: the CSP drops it silently, so data-driven sizes go through the CSSOM |
| `src/drop.rs` | `POST /api/drop/resolve`: a dropped item's name, size and modified time turned into the real path — the macOS drag pasteboard, open Explorer windows, the OS search index, then a bounded walk — never an upload |
| `src/helpers.rs` | The opt-in "Index with semlith" file-manager entries (Finder, Explorer, Nautilus, Dolphin, Thunar), installed and removed beside the user's own |
| `src/proxy.rs` | `semlith mcp` forwarding to a running daemon |
| `src/clients.rs` | `docs/clients.md`'s client stanzas, parsed, so the portal shows the tested text |
| `src/cloud.rs` | Semlith Cloud's client: `cloud.json`, the one function that calls the host (`call`), device login, status, remote stores, the merge and forward behind remote answers, push, ledger sync, report and replay. Nothing in it runs for a machine that never signed in |
| `src/main.rs` | Clap parsing and human output formatting |
### Invariants worth knowing before editing
- **The writer never runs a model.** From 0.32.0 an index pass is three stages
(`pipeline.rs`): a prepare pool that reads, hashes, extracts, scans, parses,
chunks and tokenizes each file once, with no database; the writer, which alone
touches the store and commits a file's rows in a transaction; and the embed
scheduler, which hands batches to every lane, the CPU's own included, and
returns whole windows in the order they were sent. Vectors land in id order,
window by window, because `exact.f32` is binary-searched and a shard is named
by its first id. A window's rows are committed before it is handed over, so no
window carries an id whose row could roll back. A checkpoint drains every
window in flight before it saves and commits hashes; a stop cancels them and
the undo takes their rows.
- **Every vector is checked before it lands.** `pipeline::sound`: finite and not
all zero, on every lane. An unsound one goes back to the CPU, whatever the CPU
switch says, and only the CPU may take it. The Neural Engine's fp16 LayerNorm
overflow is the failure this exists for, and nothing downstream would notice it.
- **The vector cache is the machine's, never a store's.** One file under the
semlith home, keyed by the hash of the text the model is shown with the model
and its revision, the variant, the chunking rules and the truncation folded
in. Compaction never touches it and its eviction never touches a store.
Only processes that use the lanes (the daemon, `index`, `watch`) read it, or
a process that names a cap in the environment; the library alone does not,
so the retrieval harness never scores a vector another run left behind (#88).
- **Compaction never changes a live vector.** `compact` rebuilds shards from
`exact.f32`, and that is only sound because semlith never calibrates turbovec
and turbovec encodes the same rows to the same bytes however they are batched.
Calibrating an index would break it; so would rebuilding from anything but the
sidecar. `compact::tests::compaction_keeps_every_answer_and_drops_every_dead_byte`
asserts ids, order and scores before and after.
- **A shard swap is bracketed.** `compact` writes the new set to
`index.compact/`, marks the store (`vectors_swapping` in meta), renames the
live set aside and the new one in, moves the generation, and clears the mark.
`Semlith::search_vectors` asks again when a search began or ended inside that
window, and `index::recover` finishes or undoes a swap a crash interrupted
under the next writer's lock. A reader re-opens its indexes — not only evicts
them — when the generation moves, because the shard list is read at open.
- **Store connections set `secure_delete`.** A deleted row's bytes are
overwritten when its page is freed (#148). Do not turn it off to save the ~9 %
it costs a delete.
- **There is no cookie.** The portal's session token travels in a
`Semlith-Token` header, because every port on `localhost` is the same site and
`SameSite=Strict` never separated this daemon from a page served by anything
else on `127.0.0.1`. Nothing may set a cookie or read one. A write is refused
before the token is compared unless it carries a JSON content type and, from a
client that sends fetch metadata, `Sec-Fetch-Site: same-origin` — which is why
`http::answer` checks the origin rule first and returns 403 with an empty body.
The page and its own static assets are the one credential-free surface, because
a browser attaches no header to a stylesheet.
- **The home directory is read in one place.** `home::user_home` is the only
function in `src/` that reads `HOME`, and on Windows it is the only one that
reads `USERPROFILE` or `HOMEDRIVE`+`HOMEPATH`. Nine call sites used to ask the
environment themselves and each had a different answer when it was not there,
which is the whole of issues #70 to #73: the store home, the model cache, the
portal's directory browser, the deny-list, the index boundary and `semlith
setup` disagreed about where home was. `tests/home.rs` fails on a tenth
lookup, so one cannot be added by accident.
- **Nothing falls back to the working directory.** `user_home`, `home_or_error`,
`stores_root`, `bin_dir`, `registry_path`, `agent_key_path`,
`Registry::dir_of` and `model_cache_dir` all return a `Result`, and no variant
of any of them returns a default. A machine with no home gets an error naming
`SEMLITH_HOME`; it does not get a store, a registry, an agent key and 52 MB of
weights written into whatever directory the process started in. The rules that
depend on the home fail closed with it: `filter::denied` refuses a path it
cannot check against `~/.ssh`, and the portal's `dirs` route answers 500 rather
than rooting at a volume.
- **A path on its way out is plain.** `plain()` strips the Windows verbatim
`\\?\` prefix where a path becomes text — hits, file rows, image rows, symbol
rows, call sites — and nowhere else. The store keeps the verbatim form, which
is what makes a path longer than 260 characters work, and `semlith read`
accepts either.
- **The deny-list is one table.** `filter::DENIED_DIRS` and
`filter::DENIED_NAMES` in `filter.rs`, applied by `Boundary::refuses` to walked
entries and explicitly named paths alike. A second copy anywhere is a rule that
will disagree with itself. The hidden-file rule applies to an explicit path
too: that is the whole of what `~/.ssh/id_rsa` used to slip past.
- **A store outside the store home is opened only when the registry trusts it.**
`Registry::trusts` is the one predicate; `home::resolve` and `home::all_dirs`
both ask it, and `Discovery::read` asks it before it will read a `daemon.json`
at all. `--store` and `SEMLITH_STORE` bypass it deliberately — naming a store
is an instruction rather than a discovery.
- **A store connection refuses writes until a write path asks.** `store::open`
sets `query_only`, and `Semlith::writing` lifts it around `index_set`,
`forget_held` and the ledger's `record_retrieval`. A fourth write path has to
take the same guard or SQLite will refuse it, which is the point.
- **The two Linux release binaries load ONNX Runtime from beside themselves.**
`embed::link_runtime` runs before anything asks for a model and is a no-op in
every build without the `dynamic-ort` feature. Anything that loads a model
through a path that does not call it will work on every platform except the one
the release ships.
- **A chunk starts where the code does, and the fixed window is the fallback.**
From 0.22.0 `chunk::chunk_file` cuts a file at the definitions `graph::extract`
found, walking each one upwards over the doc comment and attributes attached to
it, so a constant and the six lines saying what it is for are one chunk instead
of two halves of two. Markdown cuts at its headings. Everything else — a
language with no grammar, a document a reader turned into prose, a plain text
file — keeps the 800-character window it always had. The budget still ends a
chunk, so a function longer than the budget is split as anything else is; what
a cut buys is that a definition never *begins* mid-chunk. The two-line overlap
is dropped where the next chunk starts on a cut, because that boundary is not
arbitrary and repeating across it is the blurring the cut exists to remove.
This is why `graph::extract` now runs above the chunking in `index_set_writing`
rather than below it.
- **`Chunk::context` is shown to the model and stored nowhere.** A Markdown
chunk three headings deep is written as though the reader has the two above it
in mind, so the heading path is prepended to what is embedded. It cannot be
part of `chunk.text`: `Semlith::read` maps every line of a chunk to
`start_line + offset`, and one invented line at the front would shift every
span the store can answer with. FTS5 indexes the stored text, so the path is
not keyword-searchable either — which is correct, because it is not in the
file.
- **Definition chunking is a store format, and it is format 3.** A store written
by 0.21.0 holds fixed windows, opens unchanged and answers; its files have not
changed, the rule for what a chunk is has, so the hash check would leave it on
the old rule for ever. The first index pass under 0.22.0 therefore re-chunks
everything it walks, and writes the format row only at the end of a pass that
swept the whole store and was not stopped — a pass over one directory leaves
the rest of the store on the old rule, and the row would be a claim about files
the run never opened. `semlith stats` says which rule a store is on.
- **The model is fixed when a store is created.** Vectors from two models are
not comparable; switching means delete and re-index. Changing
`embed::GRANITE_NAME` orphans every store that recorded the old string.
- **One writer per store.** `watch` holds the lock for its whole life, so a
concurrent `index` on that store is refused by design. `semlith start` is the
same rule taken further: it is the writer for every store it opened, and the
portal and a forwarded `semlith_index` reach it through the store's queue —
`Semlith::index_within_held` and `forget_held` exist for exactly that caller.
- **`semlith mcp` forwards when a daemon is running**, found through the
discovery file beside the lock, and the daemon answers with the same
`mcp::answer` the stdio server runs. From 0.18.0 this is what every
registration semlith writes launches, which is why no client configuration
carries the agent key and why a rotation reconfigures nothing: the key is read
from `~/.semlith/agent.key` by this process, not expanded from the
environment by the client. Every failure to reach one — no file, an
unreadable file, a dead port — falls back to opening the store directly.
- **`mcp.rs`: stdout is protocol, stderr is diagnostics.** Never print to
stdout outside the JSON-RPC framing, and keep `quiet` set on the Fleet there.
- The MCP server speaks two protocol eras at once, decided per message by the
presence of `_meta.io.modelcontextprotocol/protocolVersion`. Every entry in
`mcp::SUPPORTED` has a session in `tests/mcp.rs` proving it — adding a
revision means adding that session.
- `filter.rs` derives both the vector allowlist and the FTS5 predicate from one
id set, so the halves can never disagree about eligibility.
- **Search has no graph list (0.36.0).** It fuses the vector, keyword and image
lists and lifts definitions for an identifier. The graph walk that was a third
list from 0.12.0 to 0.35.0 scored the same as no walk on the 727-question
benchmark of 2026-10-01 and cost a third of a search on a large store, so it
was removed with its proximity tiebreak and the `provenance` field.
`neighbors`, `impact`, `path`, `trace` and brief's one-hop callers and callees
are where the graph answers.
- **The rerank has no model and no learned weights, and its factors are
tiebreaks.** Freshness and test paths (graph distance went with the graph list
in 0.36.0). Each is
under 1.6x — asserted, not intended — so a chunk the query matched badly cannot
climb over one it matched well. A third factor, a lift for a chunk inside a
named definition, was specified, built, measured and removed in the same
release: in a code repository it is a second and blunter `prefer: code` applied
to every query, and it fights the real one. Do not reintroduce it, and do not
add a learned weight here. The per-repository learned profile is still a
roadmap item; the version it used to be numbered as was reassigned when the
roadmap was renumbered on 2026-09-16, so this sentence names no release.
- **An identifier-shaped query is answered by the definition, above the
fusion.** `search_preferring` reads the chunks that define the exact name
through `store::symbols_by_names` — the same filter every other list sees, so
it cannot return a chunk the filter excluded — and lifts them to the front of
the ranked list with a stable sort, badged `definition`. It is precedence, not
a weight: the list contributes 0.0 to the fusion score and exists in the fused
set only so the chunk is present and carries its badge. Reciprocal-rank fusion
is nearly flat across the first ranks, so one authoritative list placing a
definition first was outvoted by two vague lists placing something else fourth
and fifth plus a graph list derived from them — five identifier questions on
the pinned corpus had no satisfying span in the first eight results, two of
them definitions FTS5 ranked first by itself. Do not give this list a weight
instead: tuned high it drags a definition's neighbours up through the graph
list, tuned low it does nothing.
- **The query's shape is read in one place.** `shape_of` in `lib.rs`, from the
query text and nothing else, so every surface that prints it calls that
function and there is no second classifier. The portal draws the hint from
`/api/search`'s own `shape_label` and `weighting` fields for the same reason: a
classifier in the browser would be a second opinion about an answer the server
already ranked. An identifier-shaped query weights the keyword list twice; a
question-shaped one leaves the two level. `prefer` multiplies and never
filters, so `prefer: code` over a corpus of prose still answers.
- **`edges.line` is where the source text sat, not where anything is defined.**
Added by the same `ALTER TABLE` mechanism `hint` uses, so `FORMAT_VERSION` does
not move, and NULL on every row an older binary wrote. A renderer prints
nothing for a NULL rather than falling back to the enclosing definition's line:
those are different lines, often hundreds apart, and a wrong call site is worse
than no call site.
- **`aliases` is a dependency kind.** A re-export or an aliased import becomes a
symbol of its own plus an `aliases` edge to the bare name it stands for, and
the kind joins `DEPENDENCY_KINDS` so a walk can cross it — a walk that would
not answered "not connected" about code that is connected. A re-export that
does not rename emits nothing; there is no name change to cross.
- **The not-indexed list and acceptances are two additive tables.** `refusals`
and `acceptances`, `IF NOT EXISTS`, so `FORMAT_VERSION` does not move. Every
pass that decides not to index something writes a row; a person — never an
agent — decides through the session-token routes or `semlith refused`. From
0.35.0 a decision may name several files at once
(`/api/refused/decide`: in, redact, out, reset), and each file is still its
own decision and its own ledger row. Keep out is an acceptance whose mode is
`refused`, for any reviewable class. An acceptance holds salted blake3 fingerprints,
never values, and `keyscan::decide` is the one function every pass (index,
watcher, catch-up, read from disk) asks. A credential file is never accepted.
- **`read` answers from chunks, never from disk — except for a file it indexed
that has changed since.** Then (0.30.0) it reads the current lines, through
the same secret scan and the same acceptance a pass applies, and marks the
answer read from disk. `Semlith::read` returns a span
stitched from the store's own rows, by line number rather than by
concatenation, because chunks overlap by two lines and concatenating repeats
the seam. Reading the file off disk would answer for content semlith was never
allowed to index — an agent holding the agent key could name `~/.ssh/id_rsa` as
a span. A span path is resolved by suffix against what was indexed, because a
locate answer prints a store-relative path and `files.path` is absolute; two
files matching one suffix is an error naming both, never a guess.
- **There is one language table, and it is `filter::LANGUAGES`.** From 0.17.0 the
extractor has no list of its own: `graph::has_graph` answers "does this
language carry symbols and edges" by asking whether a grammar exists, and
`graph::language_of` dispatches through the same `filter::language_of_path`
the search filter uses. The six-language copy that used to live in `graph.rs`
drifted from the forty-six the filter advertised for four releases, so
`--language kotlin` narrowed a search perfectly well and `semlith symbol`
then answered nothing about the same corpus, with nothing saying why. Do not
add a second list; add a row to the one table and a grammar beside it.
`every_advertised_language_has_a_working_grammar` fails for a row that has neither a
grammar nor an entry in `graph::WITHOUT_GRAMMAR` giving the reason.
- **A grammar's licence is a gate, not a detail.** Every parser is compiled into
an Apache-2.0 binary, so a copyleft grammar would take the whole crate with
it. `tests/licences.rs` reads cargo's own metadata and refuses anything
outside a permissive allowlist, and it regenerates `THIRD-PARTY-NOTICES` so
the notice those licences require cannot go stale. A language whose only
grammar is copyleft ships without a graph and is named in
`graph::WITHOUT_GRAMMAR` — it is never taken anyway.
- **A grammar is a crates.io dependency, never a git one.** `cargo publish`
rejects git dependencies, so a grammar that is not published is either
replaced by another published crate or vendored with its licence. Two
languages already needed the first: `tree-sitter` sets `links =
"tree-sitter"`, so a grammar crate that depends on `tree-sitter` itself rather
than on `tree-sitter-language` cannot coexist with the version this crate
links. That is why clojure uses `tree-sitter-clojure-orchard` and perl uses
`ts-parser-perl`.
- **A language proves itself with a fixture, not with a grammar.** Every row in
`filter::LANGUAGES` has a file under `tests/fixtures/graph/<language>/` and a
row in `graph::tests::FIXTURES` declaring the symbols and edges it must yield.
A grammar that compiles and extracts nothing passes every other gate in the
crate; this is the one it fails. Write the row first and make the query meet
it — loosening a row to match what the query happened to produce is the move
the table exists to prevent.
- **Text predicates go inside the pattern's own parentheses.** `(#eq? @a "id")`
written after a pattern rather than inside it is a pattern of its own and
filters nothing, silently. That is how `rel="stylesheet"` briefly became an
HTML symbol.
- **`pattern` reads its grammar from `graph::language_grammar`.** The extractor's
own table, so a language `pattern` accepts is a language the graph accepts and
there is no second list to drift. Its text comes from the store's chunks too,
and a file whose first line the store does not hold is skipped rather than
parsed at line numbers that would be wrong.
- **The graph is extracted inside `index_set`, never separately.** It is written
where the file id and the new chunk ids are both already in hand, after
`delete_file` has taken the old rows away. That is the whole freshness claim:
the pass that re-embeds a file is the pass that re-extracts it, so there is no
artifact that can be stale. A `semlith graph build` command would be the one
change that breaks the release's central property — do not add one.
- **`edges.dst` is a symbol *name*, not an id.** `edges.src` is an id and
cascades with its file; the target is resolved through `symbols(name)` at
query time. With an id there, re-indexing `b.rs` would reissue its symbol ids
and silently orphan every edge pointing into it from `a.rs` — the graph would
rot on exactly the operation this design exists to make safe. It also lets an
edge to something unindexed (a standard-library call) still be recorded.
- **Confidence is four values, and only two of them are stored.** `extracted` (the
file said where the name came from) and `inferred` (a bare name match) are
written into `edges.confidence` at extraction time. `resolved` (the candidates
ranked down to one) and `ambiguous` (they did not) are computed by
`store::edges_out` when a query runs and are never written down — deliberately,
because re-indexing a target changes which definitions exist and a stored
ranking would go stale the moment it did. `edges_in` answers from the edge's own
`src` id, so callers still carry the stored value and there is nothing to
resolve. Nothing that renders an edge may present one value as another.
**What to trust, in order.** `extracted` and `resolved` are answers: the source
named the target, or the ranking left exactly one candidate standing. `inferred`
is a hint worth checking. `ambiguous` is not an answer at all — it means the
store holds several definitions of that name and cannot say which one this is,
and it carries the count so a renderer says "4 definitions" rather than printing
four calls. Never quote an ambiguous edge as a call site without opening the
file.
- **`edges.hint` is what the source said, not what the query decided.** The module
of a scoped call, the receiver of a method call, the object of a qualified
Python or TypeScript call. Nullable, NULL on every row an older binary wrote,
added by the `ALTER TABLE` in `store::add_columns` that runs on open — which is
why `FORMAT_VERSION` does not move. The ranking that reads it prefers, in order:
the calling file's own definition, a definition whose file the hint names, one in
a file the caller imports, then a unique definition in the corpus — except that a
module path (`store::edges_out`, Python's `flask.Flask` with `import flask`)
outranks the calling file unless it names the calling file, since 0.42.0. A
directory the caller and the candidate both sit in names nothing: the
checkout's own folder is often called after the package. An edge the
syntax tree already settled stays `extracted` and is not re-ranked: a fact
outranks a ranking. Rust's supplement must not emit the path segment of a scoped
call as a second `calls` edge — `store::edges_out()` is a call to `edges_out`,
not to `store`.
- **A call through a receiver never resolves to a free function (0.42.0).** A
method call's hint starts with `graph::RECEIVER` (`.`): `.HiArgs`, `.Flask`
(`self`'s class), or `.` alone. `store::through_receiver` settles such an edge
only on a method owned by the receiver's stated type; everything else is
`ambiguous` among the methods of the name, free functions left out. The
calling-file tier is exactly what made `.build_parallel().run(||` in
ripgrep's main.rs "resolve" to `main::run`, so it must not apply to a
receiver. No type checker: an untyped receiver stays ambiguous, it is not
guessed. Rust and Python mark receivers; a Python object the file imports as
a module stays a module path. `coverage_by_language`'s census reads the mark
the same way, which is why the settled share dropped in 0.42.0.
- **A module's edges leave from the module.** `Edge::from_module` says an edge
sits at the top level, and `store::write_extraction` gives it the module's
id even when a definition shares the file's name (`main.rs`, `fn main`).
- **Markdown headings come from `chunk::headings`.** One scanner, ATX and setext,
gives the chunk cuts, the heading path and the graph's heading symbols, so a
section is the same section to all three; tree-sitter-md's sections open only
at ATX headings and are not used for spans.
- **"Called and not found" leaves the language's own out.** `graph::is_builtin`
is the table (Go, Rust, Python, JavaScript, TypeScript); `store::
unresolved_targets` applies it and `store::builtin_targets` counts what it
left out.
- **Traversal is bounded and reads one hop at a time.** No `petgraph`, no
in-memory whole-graph structure: `graph::neighbours` and
`graph::shortest_path` walk the indexed `edges(src)`/`edges(dst)` columns
under a depth limit and `graph::MAX_NODES`, which is what keeps peak RSS flat
as the corpus grows. `shortest_path` follows `DEPENDENCY_KINDS` only — the
structural edges are true and useless for reachability, since every symbol is
one hop from the file that defines it.
- **A path walks definitions, not names.** A node in the traversal is a name, a
file and a line, and a chain may only leave from the definition it arrived at.
Refusing ambiguous edges is not enough on its own: at 0.14.0 every hop of
`call_tool -> record_retrieval` on this repository's own store resolved to one
definition, and the chain was still false, because hop 3 arrived at `search` in
`lib.rs` and hop 4 left from `search` in `routes.rs`. Each hop true, the chain
not. (That pair is genuinely connected from 0.15.0, which is the release that
made `call_tool` record; `search_in -> record_retrieval` is the case that still
has no chain.) So `semlith path` refuses by default and says "not connected within N hops
by resolved edges"; `--all-edges` / `all_edges: true` walks the old way and
labels it — both endpoints per hop with file and line, a seam where the chain
changed subject, a trailer whose four confidence counts add up to the hop count,
and the line "A hypothesis, not a finding." `--strict` / `strict: true` states
the default out loud and wins over `--all-edges`. One renderer serves the CLI and
the MCP reply, so the two cannot describe one answer differently.
- **`semlith neighbors` collapses and hides, and `--all` undoes both.** Callees to
a name with several definitions are one row carrying the count, because four
rows saying `get` read as four calls. Targets the store holds no definition for
are counted rather than silently omitted, and `--all` / `all: true` lists them:
"semlith shows no callees" and "everything this calls is outside the index" are
different facts.
- **`semlith_search` answers where by default; the CLI still answers with text.**
Over MCP and `/api/search`, `format: locate` returns one line per hit — path,
span, enclosing symbol and kind, the lists that found it, a freshness flag,
one line of text — grouped by file and cut
to `max_tokens` (default 1500, floor 200) with `truncated: N of M`. Ask for
`format: "excerpt"` when you actually need the body; that is what 0.14.0
returned. `semlith search` in a terminal is unchanged. Every hit carries
`fresh`, from one `stat` per distinct path against the recorded size and mtime —
conservative on purpose, so a `touch` reads as stale, because a false "check
this" costs a reread and a false "this is current" costs a wrong quotation.
- **Reverse reachability is back, free, from 0.26.0.** `semlith impact`,
`semlith_impact` and `/api/impact` left in 0.13.0 to be sold, and never were:
the monetization hold of 2026-09-15 made the whole binary free, so the thing
that was held back is now shipped with a page of its own. `graph::impact` is
breadth first over `store::edges_in` — the same `edges_dst` index
`graph::neighbours` reads — under the same two rules the path finder walks by:
a node is a definition rather than a name, and an ambiguous edge is not
crossed unless `all_edges` says to. Because it is breadth first, the hop
recorded against a caller is the fewest hops it takes, and `IMPACT_LIMIT`
bounds the answer with a count of what it left out rather than a truncated
list that reads as complete. One renderer, `Impact::render`, serves the CLI
and the MCP reply, as the chain's does.
- **An image is embedded in the same `index_set` pass that embeds text**, inside
the same held lock, on the same changed-file path. There is no image build
step and there must not be one, for the reason there is no `graph build`: the
pass that re-reads a file is the pass that re-embeds it, so no artifact can go
stale. `forget` takes the row and the vector away together.
- **The query for the image half goes through CLIP's text encoder**, never the
store's own model. A granite vector and a CLIP vector are numbers of different
lengths about different things, and comparing them returns whatever the
arithmetic happens to produce — which is why the two CLIP repositories in
`image.rs` are fixed as a pair rather than selectable.
- **A picture answers a query that describes one, and nothing else (0.42.0).**
The image list is asked only when `pictured(query)` (beside `shape_of` and
`code_shaped` in `lib.rs`: a picture word, a colour or a shape) and holds only
images CLIP scores at the floor or above, each weighted as the two lists a
chunk can be found by. CLIP reads the words in a screenshot, so a question
naming a project scores that project's screenshots as high as a description
of them (flask's `flaskr_index.png` at 0.355 for "how does flask dispatch a
request to a view"); no cosine floor or margin separated them, which is why
the gate is the query's words. A weak image is left out, not weighted down.
- **The images table and `images/` are additive, so `FORMAT_VERSION` does not
move.** Both are `IF NOT EXISTS`, and a binary that knows nothing about either
reads such a store as the text corpus it already was. Anything that would make
an older binary misread a store is a format bump instead; adding a table it
ignores is not.
- **A read across stores answers from the stores it can read.** From 0.27.0
`Fleet::chosen` and `Fleet::each` — the two functions every aggregating read
selects its members through, which is why the guard is there and not in
`routes::with_fleet`, which Search and Files never reach — drop a store whose
database cannot be read and collect it in `Fleet::failed`. One unreadable
store used to make every such route answer `500 disk I/O error` (#129). A
request scoped to a store that cannot be read still fails: there is no partial
answer to give. The probe is `store::readable`, one indexed row out of each
large table, deliberately not `stats()` — that is three full scans on a path
every search takes, measured at 2.65 ms against 0.24 ms on a 10 390-chunk
store. Nothing in the response is removed by this: `failed` is absent when
nothing failed.
- **A schedule is the daemon's work, and the file is the whole of the state.**
`~/.semlith/schedules.json` is its own file beside the registry rather than a
section of it, so a schedules file somebody's editor truncated costs them
their schedules and not every store on the machine. The cadence is stored as
an interval in seconds: the portal's three chips are shortcuts over a number,
so a cadence no chip can spell still round-trips. `next_run` is on disk, which
is the whole of why a restart loses nothing — the runner owes nothing to how
long the process lived. Two schedules never generate at once
(`schedule::generating` is one process-wide slot, and any other generator must
take it too), a due schedule over a store that is still indexing generates
nothing rather than reporting on a half-written index, and a destination that
has gone away is recorded in `last_error` and shown. `create_dir_all` is
deliberately never called on a destination: it would answer an unmounted
volume by writing into the empty mount point, which looks like success and
loses the file. The runner sleeps on a condition variable to the next due
time, capped at a minute — not to poll, but because `semlith schedule add` is
a different process and cannot reach that condvar.
- **Five renderers, one typed report.** A `Report` is figures (value, unit,
currency, over, display), a meta strip, KPIs and blocks (typed tables, facts,
notes, a recompute command); `render.rs` draws Markdown, CSV, JSON and HTML and
`pdf.rs` the PDF, all from that structure and nothing else, so no format can
hold a figure another lacks. Every number goes through `report::num` (`number`
is its `i64` face for the CLI), which prints what the portal's `n` and `pct`
print; `report::tests` pins app.js to it. `Report::render` returns text for
`report::FORMATS` (markdown, json, html); CSV (a zip of one file per table, or
one table through `render_table`) and PDF are bytes from `render_bytes`, which
is why a text-only surface calls `check_format` and one that hands back bytes
`check_any_format`. The HTML's colours are read from `src/portal/style.css`'s
tokens at render time — change a token there and every report follows — and it
embeds the portal's Plex woff2 and logos, with no script and no URL. The PDF is
typeset natively to match the HTML's print rendering, not printed by a browser:
IBM Plex (`src/report/fonts`, OFL, cut to Latin and Latin Extended-A) embedded
as a per-document subset (`subsetter`) measured with its real advances
(`ttf-parser`), and a glyph Plex lacks taken from the OS's CJK fonts at render
time, read for that render only and never kept. Every glyph has a ToUnicode
entry, so the text layer is the text even where no font had the glyph. Never a
base-14 font, never `?`. The portal's preview is that HTML in a sandboxed frame
from `/report-preview/<id>` under `report::REPORT_CSP` (no script): the frame
cannot send the session token, so the 128-bit id, minted only by the
token-guarded `/api/report?preview=1` and kept ten minutes, is the credential.
- **A report's figures come from the rows inside its window.** Each report
gathers through one function (`tally` for the ledger reports) over
`store::report_rows_since`, so a window narrows every figure on the page, not
only some of them. A redacted query is a blake3 digest keyed with the
machine's `report.salt` beside the agent key — an unkeyed digest is a
dictionary lookup away from a symbol name.
- **The ledger records by default, locally, from every surface.** `src/ledger.rs`
is the one writer, and stdio MCP, the daemon's `/mcp` route, the CLI and the
portal all call it — a search from the CLI and the same search from an agent
must be counted the same way, which is why there is one module rather than four
call sites. A row carries the client's own name from the MCP `initialize`
handshake (`claude-code`, `cursor`, `cli`, `portal`) and a session id. Graph
tools record too, against the bytes a grep for that name would have made you
read; a zero-hit retrieval is recorded and credited nothing, because a ledger
that remembers only its successes is a marketing document. Rows are
hash-chained, each carrying the hash of the one before it, so
`store::ledger_break` finds an edited or deleted row — and the chain is
versioned *by the row*: `tool` is NULL on every pre-0.15.0 row and set on every
row since, which is what picks the formula, so a store holding both kinds
verifies end to end. From 0.42.0 `retrievals.chain = 2` marks chain v2, which
also covers `query_id`, the client's version and the whole-file basis; the
late columns (usage, `synced_at`) are covered by `seal`, rewritten by
`store::reseal` whenever they are written; and the newest v2 row is recorded
as a MACed head in meta, so a cut tail fails. v1 rows after the head are an
older binary's (a rollback) and allowed; an empty table verifies. Recording and `semlith ledger` are free on every tier,
permanently; do not put either behind a key.
- **On by default is the whole point, and the principle that makes it all right
is narrower than the one it replaced.** 0.12.0 said a local tool that starts
logging without being told is no different from one that phones home. What that
bought was a ledger nobody switched on: no denominator under the savings figure
and no rows in the audit trail. What holds instead is checkable — the rows never
leave the store they were written into, the daemon says on every start that it
is recording and names the flag that stops it, and erasing every row is one
`DELETE`. `semlith start --ledger` is removed rather than kept as a no-op, so a
script that passes it fails at parse time; `--no-ledger` stops a session and
`SEMLITH_LEDGER=0` stops a machine. Do not reintroduce an off-by-default.
- **Both sides of a ratio are counted on the same instrument.** The store's own
embedding tokenizer counts excerpt tokens — the same `tokenizer.json`, the
same cache, the same pinned digest. Four characters per token survives only
as the fallback for a session that never loaded a model, and
`retrievals.tokenizer` records which counted the row, so rows counted two
different ways are never summed. The whole-file side is what reading the
file would cost an agent (`ledger::whole_of`, 0.42.0): a format a reader
extracts (PDF, Office, OpenDocument, EPUB, mail, notebooks, HTML) is the text
the store holds for it, counted by the same counter as the excerpt; an image
holds no text and counts none; plain text is its bytes over four. Never the
bytes of an extracted file — a 2.2 MB PDF of 45,849 characters was credited
553,811 tokens that way. `retrievals.whole_basis` says which rule a row used.
- **Every figure comes from `ledger::figures`.** The Ledger page, Home, the
store cards, `/api/ledger`, `semlith ledger`, `stats`, the five reports and
`semlith_report` render that one function over the rows; none of them adds
anything up itself. A retrieval is a row whose tool is not a refund
(`raw-read`), a note, a refusal, or an accept or revoke
(`ledger::is_retrieval`); a row from `cli` or `portal` is a person's and is
kept out of the agents' figures (`ledger::is_human`); rows sharing a
`query_id` are one retrieval, counted once in the totals and once in each
store's share. A per-store SQL reader that survives takes
`store::AGENT_RETRIEVAL`, the same rule as SQL. Net is summed, never clamped;
latency is one figure per retrieval that measured work.
- **A client is named on every request, not only at `initialize`.** MCP
2026-07-28 carries `clientInfo` in `params._meta` and no session, so
`mcp::client_info` reads either place and the User-Agent stands in when both
are absent (it may only confirm a name `clientid` knows). A stateless
client's conversation is `State::conversation`: one client, version and
transport until 30 minutes of quiet. It is the one place to replace if a
revision defines a conversation id.
- **The daemon's readers share, and a run is answered from before it.**
Forwarded MCP calls take a reader from `State::mcp_fleet`, a pool of two to
four fleets all opened before the first writer starts; every reader fleet
calls `share_vectors`, so one `index::Vectors` per store and generation is
held process-wide (`index::shared`, weakly), as are the query session and
the counting tokenizer. The writer keeps its own index: it moves under a run.
A run over files a store already holds calls `hold_readers` before its first
write: every reader's connection opens a read transaction
(`store::pin_snapshot`) and answers from the store as it was until `release`
ends the run and the pin with it (`Fleet::reconcile`, run before and after
every reader call too). A first import is not pinned (keyword answers while
it embeds). A pinned connection must not write: `store::record_counted`,
`fill_usage` and `mark_synced` go through a connection of their own
(`beside`). A reader opened during a run (a store joining mid-run) starts on
the run's writes so far; a pin unreleased for six hours is dropped.
- **Bytes are text only when they read as text.** `chunk::is_binary` is git's
NUL rule; `chunk::not_text` refuses a file whose first 8 KiB is over 30 %
invalid UTF-8 or control characters, with the share in its reason. Latin-1
prose stays well under it and is read.
- **`mcp::tool_names` is the one tool list.** `routes::agents` reads it rather
than repeating it, because two hand-written copies is how a tool ends up served
by the server and invisible in the portal.
- **Two credentials, and their reach is what separates them.** The per-run
session token opens the portal and every `/api/*` route; the agent key opens
`http::MCP_PATH` and nothing else. That asymmetry is the whole design: a key
that sits in a client's config file on disk must not be able to rotate a
token, adopt a store or start an upgrade, so no route outside `/mcp` may ever
learn to accept one. `tests/endpoint.rs` asserts it.
- **The agent key is persisted, and never reminted implicitly.**
`home::agent_key` creates it on first read and returns it forever after;
`home::rotate_agent_key` is a separate function precisely so nothing can
rotate by accident from the state of the disk. A key that changed per run
would make every stanza stale on every restart, and `setup.rs` can only repair
Claude Code's config — everything else would need a human. Rotating the
session token therefore does not disconnect agents, and `/mcp` keeps serving
the previous key for `http::KEY_GRACE` — fifteen minutes, and never past the
daemon's exit — so an open session finishes. It used to be "until this process
exits", which on a daemon somebody leaves running is a second live credential
rather than a grace period.
- Extraction dispatches on extension *before* looking at bytes — `.docx` and
friends are ZIP archives and the binary check would reject them all.
- **Markup is stored as written except in a docs store (0.42.0).**
`chunk::MARKUP` (`html`, `htm`, the only two a reader ever stripped) goes
down the plain-text arm, so `read` and an exact search answer a template's
own lines; the writer swaps in `chunk::markup_text` only when the store's
`kind` or `lean` is `docs`, and for a page under the store's `downloads/`.
The `markup` meta row (`raw` or `extracted`, absent before 0.42.0 and read as
`extracted`) says how a store's markup was last swept; when it differs from
the store's rule, the pass drops those files' hashes so they alone are read
again, and the row moves only at the end of a full sweep. Additive, so
`FORMAT_VERSION` does not move.
- **Semlith Cloud is reached only after `semlith cloud login`.** With no
`~/.semlith/cloud.json` nothing opens a socket, starts the sync thread or
probes a host. `cloud::call` is the only function that calls one: it takes a
host and a path, so a token reaches the host it is stored with and no other,
it follows no redirect, it is refused under airgap, and it notes every call
on the Privacy page. Remote stores live in `registry.json`'s own `remote`
map, never in `stores`, because every older binary walks `stores` as
directories. `tests/cloud.rs` proves it against a stub host.
- **`add` is the only command that reaches the network besides `upgrade` and
the model download**, it is https-only, it refuses before opening a socket
under `--airgap`, and it writes only inside the store's own `downloads/`.
Every one of those is a tested case in `tests/add.rs`, not a comment. See
[#50](https://github.com/semlith/semlith/issues/50).
- `filter::LANGUAGES` has a `filenames` column as well as `extensions`, because
a Dockerfile and a Makefile have no extension. Both resolve into the same
pattern group, so there is still one code path behind the vector allowlist and
the FTS5 predicate.
`tests/retrieval.rs` has a fifth question class from 0.27.0, `image`, over a
**second** pinned corpus at `tests/fixtures/retrieval/images/`. It is separate
from the corpus above on purpose: that one's `files` and `bytes` are asserted on
every run because every published figure is a reading of it, so adding pictures
to it would rebaseline hit@k, bytes per answer and the census in one commit. The
class is opted into with `SEMLITH_RETRIEVAL_IMAGES`, so CLIP's download stays
out of a run that is measuring text, and `the_image_class_is_measured` runs it
alone in about a minute. Its corpus is generated by
`tests/fixtures/retrieval/images/generate.py` — regenerate rather than edit, and
re-pin `corpus.yaml` after. The three text files beside the eight pictures are
not optional: issue #122 was a *source file* outranking a picture, and a picture
that wins against nothing proves nothing.
Env overrides: `SEMLITH_DOWNLOAD_STALL` (seconds a model download may receive
nothing, default 60), `SEMLITH_STORE` (PATH-style separated), `SEMLITH_HOME`,
`SEMLITH_PORT`, `SEMLITH_AIRGAP`, `SEMLITH_EMBED_THREADS`,
`SEMLITH_INDEX_MEMORY`, `SEMLITH_SHARD_VECTORS`, `SEMLITH_CHECKPOINT_FILES`,
`SEMLITH_MODEL_CACHE`, `SEMLITH_MCP_INDEX_BUDGET`, `SEMLITH_LEDGER` (`0`, `off`
or `false` stops the ledger recording on this machine), `SEMLITH_ACCEL` (the
lanes on, comma-separated: `cpu`, `gpu`, `ane`, `cuda`, `trt`, `openvino`,
`llama`, `gpu-beside-ane`), `SEMLITH_VECTOR_CACHE_MB` (the vector cache's cap;
`0` turns it off), `SEMLITH_GPU_ADAPTER` (the WebGPU adapter to prefer, by a
substring of its name), `SEMLITH_OPENVINO_DEVICE` and `SEMLITH_LLAMA_DEVICE`
(force a device for those lanes; `CPU`/`cpu` is how CI checks their known answer
without the hardware). Harness-only, undocumented for users:
`SEMLITH_SERVICE_DIR` stands a directory in for the service manager, so a
test of the portal's Start at login writes a file there and never asks
launchd, systemd or Task Scheduler anything;
`SEMLITH_DROP_TIERS=walk` leaves the drag pasteboard, Explorer and the OS
index out of `/api/drop/resolve`, so a test of the walk and the temp rule is
not spent waiting on a cold runner's search service;
`SEMLITH_PACK_<NAME>` names an unpacked pack directory in place of the pinned
download, which is how a pack is tried before it is published; and
`SEMLITH_COREML_WORKER=current` runs the Core ML lanes from the binary being
built rather than the stable copy under `accel/coreml-worker-v<N>`, which is
what developing the worker needs. The copy is made once per `COREML_WORKER`
(`src/accel.rs`) and kept across upgrades, because macOS caches a model's
Neural Engine compilation per program: bump `COREML_WORKER` whenever the Core
ML worker's code, arguments or frames change, or users keep running the old
copy.
Deeper rationale lives in `docs/architecture.md`; what is and isn't a stability
contract lives in `docs/compatibility.md`.
## Conventions
- Comments explain *why*, not *what*. The ones worth writing save the next
person an hour — why a batch size is 32, why the hash is written after the
index.
- Deliberate shortcuts get a `ponytail:` comment naming the ceiling and the
upgrade path.
- Errors a user can act on: `bail!("store was built with X, not Y")`, not
`bail!("model mismatch")`.
- Conventional Commits. One logical change per PR. Update `CHANGELOG.md` under
`## [Unreleased]`.
- Behaviour changes to indexing or search need measured numbers, not adjectives.
The 100k benchmark store is a fixture kept at `~/.cache/semlith/bench/100k-store`
— build it once (recipe in `CONTRIBUTING.md`), not per release.
- A count stated in the README is asserted against its source by
`tests/readme.rs`, which also holds the under-500-line ceiling and the grep
for release-specific prose. Change a count in the code and the README fails
the build; add a count to the README and add its row there.
- Public surface changes must land in `docs/compatibility.md`; the client
stanzas in `docs/clients.md` are executed by `tests/clients.rs`, so a renamed
flag breaks there. From 0.18.0 that file also carries the registration facts,
in fence info strings a renderer hides: `sh register` is the command
`semlith setup` runs and must carry the scope that client spells "every
project", `sh unregister` is what clears an existing entry first,
`<lang> config path=…` (with `os=` and double quotes where a path has a space)
is a client's user-level configuration file, and `scope=project` marks a
registration command that would register only the current directory — semlith
does not run those. `clients.rs` also parses that file's structure, not only
its fences: the stdio stanzas are read from under `### Setting it up in your
client` and grouped by `#### Terminal`, `#### Editors` and `#### Desktop
apps`, and the HTTP ones from under `### Connecting over HTTP`. Reword any of
those five headings and the portal's Agents page silently empties — change the
prose around them, not them. The stanzas lived in the README until 0.17.2;
they are in `docs/clients.md` now, and the README only links to it.
## Deliberately out of scope
**Local means local, and it is checkable.** Since 0.9.0 Semlith does listen on a
port — `semlith start` serves the portal, and from 0.13.0 MCP at `/mcp` as well
— under rules that are tested rather than stated, and that are not up for
relaxation without an issue like
[#41](https://github.com/semlith/semlith/issues/41):
- `127.0.0.1` is the only bind address, and there is no flag to change it.
- The portal and every `/api/*` route need the per-run token, in a
`Semlith-Token` header. Without it: 401 and an empty body. `/mcp` takes the
agent key as a bearer instead, and is the only route that does.
- The `Host` header must be `localhost`, `127.0.0.1` or `::1`. Otherwise: 400.
- Every response carries a `Content-Security-Policy` allowing only `'self'`, and
no CORS header is emitted anywhere. The one exception is narrower, never
wider: a report preview (`/report-preview/<id>`) is served under
`report::REPORT_CSP`, which allows no script, no network and no form at all.
- Every byte the portal loads is `include_bytes!`d into the binary. No CDN, no
build step, no npm. The page loads with the cable unplugged.
- No telemetry, no analytics, and no update check Semlith makes on its own.
`semlith upgrade` and `semlith upgrade --check` exist from 0.10.0 and reach
GitHub, but only in the second a user asks: there is no startup check, no
timer, and no banner that appears without a click. `semlith add` is the same
shape from 0.11.0 — one request, for one URL, because somebody asked for it,
with no crawling and no re-fetching, made directly and never through a proxy
(0.46.0): through one the proxy resolves the name, and `PublicOnly` would
check only the proxy's address. The other downloads are model weights
and the pinned components the accelerator lanes run on:
- the embedding model, once, on the first index, and `semlith setup`'s
pre-fetch of the same file;
- CLIP's two halves, on the first image a store indexes and never at start;
- from 0.28.0, the WebGPU plugin (Microsoft's `onnxruntime-ep-webgpu` wheel,
from PyPI's `files.pythonhosted.org`) and the fp16 export of the embedding
model (Hugging Face, at the same pinned commit). They are fetched on the
first run in a daemon that has found a hardware GPU with the GPU lane on,
and never on a machine whose only adapter is a software renderer;
- from 0.28.0, the CUDA pack (ONNX Runtime's GPU build from GitHub, and
NVIDIA's CUDA libraries from their PyPI wheels). It is fetched only after
somebody turns CUDA on with `semlith accel on cuda` or the switch on the
page, and its size is stated before the download starts;
- from 0.32.0, the packs in `src/packs.rs`: granite as Core ML models (this
repository's own release, fetched by `semlith setup` on Apple silicon and by
`semlith accel on ane`), and — only on an explicit turn-on of an
experimental lane — NVIDIA's TensorRT for RTX and Intel's OpenVINO plugin
wheels from PyPI and llama.cpp's server from ggml-org's release with
granite as GGUF from this repository's. The llama.cpp server listens on
`127.0.0.1` with a key only its worker holds, and dies with the worker.
Every one of them is pinned by digest in `docs/models.md`. `--airgap` refuses
all of them unless they were pre-seeded into the model cache, naming what it
would have fetched.
Discuss in an issue before building: any other bind address, hosted embedding
APIs, any outbound connection, or a user-editable config file.
`~/.semlith/registry.json` is not one: it is tool-written state, like
`store.db`, `agent.key` and the lock file. Semlith writes it, nothing documents
a way to hand-edit it, and `--store`/`SEMLITH_STORE` remain the only way a user
names a store.
## Portal parity
**A CLI command or an MCP tool is not done until its portal view exists.** Every
release that adds one adds the view in the same release; parity debt is not a
thing this repository carries.
A *view* is not always a page. `semlith languages` is the table on Settings ›
About and `semlith_read` is Search's Read whole symbol — both have a surface a person
can open, which is what the rule is for. What the rule forbids is a capability
with no surface at all, and the exemption list in `tests/portal.rs` is where a
deliberate absence is argued rather than assumed: `start` and `mcp` have no
state of their own to show, and `models` lists every model a store could be
built with, which no page draws. `pattern` is Search's fourth mode from 0.35.0.
Adding a row there is allowed; adding one without the
reasoning beside it is not. `tests/portal.rs` is the gate: it reads the
subcommand list out of `--help` and the tool list off a running daemon, and
fails if any of them — bar the exemptions, which have their reasons recorded
there — has no route. It also reads every route `app.js` calls and fails on one
the daemon does not serve. The Agents page is the one view whose content is not
computed: it renders the stanzas `src/clients.rs` parses out of
`docs/clients.md`, so a change to that page is usually a change to that file.
`docs/portal.md` documents every page, and a new one belongs there in the same
release.
## What CI runs, and when
`ci.yml` runs `check` (fmt, clippy, the offline suite, on three OSes), `msrv`
and `scripts` on every push. Everything slower waits on the `changes` job,
which runs `.github/changes.sh`: a push that touched only prose (top-level or
`.github/` Markdown, `docs/`, `assets/`, the licence files) skips `gpu`, `cuda`,
`release-suites` and `mixed-vectors`, and so does a draft pull request until it
is marked ready. A file `src/` pulls in with `include_str!`/`include_bytes!`
(`docs/clients.md`, `docs/skill/*`) is never prose, and a rename counts as both
of its paths. A push is compared with the previous head only when this
workflow's run on that head, on the same branch and event, finished green and
its `changes` job printed `heavy=true`; otherwise the whole pull request is
compared with its base (a push to a branch is code), so a cancelled run, a
draft's fast lane or another branch's run never lets code through untested.
`.github/changes-test.sh` and `.github/verify-tag-test.sh` are the scripts
lane's self-tests for the two scripts; a change to either script changes its
test in the same commit.
`release-suites` runs each slow test on its own macOS runner and
`mixed-vectors` each of its three runs on its own runner, with
`mixed-vectors-median` writing the table. `native-smoke.yml` builds the binary
once per OS, then runs the portal check, the smoke harness and the browser drive
as three jobs side by side, each installing that build through
`.github/actions/install-semlith`. The models are cached per digest; the GPU
lane's downloads never are, because a no-GPU runner asserts it fetched none.
## Release
Tags `v*` trigger `.github/workflows/release.yml`, which first verifies the tag
is on `main` (`.github/verify-tag.sh`) and matches `Cargo.toml`'s version. Only
an org admin can create a `v*` tag (the `tag-creation` ruleset; `tag-protection`
separately stops one being moved or deleted, and neither is ever deleted, only
its enforcement flipped). The signing key lives in the `release` environment,
which only `v*` tags can deploy to, and only the `sign` job reads it; `attest`
holds `id-token`, and `release` holds `contents: write` and runs the
third-party release action. Keep each power in its one job (0.46.0, SEC-001
and SEC-007 of the 2026-10-23 audit). Tag builds use `--locked` and no cache.
GitHub Actions must stay SHA-pinned with a
trailing `# vN` comment for Dependabot.
**A release with a security fix raises the installers' floor.** `floor=` in
`install.sh` and `$floor` in `install.ps1` are the oldest release a resolved
`latest` may install; set both to the new version in any release whose
predecessors should no longer be installable from `latest`. No Intel macOS target — ONNX Runtime
stopped publishing `osx-x86_64`.
The Linux jobs build with `--no-default-features --features dynamic-ort` and
pack Microsoft's own ONNX Runtime into the archive beside the binary. The runner
is not what decides the floor and never was: the prebuilt ONNX Runtime `ort`
downloads references `__isoc23_strtol@GLIBC_2.38`, so 0.13.0's attempt to build
on `ubuntu-22.04` failed to link, and no older runner or container could have
helped. Not linking it is what helped. The `glibc floor` step runs `objdump -T`
over the packaged binary *and* the library beside it — what a user runs is the
pair — and **fails above `GLIBC_2.35`**. It printed a number for three releases
and the floor rose to 2.39 without failing a build; a gate that does not exit
non-zero is not a gate.
The runner matters as much as the feature, and the two are easy to confuse. Not
linking ONNX Runtime is what makes `ubuntu-22.04` *possible* — 0.13.0 tried it
and the link failed. It is the runner that then sets the floor: a Rust binary
references whatever symbol versions the glibc it was built against provides, so
one built on 24.04 needs 2.39 no matter what else it links. The first tagged run
with `dynamic-ort` demonstrated exactly that — GLIBCXX fell from 3.4.32 to
3.4.22, which is Microsoft's library, and GLIBC stayed at 2.39, which was the
binary. Both Linux jobs build on 22.04 for that reason; moving either forward
moves the floor.
`cargo install`, macOS and Windows link ONNX Runtime as they always have. The
two Linux release binaries are the only build that differs, which is why
`link_runtime` is a no-op everywhere else.
## `unsafe` in this crate
Every site, with the reason it is sound. Adding one means adding a row: a
`SAFETY:` comment on the block is necessary and not sufficient, because what
makes this set reviewable is being able to read the whole of it at once.
| Where | Call | Why it is sound |
|---|---|---|
| `embed::performance_cores` | `libc::sysctlbyname` | A NUL-terminated C string, and an `i32` whose size the call is told and will not exceed. macOS only. |
| `embed::performance_cores` (windows) | `GetSystemCpuSetInformation`, `ptr::read_unaligned`, a union field read | Called first with a null buffer of length zero, which only writes the size it needs into a live `u32`; then with a `Vec<u8>` of exactly that length, which the call is told and will not exceed. Each entry is read unaligned only while it fits inside the bytes the call wrote, the walk advances by the entry's own `Size` and stops if that is zero, and `CpuSet.EfficiencyClass` is read only when `Type` is `CpuSetInformation`. Every field is an integer, so any bytes are a valid value. Windows only. |
| `priority::platform::set` (macOS) | `libc::setpriority(PRIO_DARWIN_PROCESS, 0, …)` | Plain integers; `who` 0 names this process. Nothing is read back through a pointer. macOS only. |
| `priority::platform::set` (windows) | `GetCurrentProcess`, `SetPriorityClass`, `SetProcessInformation(ProcessPowerThrottling)` | The pseudo-handle for this process needs no closing, and the `PROCESS_POWER_THROTTLING_STATE` is passed with its own size, which is the documented contract. A build without power throttling fails the second call, and the class alone is the switch. Windows only. |
| `priority::platform::enter_indexing`/`leave_indexing` (macOS) | `pthread_get_qos_class_np`, `pthread_set_qos_class_self_np` | This thread's own handle, out-parameters that are live locals of the types the calls write, and class values from the enum. Nothing outlives the call. macOS only. |
| `priority::platform::enter_indexing`/`leave_indexing` (Linux) | `sched_getscheduler`, `sched_getparam`, `sched_setscheduler`, `mem::zeroed` | 0 names this thread; the `sched_param` structs are live locals the calls read and write, and what was read is what is put back. Linux only. |
| `coreml::mac::Session` | `MLModelConfiguration::setComputeUnits`, `setFunctionName`, `MLModel::modelWithContentsOfURL_configuration_error`, `MLMultiArray::initWithShape_dataType_error`, `dataPointer`, `ptr::copy_nonoverlapping`, `MLDictionaryFeatureProvider::initWithDictionary_error`, `predictionFromFeatures_error`, `featureValueForName`, `multiArrayValue` | Arrays of exactly the shape and type the model was converted with, contiguous as `initWithShape` documents, written through their own data pointer for exactly `batch * bucket` elements before they are read. The output's shape and strides are read rather than assumed and nothing past them is read. Every object is a retained value alive for the call. macOS only. |
| `trt::nvml` (Linux, Windows) | `dlopen`/`LoadLibraryA`, `dlsym`/`GetProcAddress`, `mem::transmute_copy`, `dlclose`/`FreeLibrary`, `nvmlInit_v2`, `nvmlDeviceGetHandleByIndex_v2`, `nvmlDeviceGetName`, `nvmlDeviceGetCudaComputeCapability`, `nvmlSystemGetDriverVersion`, `nvmlShutdown` | Each symbol is looked up by a NUL-terminated name and used at the signature in nvml.h; `Library::sym` asserts it is pointer-sized. Out-parameters are live locals of the size NVML is told: 96 bytes for the name, 80 for the driver version, two ints for the capability. The handle is used only between init and shutdown, and the library is closed on drop. Strings are read with `CStr::from_bytes_until_nul`. |
| `trt::native::preload` (Linux) | `libc::dlopen(RTLD_NOW\|RTLD_GLOBAL)`, `libc::dlerror` | A `CString` path that outlives the call. The handle is deliberately never closed. `dlerror` is read on the same thread straight after the failure, null-checked and copied. |
| `trt::native::search_beside` (Windows) | `SetDllDirectoryW` | A NUL-terminated UTF-16 path that outlives the call, which copies it. The worker runs one lane. |
| `openvino::ov_device` | `ort::api().EpDevice_EpMetadata`, `GetKeyValue` | The device pointer is one ORT handed out for this environment, alive as long as the device. The key-value pairs belong to the device, are only read, and the value is copied at once. |
| `llama::watchdog` (unix) | `libc::fork`, then `close`, `sleep`, `getppid`, `kill`, `_exit` in the child | The child of a fork in a threaded process makes only async-signal-safe calls, never returns into Rust, never allocates or locks. It closes fds 0 to 2 so the daemon still sees the worker's pipes close. |
| `llama::Server::drop` (unix) | `libc::kill`, `libc::waitpid` | The watchdog is this process's own unreaped child, so its pid cannot have been reused. It is signalled and reaped once. |
| `llama::kill_on_close` (Windows) | `CreateJobObjectW`, `SetInformationJobObject`, `AssignProcessToJobObject` | An anonymous job. `JOBOBJECT_EXTENDED_LIMIT_INFORMATION` is `repr(C)`, size-asserted at 144 bytes, and passed with its own size. The child's handle is alive while `child` is. On any failure the drop guard remains. |
| `cuda::record` (Linux) | `libc::dlopen`/`dlsym`/`dlclose`, `mem::transmute_copy`, `nvmlInit_v2`, `nvmlDeviceGetCount_v2`, `nvmlDeviceGetHandleByIndex_v2`, `nvmlDeviceGetName`, `nvmlDeviceGetMemoryInfo`, `nvmlSystemGetDriverVersion`, `nvmlShutdown` | Each symbol is looked up by a NUL-terminated name and transmuted to the signature in nvml.h only when it is non-null; a size assert checks it is pointer-sized. Every out-parameter is a live local of the size NVML is told: 96 bytes for the name, 80 for the driver version, three `u64`s for `nvmlMemory_t`. The device handle is used only between init and shutdown, and the library is closed after shutdown. The strings are read with `CStr::from_bytes_until_nul`, so nothing reads past the buffer. |
| `cuda::load` (Linux) | `libc::dlopen(RTLD_NOW\|RTLD_GLOBAL)`, `libc::dlerror` | Each path is a `CString` that outlives the call. Handles are deliberately never closed, because the libraries must live as long as the worker process. `dlerror` is read on the same thread straight after the failure, null-checked, and copied. |
| `cuda::append_cuda` | `CreateCUDAProviderOptions`, `SessionOptionsAppendExecutionProvider_CUDA_V2`, `ReleaseCUDAProviderOptions`, `GetErrorMessage`, `ReleaseStatus` | The options pointer is a live out-pointer that ORT fills and owns. It is released exactly once, whatever the append answered. The session-options pointer is the builder's own and is valid while the builder is. A non-null status has its message copied before it is released once. |
| `gpu::GpuSession::open` | `ort::api().HardwareDevice_VendorId` | The device pointer is one ORT handed out for this environment, alive as long as the environment, and the call only reads it. |
| `gpu::platform::detect` (Linux) | `libc::dlopen`/`dlclose` of `libvulkan.so.1` | A NUL-terminated name; the handle is closed straight away and nothing is looked up through it. |
| `embed::check_cache_dir` | `libc::getuid` | Reads this process's own uid and cannot fail. |
| `system::sysctl` | `libc::sysctlbyname` | A NUL-terminated C string, and a `u64` whose size the call is told and will not exceed. macOS only. |
| `system::macos_available_mb` | `libc::host_statistics64`, `libc::mach_host_self`, `libc::sysconf`, `mem::zeroed` | The struct is zeroed and its size passed as a count the call will not exceed; the host port is this process's own and is not retained. `mach_host_self` is deprecated in libc 0.2 in favour of the `mach2` crate, which is a dependency this release does not take for three lines — the deprecation is allowed at the call with its reason. macOS only. |
| `system::platform` (windows) | `GlobalMemoryStatusEx` | A zeroed `MEMORYSTATUSEX` whose `dwLength` is set to its own size before the call, which is the documented contract. Windows only. |
| `daemon::owner_only` | `libc::getuid` | As above. |
| `daemon::alive` | `libc::kill(pid, 0)` | Signal 0 delivers nothing; the documented liveness probe. |
| `upgrade::writable` | `libc::access` | Two `CString`s that outlive the call; `access` reads them and returns. |
| `watch::on_signal` | `libc::signal` | Installs and restores a handler that only stores into an `AtomicBool`, which is what an async-signal-safe handler may do. |
| `main::quiet_on_a_closed_pipe` | `libc::signal` | Restores `SIGPIPE` to `SIG_DFL` — the disposition the process would have had if the Rust runtime had not changed it — once, at the top of `main`, before any thread is spawned and before anything is printed. unix only. |
| `main`'s `--airgap` | `std::env::set_var` | Before any thread is spawned. |
| `main`'s umask (0.41.0) | `libc::umask(0o077)` | Sets this process's file-creation mask, once, at the top of `main` before any thread is spawned; it takes and returns an integer. unix only. |
| `daemon::stdout_is_a_log` (0.41.0) | `libc::fstat`, `mem::zeroed` | This process's own stdout descriptor and a zeroed live `stat` the call fills; only `st_mode` is read. unix only. |
| `tests/security/s13.rs` | `libc::mkfifo` | A `CString` path inside the test's own temporary directory, outliving the call. unix only. |
| `tests/system_reading.rs` (windows) | `GlobalMemoryStatusEx` | As `system::platform`: a zeroed `MEMORYSTATUSEX` with `dwLength` set to its own size. Windows only. |
| `worker::Counting` | `GlobalAlloc` over `System` | Every call forwards the caller's own layout and pointer to `System` unchanged; the count only decides whether to call it, and a refused allocation returns null, the documented failure. The limit is non-zero only inside a parse worker. |
| `worker::limit_memory` (Linux) | `libc::setrlimit(RLIMIT_AS)` | A live local `rlimit` read by the call. Lowers this process's own limit, once, before any request is read. Linux only. |
| `worker::footprint` (macOS, 0.46.0) | `libc::proc_pid_rusage(RUSAGE_INFO_V4)`, `mem::zeroed` | A zeroed live local `rusage_info_v4`, all integers, passed with the flavour that writes exactly that struct; nothing is read through a pointer afterwards. The pid is this process's own unreaped child, so it cannot have been reused. macOS only. |
| `llama::job` (Windows) | as `llama::kill_on_close`, plus `JOB_OBJECT_LIMIT_PROCESS_MEMORY` | The same structure with `process_memory_limit` set; the parse worker's job uses it to cap the C half of a parser. |
| `tests/parse_worker.rs` | `std::env::set_var` | The only test in its binary, before any worker is spawned or any other thread reads the environment. |
| `tests/cuda.rs` (Linux x86_64), `accel::tests` | `std::env::set_var`/`remove_var` | Only that test in its binary reads the variable, and it is removed straight after. |
| `tests/index_failure.rs`, `tests/priority.rs` | `std::env::set_var` | The fault variable is read only by the index pass the same test runs on its own thread, and removed afterwards. |
| Test helpers in `embed`, `home`, `setup`, `upgrade` | `std::env::set_var` | Each guarded by a mutex that every test reading the variable also takes, and each restores what was there. |
None of them holds a raw pointer across a call, and only `worker::footprint` is
on a path that runs per file: the parent reads a macOS worker's footprint every
25 ms while it parses, one syscall into a local.