AGENTS.md git:20260924.30b3473A

Immutable. This exact content is served forever at /api/v1/blob/d49be1bfe771c718.

# AGENTS.md

Guidance for AI coding agents working on the Soliplex project. Human
contributors should read [DEVELOPMENT.md](DEVELOPMENT.md), which covers the
same ground in prose. (`CLAUDE.md` is a thin stub that imports this file so
Claude Code loads it automatically.)

## Project Overview

Soliplex is an AI-powered RAG system with a FastAPI backend, Flutter web
frontend, and terminal UI. It provides semantic document retrieval,
multi-room chat, and multi-provider LLM support.

**This repository contains the Python backend and TUI only.** The Flutter
frontend lives in a sibling repo at <https://github.com/soliplex/frontend>
with its own tooling.

## Build and Test

```bash
# Install dependencies (use uv, not pip)
uv sync --group dev

# Run unit tests with 100% coverage requirement
uv run pytest

# Run a specific test file / test.  '--no-cov' is required: 'addopts'
# always applies the 100% gate, which a partial run cannot satisfy.
uv run pytest --no-cov tests/unit/test_agents.py
uv run pytest --no-cov tests/unit/test_agents.py::test_name

# Run the unit tests in parallel (pytest-xdist); see below for '-n'
uv run pytest -n 8

# Run functional tests (require a running LLM); never pass '-n' here
uv run pytest --no-cov tests/functional/ -m needs_llm

# Lint and format
uv run ruff check
uv run ruff format --check

# Auto-fix lint and format issues
uv run ruff check --fix
uv run ruff format

# Start a dev server (no auth)
uv run soliplex-cli serve example/minimal.yaml --no-auth-mode

# Validate a configuration
uv run soliplex-cli audit example/minimal.yaml
```

## Code Style

- Line length: 79 characters
- Single-line imports enforced (isort via ruff)
- Ruff rule sets: F, E, B, U, I, PD, TRY, PT
- Target version: Python 3.13
- Use `uv run` to execute all Python commands

## Pre-commit Hooks

Optional: `uv run pre-commit install` automates the CI checks before each
commit (`uv run pre-commit run --all-files` runs them once against the whole
tree). The configured hooks (see `.pre-commit-config.yaml`) enforce:

- `ruff-check` / `ruff-format` -- lint and format Python sources
- `pymarkdown` -- lint Markdown files
- `agentskills-validate` -- validate the published `soliplex-docs` agent
  skill; runs only when something under `skills/soliplex-docs/` changes
- `lint-textio` -- reject text file IO in `src/soliplex/` without an
  explicit `encoding=` (falls back to the host locale encoding, `cp1252` on
  Windows); `scripts/lint_textio.py`, stdlib-only
- `lint-textio-self-test` -- run `scripts/lint_textio.py --self-test` when
  that script itself changes, so a check gone blind fails loudly instead of
  passing by finding nothing
- `actionlint` -- lint GitHub Actions workflow files
- `check-toml` / `check-yaml` -- validate TOML and YAML syntax
- `gitleaks` -- scan for committed secrets
- `pip-audit` -- scan dependencies for known vulnerabilities (runs when
  `pyproject.toml`, `uv.lock`, or `.pre-commit-config.yaml` changes)
- `debug-statements` -- reject leftover `pdb` / `breakpoint()` calls
- `trailing-whitespace` / `end-of-file-fixer` -- normalize whitespace
- `check-merge-conflict` -- reject unresolved merge-conflict markers
- `no-commit-to-branch` -- block direct commits to `main` / `master`

## Testing Requirements

- Unit tests live in `tests/unit/`, mirroring the `src/soliplex/` structure
- 100% branch coverage is enforced via pytest-cov (`--cov-fail-under=100`)
- Coverage measures four targets (see `addopts` in `pyproject.toml`):
  `src/soliplex`, `tests/unit`, `scripts`, and
  `skills/soliplex-docs/scripts` -- the test suite and the helper scripts
  are held to the same 100% bar as `src/`
- Those targets are *paths*, not importable names, and must stay that way.
  `soliplex` is a namespace package (there is no
  `src/soliplex/__init__.py`); coverage cannot enumerate one by walking the
  filesystem, so `--cov=soliplex` would measure only the modules some test
  happened to import, and a module nobody imports would pass unnoticed
- `[tool.coverage.run] omit` is therefore the single place that decides
  what is exempt: `scripts/lint_textio.py` (it runs its own `--self-test`
  during lint) and `src/soliplex/tui/*` (the TUI is deliberately
  untested). Everything else under `src/soliplex/` must reach 100%
- Use pytest-asyncio for async tests
- Functional tests (`tests/functional/`) require a running LLM and are
  skipped by default (marker: `needs_llm`)
- `--cov-fail-under=100` lives in `addopts`, so it applies to *every*
  `pytest` invocation, not just full runs. Pass `--no-cov` when running a
  subset -- a single file, a single test, or the functional suite --
  otherwise the run fails on the threshold no matter how the tests
  themselves fared
- `pytest-xdist` is a dev dependency, but no `-n` appears in `addopts`, so
  runs are serial unless you ask for parallelism
- **Unit tests are parallel-safe.** Choosing `-n`: start from `nproc` and
  take about half the logical cores (so `-n 8` on a 16-core box). Each
  worker pays a fixed startup and collection cost, so the gain flattens
  well before one-worker-per-core, and leaving half the cores free keeps
  the machine usable while the suite runs. Tune from there if a run feels
  slow -- the best value is machine-specific
- `-n auto` means one worker per logical core. It is the right choice only
  where the core count is not known ahead of time, which is why CI uses it
  and you generally should not
- **Functional tests must stay serial -- never pass `-n` to them.** They
  share on-disk state and module-scoped app fixtures: e.g.
  `tests/functional/test_sandbox_workdirs.py` creates, and on teardown
  `rmtree`s, a workdir tree keyed by a constant `ROOM_ID`, so parallel
  workers delete each other's fixtures. CI runs them in a separate,
  `-n`-less step
- Coverage and xdist compose (pytest-cov merges the workers' data), so
  `-n` does not weaken the 100% gate
- **The suite runs on Windows and macOS too.** The one exception is
  the bubblewrap sandbox, which is Linux-only, and so are the symlinks,
  FIFOs and `O_NOFOLLOW` opens its tests set up. The two modules
  covering it (`test_bwrap_sandbox.py` under `tests/unit/skills/`, and
  `test_sandbox_workdirs.py` under `tests/unit/views/`) therefore carry
  `pytestmark = _platform.requires_posix_sandbox` and skip whole off
  POSIX
- The 100% gate still applies off POSIX. `tests/conftest.py` drops the
  two skipped modules, and the two they cover, from the coverage
  *report* (`tests/_platform.py: POSIX_ONLY_COVERAGE_OMIT`), warning
  that it has -- so everything the host can measure is still held to
  100%, and only the sandbox needs re-checking on Linux
- CI checks that: `.github/workflows/python-test.yaml` runs the unit
  suite on `windows-latest` as well, on Python 3.13 only, with the
  functional step skipped there. The 100% gate applies to that job like
  any other
- Never loosen an assertion to make a platform pass. An assertion on a
  rendered path should build its expectation with `pathlib` so it holds
  on either separator; a test that needs a POSIX-only primitive and is
  *not* about the sandbox needs a gate of its own rather than a weaker
  check

## Repository Structure

Non-obvious modules and directories (the rest are self-explanatory from
their filenames -- `ls src/soliplex/` for the full layout):

- `agui/` -- AG-UI protocol (threads, runs, persistence)
- `authz/` -- authorization policy engine
- `config/` -- YAML config parsing (16 modules; see `installation.py` for
  the top-level entry)
- `tools/` -- agent tools (RAG, feedback, file uploads)
- `agents.py` -- Pydantic AI agent creation
- `completions.py` -- OpenAI-compatible streaming endpoint (not just
  LLM-level completions)
- `installation.py` -- installation lifespan, admin bootstrap, and global
  state management
- `main.py` -- FastAPI app factory (`create_app`)
- `tests/unit/` -- 100% coverage required; mirrors `src/soliplex/`
- `tests/functional/` -- tests requiring an LLM (marked `needs_llm`) are
  skipped by default; other functional tests run
- `example/` -- sample configs (rooms, completions, oidc, quizzes, skills)
- `schemas/` -- AG-UI feature JSON schemas

Key files:

- `pyproject.toml` -- dependencies, scripts, tool config
- `src/soliplex/config/installation.py` -- master config parsing
- `src/soliplex/main.py` -- FastAPI app factory
- `example/installation.yaml` -- full config example
- `example/minimal.yaml` -- minimal config for development
- `.env.example` -- environment variable reference

## Configuration System

- YAML-based hierarchical config in `src/soliplex/config/` (16 modules)
- Top-level entry: `InstallationConfig` in `config/installation.py`
- Config classes use dataclasses with a `from_yaml` classmethod
- Private fields `_installation_config` and `_config_path` carry context so
  nested configs can resolve env vars, secrets, and paths relative to the
  config file without threading them through every `from_yaml` call
- Environment variables resolved via `Installation.get_environment()`
- Secrets resolved via a configurable source chain (env vars, files,
  subprocess, random generation) in `config/secrets.py`

## Database Migrations

The operator- and developer-facing reference is
[docs/server/migrations.md](docs/server/migrations.md) (published as
"Database Migrations"); what follows is the part that constrains how code
in this repository is written.

Alembic drives **two** databases from one revision tree, which lives in the
package -- `src/soliplex/alembic_migrations/` -- and therefore ships in the
wheel, so a deployment can migrate from its own image:

- `agui` -- `thread_persistence_db`, schema `soliplex.agui.schema`
- `authz` -- `authorization_db`, schema `soliplex.authz.schema`

Every revision has `upgrade_agui()` / `upgrade_authz()` pairs behind an
`upgrade(engine_name)` dispatcher.

**The revisions are the schema.** a writable open brings its database to the
alembic head, which creates it when it does not exist and leaves it
stamped. That is only safe because migrating an empty
database from base reproduces the models exactly --
`scripts/lint_alembic_chain.py` proves it on every CI run, and its
`--self-test` proves the comparison is not blind.

`soliplex.alembic_migrations` is the API: `ensure_current_engine(engine,
database, *, sole_writer)` for an async engine, and the
`ensure_current_connection()` it runs on the caller's connection. One
database at a time, and four situations:

- at head -- returns, having written nothing (one `SELECT`);
- empty, or behind head -- migrates, **but only when this process is the sole
  writer**. `serve --workers N` (N > 1) sets `_SOLIPLEX_MULTIPLE_WRITERS` in
  the private env-var contract, so no worker migrates and the app raises
  `MigrationRequired` instead; migrate ahead of time with every writer
  stopped;
- tables but no `alembic_version` row -- raises `UnstampedDatabase`, naming
  the one-off bootstrap script (issue #1367). Such a database was built by
  soliplex <= 0.81, when nothing stamped; see #1368 for the promise that no
  later release leaves one that way;
- stamped at a revision `knows_revision()` does not find -- raises
  `DowngradeRequired`. The code was rolled back without downgrading its
  databases first, and this release cannot move them: the revisions between
  its head and that stamp exist only in the newer release's tree. Checked
  *before* the sole-writer gate, because stopping writers cannot help;
- behind head, with a `migration_policy` in force -- raises
  `ExplicitMigrationRequired` or `MigrationsDisabled`. The caller resolves
  the policy with `migration_policy()` and passes it as the required
  `policy=` keyword; `_POLICY_REFUSAL` maps it to the refusal. Also checked
  before the sole-writer gate, and for the same shape of reason: when the
  configuration says this process never migrates, stopping the other
  writers is not the remedy. Nothing is refused for a database already at
  head, so a `disabled` service starts normally against a current one.

Alongside those: `head_revision()`, `database_state()`, `upgrade()` /
`downgrade()` (named databases; `sql=True` emits `<database>.sql` instead
of running anything), `revision_chain()` / `split_chain()` for reporting
what a database has applied and what is pending, and `migration_dburi()` /
`migration_policy()` for which credential a migration uses and whether one
is allowed here at all.

Every writable open goes through one of two helpers, which own the
migration and the engine's lifetime: `installation.open_engines` (for
`lifespan`) and `cli_util.open_db` (the pre-flight for every `admin-users`
/ `room-authz` command, and for `cli/ask.py`). `open_db` also owns the CLI
policy an in-memory DBURI needs -- `alembic_migrations` knows nothing about
RAM databases. `audit` is a reader: it passes `must_exist=True`, so nothing
is created or migrated, and reports an uncreated database as "nothing
configured". `audit databases` probes the runtime async DBURI that way when
one is configured, and otherwise the migration DBURI over a sync engine,
through the `cli_util.probe_database` the `database` group uses. The
`database` group goes through neither: it opens a sync engine on the
migration DBURI itself, because reporting must create and migrate nothing,
and the credential it uses is not `open_db`'s.

```bash
# Upgrade both databases
uv run alembic -x soliplex.installation_path=<dir> upgrade head

# Emit SQL instead: writes 'agui.sql' and 'authz.sql' into the cwd, and
# never connects
uv run alembic -x soliplex.installation_path=<dir> upgrade head --sql

# New revision (per-database stubs from the tree's 'script.py.mako')
uv run alembic -x soliplex.installation_path=<dir> revision -m "soliplex-vX.Y"
```

**Those are checkout-only.** `script_location` lives in `[tool.alembic]` in
`pyproject.toml`, which no deployment image carries, so the bare `alembic`
CLI there fails with `No 'script_location' key found in configuration`. A
deployment migrates through soliplex's own writable open, or through the
`soliplex-cli database` group (`status` / `upgrade` / `downgrade`, in
`cli/database.py`), both of which set `script_location` from the package
directory. `migration_dburi` is read only by the deliberate tools -- that
group, and `scripts/bootstrap_alembic_version.py` -- and, to report on it,
by `audit databases` when no runtime async DBURI is configured; the
`migration_policy` is enforced in both places, by `ensure_current_*` (see
below).

- The DB URIs come from the installation config rather than from any alembic
  config file, so `-x soliplex.installation_path=` is mandatory; without it
  `env.py` prints its usage and exits 2. `resolve_dburis()` goes through
  `load_installation_config()`, which is `load_installation()` plus
  `resolve_environment()` -- all the DBURIs depend on, and nothing more, so
  an unrelated broken room or OIDC config cannot block a migration. The
  `database` CLI group and `audit databases` load their config the same
  way, for the same reason
- In-process callers pass either an explicit `{name: dburi}` mapping or a
  live `connection` and the `database` it belongs to through
  `config.attributes`. The connection form is what lets an in-memory
  database be migrated at all: it lives inside one engine, so the migration
  has to run on the caller's own connection
- `env.py` is a four-line shim over `alembic_migrations.run()`, and builds
  its engines without creating anything, so `current`, `history`, `check`
  and `--sql` leave the schema alone. That is what makes `alembic check`
  meaningful: it compares the models against the migrated database
- Revisions must not import live app code: freeze any borrowed helper into
  the revision, so replaying history never depends on the current tree
- Mode-dependent logic splits into `_x_online()` / `_x_offline()` behind a
  `context.is_offline_mode()` dispatcher; offline has no bind, so express it
  as set-based `op.execute()` SQL
- Branch on `op.get_context().dialect.name` where SQLite and PostgreSQL
  differ
- `DATABASE_NAMES` in the package names the two databases; alembic has no
  `databases` option of its own to read. `script.py.mako` imports it to
  emit one `upgrade_<name>()` / `downgrade_<name>()` pair per database, and
  the chain lint builds both databases from it, while `env.py` iterates
  whatever DBURIs `resolve_dburis()` hands it. There is no `alembic.ini`
  at all:
  `script_location`, `prepend_sys_path` and the `ruff` `post_write_hooks`
  entry live in `[tool.alembic]` in `pyproject.toml`, and
  `configure_logging()` skips `fileConfig()` unless the file alembic names
  exists
- Only `src/soliplex/alembic_migrations/versions/*` is exempt from the 100%
  coverage bar. The package's `__init__.py` *and* `env.py` are held to it,
  which is why the tests drive alembic for real against throwaway SQLite
  files rather than mocking it

## Adding a New Tool

1. Create or modify a tool module in `src/soliplex/tools/`
2. Tool functions are async and accept `RunContext[AgentDependencies]`
3. If the tool needs configuration, add a `ToolConfig` subclass in
   `config/tools.py`
4. Register it in `TOOL_CONFIG_CLASSES_BY_TOOL_NAME` (found in
   `config/tools.py` and `config/meta.py`)
5. Reference the tool in room configuration under `agent.tools`

## Adding a New Room

1. Create `example/rooms/<room_id>/room_config.yaml`
2. Required fields: `id`, `name`, `description`, `agent`
3. Optionally add `prompt.txt` for an external system prompt

## Adding API Endpoints

1. Create or modify a router in `src/soliplex/views/`
2. Register the router in `main.py` with the appropriate prefix
3. Add unit tests achieving 100% branch coverage

## Key Architecture

- FastAPI app created via `create_app()` in `main.py`
- Rooms contain agents, each with tools, skills, and an LLM provider config
- AG-UI protocol handles thread/run lifecycle with SSE event streaming
- Authorization via a policy engine in `authz/`
- MCP server exposes Soliplex tools; MCP client consumes external tool servers
- Authentication via OIDC/JWT in `authn.py`
- Public API models defined in `models.py`

## Key Dependencies

See `pyproject.toml` for authoritative version constraints.

- FastAPI / Uvicorn -- REST API and ASGI server
- pydantic-ai-slim[google] -- agent framework
- haiku.rag-slim -- RAG functionality
- FastMCP -- Model Context Protocol
- ag-ui-protocol -- AG-UI event protocol
- SQLModel / aiosqlite -- database ORM
- haiku-skills -- Haiku skills framework

## Entry Points

- `soliplex-cli` -- backend CLI; run `soliplex-cli --help` for the full
  command list
- `soliplex-tui` -- terminal UI client
- `soliplex-tui-serve` -- TUI server

## Environment Variables

See `.env.example` for the full reference. Key variables:

- `OLLAMA_BASE_URL` -- Ollama server URL (without `/v1` suffix)
- `OPENAI_API_KEY` / `GEMINI_API_KEY` -- LLM provider keys
- `SOLIPLEX_URL_SAFE_TOKEN_SECRET` -- MCP token secret (auto-generated if
  unset)
- `LOGFIRE_TOKEN` -- Pydantic Logfire token (optional)
- `SOLIPLEX_CLI_LOG_CONFIG` -- path to a Python logging-config YAML enabling
  audit logging for privileged CLI commands (also the `--cli-log-config`
  group option on `admin-users` / `room-authz` / `audit`); unset means CLI
  audit records are suppressed (see `docs/config/logging.md`)

## Documentation

Detailed configuration and usage docs are in [docs/](docs/) (served via
Zensical). Example configurations are in [example/](example/).