CLAUDE.md · diff
git:20261027.6d22863 to git:20261028.c0a9088
13 added, 9 removed. Audit A to A.
# CLAUDE.md
# Claude Code instructions
When generating commit messages, pull request text, patches, or any code-related output, never include this line:
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Omit any Claude co-author trailer unless I explicitly ask for it.
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
# Splitting tasks
Split big tasks into separate worktrees and do in parallel, but TDD takes precedence.
Each worktree agent must follow TDD internally: write its tests first (red), then implement (green).
If multiple worktrees touch independent features, each worktree owns its own red-green-refactor cycle.
Each worktree agent should write ~200-300 lines max per file. If writing more, split into additional worktrees.
When finished - always try /simplify on the change.
## Timeout handling
If you see API timeouts, split the current task into smaller parallel worktrees rather than retrying.
## Worktree hygiene
- Never push worktree branches to remote.
- When a worktree agent finishes, immediately remove the worktree (`git worktree remove --force`) and delete its local branch (`git branch -D`).
- Before starting new work, check for and clean up any stale worktrees from previous sessions (`git worktree list`).
## Permissions hygiene
- Do not accumulate one-off permission rules in `.claude/settings.local.json`.
- Prefer broad wildcards (e.g., `Bash(git:*)`) over specific subcommand rules.
- Keep the allow list under 30 entries.
## Code practices
- Keep functions small and single-purpose; prefer clear names over comments.
- Don't add abstractions, config flags, or error handling for cases that can't happen — match the scope of the change to what was actually asked.
- Reuse existing helpers/patterns in the codebase instead of duplicating logic.
- Keep diffs minimal and focused; don't refactor unrelated code in the same change.
- Run the relevant tests (see Running Tests) before considering a change done.
## Test-driven development (required)
All feature work and bug fixes must follow strict TDD:
1. Write the test first. It must fail for the right reason (red).
2. Write the minimum implementation code needed to make it pass (green). Do not modify the test to make it pass.
3. Refactor implementation code only, keeping tests green.
Rules:
- Never edit a test to force a passing result — if the test seems wrong, stop and ask before touching it.
- Do not write implementation code before its test exists.
- Each new behavior gets a test asserting it before any code implements it.
## Project Overview
vmn is a CLI tool and Python library for automatic semantic versioning. Versions live in git annotated tags — zero lock-in, zero databases.
Key differentiators vs semantic-release/release-please:
- Language-agnostic (not JS-centric)
- Multi-repo dependency tracking with `vmn goto` state recovery
- Microservice topology (root apps with independent service versions)
- 4-segment hotfix versioning (`major.minor.patch.hotfix`)
- Auto-init on first `vmn stamp` — no separate `vmn init` required
- Works offline, with shallow clones, in air-gapped environments
## Development Setup
The repo is a uv workspace of three packages (see docs/packaging.md):
`packages/vmn` (`version_stamp`), `packages/vmn-exp-sdk` (the job-side
`vmn_exp.sdk`/`storage`/`core`/`registry`/`integrations`/`_base`) and
`packages/vmn-exp` (`vmn_exp.cli`/`ui`/`snapshot`/`importers`/`gitmode`, the
`vmn-exp` command and `webui/`). `vmn_exp` is a PEP 420 namespace package split
across the last two — never add a `vmn_exp/__init__.py`.
```sh
python3 -m venv ./venv
source ./venv/bin/activate
pip install -U pip "setuptools>=64"
pip install -r ./tests/requirements.txt
pip install -r ./packages/vmn-exp/tests/test_requirements.txt # core + vmn-exp test deps
pip install -e packages/vmn -e packages/vmn-exp-sdk -e "packages/vmn-exp[ui]"
make _build_ui # the web UI bundle (vmn_exp/ui/static, export_bundle) is git-ignored build output; needs npm
vmn --version # e.g. 0.10.2-rc.11+dev: a source checkout adds +dev
```
`uv sync` also works from the repo root. Build the wheels with
`uv build --all-packages` (or `make _build NAME=vmn|vmn_exp`, which builds the
web UI first; a bare `uv build` ships whatever bundle is on disk).
Import rules (packages/vmn-exp/tests/test_packaging_split.py): `version_stamp` never imports
`vmn_exp`; the vmn-exp-sdk subpackages import neither `version_stamp` nor the
vmn-exp subpackages (except a lazy `vmn_exp.gitmode`); vmn-exp reaches
`version_stamp` only through `version_stamp.api`. vmn loads `vmn goto` of dev
versions (`register_dev_version`) from the `vmn.plugins` entry point that
vmn-exp declares.
## Running Tests
Two suites (vmn-exp will become a separate product):
- **core** — `tests/`: vmn alone. `tests/conftest.py` puts only `packages/vmn/src`
on `sys.path` and makes `vmn_exp` unimportable (`tests/no_vmn_exp.py`), so it
passes without vmn-exp installed; deps in `tests/test_requirements.txt`.
- **exp** — `packages/vmn-exp/tests/`: vmn-exp. Its conftest adds all three
`src/` roots plus `tests/`, reusing the core fixtures (`tests/vmn_fixtures.py`)
and helpers (`exp_helpers.py` = `tests/helpers.py` + `_experiment`/`_exp`/
`_storage`/`cli_module`); deps in `packages/vmn-exp/tests/test_requirements.txt`
(includes the core file). A test that needs `vmn_exp` or a vmn-exp command
belongs here; one asserting vmn works *without* vmn-exp belongs in core.
Tests require Docker. Activate the test venv first — the suites run through the
active interpreter's `coverage`/`pytest`, and the release-notes tests need the
`git-cliff` binary that `tests/test_requirements.txt` installs into it.
```sh
source ./venv/bin/activate
./tests/run_pytest.sh # core suite (= --suite core)
./packages/vmn-exp/tests/run_pytest.sh # exp suite (= tests/run_pytest.sh --suite exp)
```
Run a specific test:
```sh
./tests/run_pytest.sh --specific_test <test_name>
./packages/vmn-exp/tests/run_pytest.sh --specific_test <test_name>
```
UI load harness (real `vmn-exp ui` + live job processes; see packages/vmn-exp/tests/uiload/README.md):
```sh
python packages/vmn-exp/tests/uiload/run.py live --profile smoke --duration 0 # watch in a browser
VMN_UILOAD_PROFILE=smoke python -m pytest -s packages/vmn-exp/tests/test_uiload_profile.py
```
CI is the local Muster pipeline in `ci/pipeline.py` (`./ci/start.sh`, see
`ci/README.md`) — there is no GitHub Actions workflow.
Skip a test:
```sh
./tests/run_pytest.sh --skip_test <test_name>
```
Tests run in parallel (29 workers by default) using pytest-xdist. Timing benchmarks carry
`@pytest.mark.serial`: `run_pytest.sh` (and the CI `tests_exp` stage) run them alone in a
second, one-process pass, since under 29 busy workers they measure contention.
### Key Concepts
- **App name**: Identifier for a versioned app (e.g., `my_app` or `root_app/service1`). Cannot contain `-` or start with `/`
- **Root app**: Parent container for microservices, format `root_app/service_name`. Root version is an auto-incrementing integer.
- **Version format**: `major.minor.patch[.hotfix][-prerelease.rcn][+buildmetadata]`
- **Tag format**: `{app_name}_{version}` where `/` in app names becomes `-`
### Data Flow
1. Version info stored in git annotated tag messages as YAML (`vmn_info` + `stamping` sections with changesets). Changeset `remote`s carry no credentials (`core/utils.py:redact_changesets`: `https://user:token@host/x` → `https://host/x`, ssh keeps its user); `vmn goto` cloning a dep from such a URL borrows the checkout's own remote's credentials on the same scheme+host (`borrow_credentials`).
2. Local state tracked in `.vmn/{app_name}/last_known_app_version.yml`
3. `stamp` command: increments version → writes to backends → commits → tags → pushes
### Configuration
Per-app config in `.vmn/{app_name}/conf.yml`. Key fields:
- `template`: Version display format (e.g., `[{major}][.{minor}]`)
- `conventional_commits`: Auto-detect release mode from commit messages (`fix:` → patch, `feat:` → minor, `BREAKING CHANGE` → major). When enabled, `-r` flag is optional.
- `release_mode_policy`: `optional` (--orm behavior) or `strict` (-r behavior) — controls how detected release mode is applied
- `default_release_mode`: Fallback release mode (`patch`/`minor`/`major`/`hotfix`) when none is resolved from CLI or conventional commits
- `changelog.path`: Generate CHANGELOG.md on stamp (requires conventional_commits)
- `github_release.draft`: Create GitHub Release on stamp (requires `gh` CLI + `GITHUB_TOKEN`)
- `deps`: External repository dependencies for multi-repo tracking
- `version_backends`: Auto-embed version into package.json, Cargo.toml, pyproject.toml, or any file via regex/Jinja2
- `policies.whitelist_release_branches`: Restrict which branches can stamp/release
- Branch-specific overrides: canonical layout `.vmn/{app}/branch_conf/{branch}/conf.yml` (branch slashes become real directories; root apps use `root_conf.yml`). Legacy flat `<branch>_conf.yml` and nested `{branch}/conf.yml` files are still read (precedence canonical > flat > legacy) and are auto-migrated to the canonical layout on the next `vmn stamp`.
### Test Infrastructure
- `tests/vmn_fixtures.py`: Pytest fixtures including `FSAppLayoutFixture` for creating isolated git repos, registered by both suites' `conftest.py`
- Tests create temporary git repos with remotes to simulate real workflows
## CLI Commands
- `vmn stamp -r <mode> <name>`: Stamp a new version (mode: major/minor/patch/hotfix). Auto-inits repo/app. Idempotent.
- `--pr <id>`: Create prerelease (e.g., `--pr rc` → `0.0.1-rc.1`)
- `--orm`: Optional release mode — only advances if no prerelease exists at target
- `--pull`: Pull remote first, retry on conflict
- `--dry-run`: Preview without committing
- Without `-r`: works during prerelease sequence, or always with `conventional_commits` enabled
- `--git-push-user` / `--git-push-token`: push credentials for hosts where the checkout has none (fall back to `VMN_GIT_PUSH_USER` / `VMN_GIT_PUSH_TOKEN`). Both required together; injected into an ephemeral HTTPS push URL only, git remote config is left untouched.
- `vmn release <name>`: Promote prerelease to final. `-v <version>` for explicit, `--stamp` for full stamp flow. Also takes the `--git-push-*` flags.
- `vmn show <name>`: Display version info. `--verbose` for full YAML, `--raw`, `--base` (`major.minor.patch[.hotfix]` only), `--type`, `-u` for unique ID, `--dev`, `--conf`, `--from-file`, `--ignore-dirty`, `-t <template>`.
- `vmn goto -v <version> <name>`: Checkout repo + all deps to exact state at version. Without `-v`, returns to a branch tip via `ensure_on_branch()` (`backends/branch_checkout.py`); a detached HEAD's branch comes from `backends/branch_resolver.py` (CI env hint > branch last checked out > unique tip > HEAD version's `stamped_on_remote_branch` > unique container > alphabetical + warning). Building a backend never creates/switches branches (branch state is resolved lazily, `backends/branch_state.py`); commands that commit declare `commits` in their `CommandSpec` (`cli/constants.py`, `cli/plugin_api.py`; built-ins `stamp`/`init`/`init-app`/`release --stamp`) and the CLI then calls the stamper's `vcs.ensure_on_branch()` (reloads conf when the branch changes), as do goto-back and the vmn-exp cold start / `vmn-exp` auto-init. Real branch names only; a legacy `vmn_tracking_remote__*` branch is created only when the local branch diverged. `active_branch` is the branch identity (a legacy branch counts as its upstream), `local_branch` the git ref (push refspec, checkout, outgoing range). `release`/`add` push only their tag. `tests/test_git_process_budget.py` caps git processes for detached stamp/goto. `--deps-only`, `--root`, `--pull` (fetch first when the version is not local), `--force` (dev versions: restore despite untracked files over the snapshot caps). A dev-version goto and `vmn-exp restore` restore through `version_stamp.snapshot.restore.restore_record` (via `version_stamp.api`), like `vmn snapshot restore`: dirty work is saved as a thin snapshot record first (hint `vmn goto -v <saved> <app>`).
- `vmn gen -t <template> -o <output> <name>`: Generate file from Jinja2 template.
- `vmn add -v <version> --bm <metadata> <name>`: Attach build metadata. `--vmp` for a metadata YAML path, `--vmu` for an associated URL.
- `vmn snapshot [create|list|show|note|delete|restore|export|diff] <name>` (built into vmn, `cli/snapshot_cmd.py` + `version_stamp/snapshot/`; `create` is the default; app must be stamped): save/restore uncommitted work (tracked diff, local commits, untracked files, deps) as a deterministic dev verstr. Refs: `-v` verstr/unique prefix/`@N`/`latest`, or `--latest`; `show`/`export`/`diff` default to the latest, `note`/`delete`/`restore` require a ref. Flags: `--note`, `--meta k=v` (repeatable) / `--meta-file` (create), `--filter k=v`/`--last N`/`--verbose` (list), `--json` (list/show), `--to <snapshot ref|stamped version>` (diff; default current working state) and `--tool` (default git `diff.tool`), `-o` (export: dir or `.tar.gz`/`.tgz`, default `<verstr>.tar.gz`), `--force` (restore). Records are thin (`.vmn/store/snapshots/<app-key>/<verstr>/metadata.yml` with `code:`), referencing code objects shared with experiment runs (`.vmn/store/code/<app-key>/`); re-creating a saved state keeps its verstr/timestamp; `delete` drops the code object only when no snapshot or run uses it. `restore` auto-saves dirty work first (note `auto-saved before restore`, prints the restore command), refuses without `--force` when untracked files over the size caps would be deleted, then resets detached to the base commit and applies patches (missing deps are skipped with a warning). `list`/`show`/`diff`/`export` are lock-free (`constants.READ_ONLY_ACTIONS`). With vmn-exp installed: `--store <uri>` > `VMN_EXPERIMENT_STORE` > conf `experiment.storage.uri` put records in that store's `snapshots/` area and code objects in its `code/` area (`vmn_exp/snapshot/stores.py`, registered via `plugin_api.register_snapshot_store_opener`); `--local` overrides; `VMN_EXP_OFFLINE` keeps them local; without vmn-exp `--store` errors. `vmn goto -v <snapshot verstr>` works (vmn-exp's dev-version loader, `cli/code_record.py`). `vmn-exp snapshot` exits 2 pointing at `vmn snapshot`; the stamp dirty-tree hint points at `vmn snapshot create`. See docs/snapshots.md.
- `vmn-exp [action] <name>` (the `vmn-exp` command; a leading `exp`/`experiment` is accepted): Local-first experiment tracking (a snapshot + an append-only metrics/notes log). Actions: `create` (default), `run`, `add`, `list`, `show`, `compare`, `diff`, `restore`, `export`, `prune`, `tag`, `archive`, `unarchive`, `import-mlflow`, `watch`, `importance`, `lineage`, `rewind`, `rerun`, `push`, `migrate`, `compact`, `export-metrics`, `similar`, `similar-backfill` (sweeps are the separate `vmn-exp sweep` command). `create`/`run --name` names a run; `list --query <expr>` filters, `list`/`show --json` print machine-readable output; archived runs are hidden unless `list --archived`. `vmn-exp run <name> -- <cmd>` runs a command and ingests `key=value` lines the child writes to `$VMN_METRICS_FILE`. See docs/vmn-exp/experiments.md.
- `run` publishes `run_state.yml` (state/pid/host/heartbeat/exit_code, `runner` `exp run`|`sdk`, `cwd` repo-relative or null) next to `metadata.yml` and refreshes a heartbeat while the child lives; `--heartbeat-interval <sec>` (default 30).
- `run` tees the child's stdout/stderr (pipes, not a pty; `PYTHONUNBUFFERED=1` unless set) into a size-capped `outputs/output.log` (head + tail kept past the cap; `--output-cap-mb` > `VMN_EXP_OUTPUT_CAP_MB` > 10), uploaded after the log sync, throttled to ~64 KB/s of upload bandwidth, and once unthrottled at the end; `--no-capture-output` opts out. SDK: `start_run(capture_output=True)` (off by default) tees fds 1/2 the same way.
- System metrics (`sys_*`: CPU/RAM, GPU with pynvml) are sampled on the heartbeat by default. Opt out: `--no-system-metrics` / `start_run(system_metrics=False)` > `VMN_SYSTEM_METRICS=0` > conf `experiment.system_metrics: false`. psutil is a hard dependency of vmn-exp-sdk. When `vmn-exp run` samples the child's tree it sets `VMN_EXP_SUPERVISOR_SAMPLES=1`, and `start_run()` inside the child then defaults `system_metrics` off (explicit `True` wins).
- Alerts: `run.alert(title, text, level)` (rate-limited per title), plus `failed`/`stuck` transition alerts to sinks in conf `experiment.alerts` (`on: [failed, stuck, alert]`, `sinks: webhook|slack|command`) or `VMN_EXP_ALERT_WEBHOOK_URL`/`_SLACK_URL`/`_COMMAND`/`_ON`. `failed` fires from the run itself; `vmn-exp watch <app> [--interval] [--within]` (cron-friendly) detects `stuck` and missed `failed`, deduped via the run's `alerts_sent.yml`. Sink errors never break a run.
- `vmn-exp run` supervision is crash-proof: the metrics tailer reads bytes (non-ASCII lines never kill it), ingestion/heartbeat/sync errors are best-effort, SIGTERM/SIGINT/SIGHUP are forwarded to the child (SIGKILL after `--kill-grace-sec`, default 30 / `VMN_EXP_KILL_GRACE_SEC`) and the final state is always written (`exit_code` = `128+N` when signal N ended the child; `signal`/`received_signal` fields). The child runs in the invocation cwd (or `$VMN_WORKING_DIR`). Remote sync runs off the heartbeat loop.
- Read-only actions (`list`/`show`/`compare`/`diff`/`export`/`watch`/`importance`/`lineage`/`export-metrics`/`similar`) never take the repo lock. `list` row numbers are the storage index `@N` resolves, whatever `--sort`/`--last`/`--top` do. `show` prints the last 50 log entries (`--full-log` for all); `--patch [--max-bytes N]` (default 200000) adds the run's own text patches, never the untracked tarball (`--json`: `patch` = `{text, truncated}` or `{text: null, reason: no_code|code_missing|untracked_only}`; no patch = a stderr note, exit 0). `prune` never deletes `running` or `stuck` runs, or a run carrying a `--protect-tag`-named tag key (`--force` overrides either), or a run with a kept descendant; `--dry-run` previews, `--local-only` keeps remote copies, `-v <ref>` (repeatable, not combined with `--keep`/`--older-than`) deletes exactly the named run(s) instead of a bulk policy. `--query <expr>` selects candidates via the query language (same rows as `list --query`); is a dry-run preview unless `--yes`/`-y` is given; `--dry-run` beats `--yes`; `--keep`/`--older-than` apply within the query scope; cannot be combined with `-v`.
- Metric summaries: the fold keeps last, first (earliest by fold key), finite min/max and an exact finite sum/count per repeated metric (`core/metric_summary.py`; index `SCHEMA_VERSION` bumps on any fold change). `row["metrics"][k]` is the summary per policy: run `define_metric(name, summary=min|max|last|first|mean, goal=, step_metric=, hidden=)` (exact name beats glob) > conf `experiment.metrics.<name|glob>.summary|goal|step_metric|hidden` > `last`; `goal` derives the summary; `summary="none"` is rejected. Sorting, queries, prune `--keep`, compare and the UI all rank on it; `row["metric_summary"][k]` holds `{last,min,max,first,mean}` for metrics logged more than once. The *effective schema* (`core/metric_schema.py`: conf + run-declared `goal`/`hidden` for names conf lacks, latest run wins; `IndexSnapshot.declared_schema()`) drives sort direction (`list_runs`, `vmn-exp list`, UI leaderboard; glob goals count) and UI hidden columns/`/metrics-schema`, never summaries. `hidden` is display-only; run detail carries `hidden_metrics`. CLI: `vmn-exp add <app> -v <ref> --define-metric NAME [--goal] [--summary] [--step-metric] [--hidden]`. `step_metric` joins a metric's series on another metric's value at the same step (reader `get_run(x=)`, series endpoints `x=`, UI x-axis picker).
- Record format: new runs and registry records carry `format_version` (`core/record_format.py`, currently 2); readers skip records from a newer format, and v1 records without it, with a warning.
- - Store layout v2 (docs/plans/14-store-layout.md): every store root (repo-local `<repo>/.vmn/store/` with one `.gitignore` of `*`, `file://`, buckets; URI without a path = root `vmn`) holds `store.yml` (`layout: 2`, created by the first writer) and areas `runs/ snapshots/ code/ sweeps/ registry/ reports/ comments/ journal/` (`storage/areas.py`), scoped by tag-form app keys (`/`→`-`). Records: `log/<stream>[@<seq>].jsonl` (stream = `<writer>~<owner>`, see Streams below), `outputs/` (`output.log`, `media/`, `tables/`), `artifacts/` (user files only). Unknown layouts and v1 stores (records, no `store.yml`) are refused with a migrate hint (`storage/store_marker.py`); writers also refuse while `migrating`. Per-host state lives outside the store (`storage/host_dirs.py`): index cache `$XDG_CACHE_HOME/vmn-exp/index/<store-id>/`, push ledger `$XDG_STATE_HOME/vmn-exp/push/<store-id>/` (macOS `~/Library/Caches|Application Support/vmn-exp`; `VMN_EXP_CACHE_DIR` replaces both bases). `vmn-exp migrate [--store <uri>|--dir <path>] [--dry-run] [--skip-live]` (`cli/migrate*.py`) converts v1 stores record by record (resumable; `migrating: true` meanwhile; running/stuck runs stop it unless `--skip-live`); without a flag it moves the checkout's `.vmn/<app>/experiments/` etc. into `.vmn/store/`.
+ - Store layout v2 (docs/plans/14-store-layout.md): every store root (repo-local `<repo>/.vmn/store/` with one `.gitignore` of `*`, `file://`, buckets; URI without a path = root `vmn`) holds `store.yml` (`layout: 2`, created by the first writer) and areas `runs/ snapshots/ code/ sweeps/ registry/ reports/ comments/ settings/ journal/` (`storage/areas.py`), scoped by tag-form app keys (`/`→`-`). Records: `log/<stream>[@<seq>].jsonl` (stream = `<writer>~<owner>`, see Streams below), `outputs/` (`output.log`, `media/`, `tables/`), `artifacts/` (user files only). Unknown layouts and v1 stores (records, no `store.yml`) are refused with a migrate hint (`storage/store_marker.py`); writers also refuse while `migrating`. Per-host state lives outside the store (`storage/host_dirs.py`): index cache `$XDG_CACHE_HOME/vmn-exp/index/<store-id>/`, push ledger `$XDG_STATE_HOME/vmn-exp/push/<store-id>/` (macOS `~/Library/Caches|Application Support/vmn-exp`; `VMN_EXP_CACHE_DIR` replaces both bases). `vmn-exp migrate [--store <uri>|--dir <path>] [--dry-run] [--skip-live]` (`cli/migrate*.py`) converts v1 stores record by record (resumable; `migrating: true` meanwhile; running/stuck runs stop it unless `--skip-live`); without a flag it moves the checkout's `.vmn/<app>/experiments/` etc. into `.vmn/store/`.
- Fork/rewind: `start_run(fork_from="<ref>", fork_step=N)` (or `"<ref>?_step=N"`; CLI `--fork-from/--fork-step`) makes a new run whose log starts with the source's entries up to N (marked `inherited`) and records `forked_from` (not a child). `start_run(run_id=<ref>, rewind_to_step=N)` appends a `rewind` marker; every reader hides earlier entries with step > N. CLI `vmn-exp rewind <app> -v <ref> --step N` appends the same marker without reopening the run (repo lock like `tag`; works with `VMN_SNAPSHOT_METADATA`; ui job action `exp_rewind`); `vmn-exp run` has no resume, so continue via SDK `run_id=` or `--fork-from`. A running run can't be rewound.
- Rerun (`cli/rerun.py`, `cli/rerun_report.py`, `cli/rerun_workdir.py`, pure `core/rerun.py`): `vmn-exp rerun <app> -v <ref>|--latest [-- <cmd>]` restores the run's code strictly into throwaway detached worktrees beside the repo (`$TMPDIR/vmn-rerun-<app>-*` or `--worktree-dir`; any restore failure = exit 1, nothing created), claims a record under the original's `code_verstr` that copies its code identity (shares the code object, keeps it alive through prune; ) with `rerun_of`, releases the repo lock and supervises like `run` in the recorded cwd (`--cwd` overrides; child gets `VMN_EXPERIMENT_DIR`=original store, `VMN_WORKING_DIR`=workspace cwd, sweep trials `VMN_SWEEP_PARAMS`). Workspace removed afterwards unless `--keep-worktree` (recorded as `workdir`). `--dry-run` prints the plan; `--print [--json]` prints command/cwd/code_verstr/code + `recipe`/`export_recipe` for cluster jobs (vmn does not schedule). Refused: no code, no recorded command, SDK runs without `--`, `--fork-from`, git-free mode. Not read-only. `rerun_of` is a row/query/detail field; `show` prints `Rerun of:`; the UI has links, no Rerun button.
- Offline + push: `VMN_EXP_OFFLINE` (`core/storage_resolve.py:drop_remote_if_offline`, `core/writer.py`) makes every storage factory (CLI, SDK, `vmn snapshot`) record to the local root only (error without one), naming new runs `<code_verstr>.<writer_id>[.N]`. `vmn-exp push <app> [-v <ref>...] [--store|--bucket/--prefix/--endpoint-url] [--dry-run] [--json] [--adopt]` (`cli/push.py`, `core/push_*.py`) uploads local runs (all, or the refs) to the remote store (flags > `VMN_EXPERIMENT_STORE` > conf; offline never hides it; `file://` and no remote are refused) under the same name, parents first: code object if missing, claim with the run identity as `create_exclusive(claim_token=)` (a crashed claim of ours resumes), other files by sha, logs per stream from the remote offset (exclusive `create_log_segment`), missing/resized metric objects (`core/push_metrics.py`: a `.vmx` via `put_indexed(covers=)`, dropping the remote segments it covers; else the local blocks past the remote ones as the next `create_metric_segment`), missing/resized artifacts, three-way `archived`/`note` merge. Resumable via a per-host ledger at `$XDG_STATE_HOME/vmn-exp/push/<store-id>/<remote_id>/` (`storage/host_dirs.py`). A remote record of another run under the name → rename on both sides to the next `<code_verstr>.rN` (`renamed_from`, children's `parent` follow); running/stuck, locally registered and sweep-outer runs are skipped instead. Prints `<verstr> <status>[ -> new][ (detail)]` lines then `pushed N, up-to-date M, renamed R, skipped S, failed F`; exit 1 if any failed. Takes the repo lock; works git-free. UI job action `exp_push` (body `{"verstrs": [...]}`).
- Child markers: creating a run with a parent also writes an empty `children/<child>` object in the parent's record (`core/children.py`, `storage/child_markers.py`; best-effort, not a record file, never journaled; prune removes it, push marks the remote parent), so `list_children` is one LIST and `core/subtree_direct.py:subtree_rows_direct` reads a subtree's rows without the index. `vmn-exp backfill-children <app> [--dry-run]` writes missing markers for existing runs from each run's `metadata.yml` alone, never the index (`read N/M runs` on stderr; `wrote N, present M[, failed F]`). Store upkeep (`ui/store_upkeep.py`): `vmn-exp ui` with background refresh and without `--read-only` does both by itself for store workspaces — after a refresh, one worker thread per server writes missing markers (a parent listed when first seen and again when >100 of its children are unchecked, else PUT per child) and compacts failed/stuck runs once per process, newest first, 30 s per pass, a pass per index at most every 60 s, on the leading replica only.
- Lineage: rows carry `outputs` (path/digest/size of every artifact and logged image/table, queryable as `outputs."<path>".digest`). The index keeps them off its lean snapshot rows (`IndexSnapshot.outputs_of(verstr)`, like `metric_summary`), so UI list/leaderboard/facets/columns payloads never carry them; row copies (`list_runs`, `get_run`, `list --json`), UI run detail, `LineageIndex(rows, outputs_of)` and queries (`filter_rows(..., extra={"outputs": snap.outputs_of})`) do. `run.use_artifact(ref, path)` records a `vmn://<app>/<verstr>/<path>` input and returns the file; inputs link to producers by that URI (any app) or by digest (same app). `reader.get_lineage`, `vmn-exp lineage <app> -v <ref> [--depth]`, `GET .../experiments/{verstr}/lineage` (all via `registry/lineage.py:run_lineage`). Registry join: an upstream `uri` link to a live version's artifact gains `model`/`version`/`kind` (the `<name>@<N>` input's version, else the first); `vmn-registry://` inputs are never nodes or digest matches, they fill the payload's `datasets` (`found` = version live). `registry.lineage.version_lineage(storage, name, n)` = `{model, version, kind, status, producer, consumers}` (consumers from `registry/<name>/uses`, pruned ones `found: false`), served at `GET .../models/{name}/versions/{n}/lineage`; `core/lineage.py` stays pure (the registry is injected as `registry=`).
- Similar runs (docs/plans/18-similar-runs.md, docs/plans/24-persistent-similar-index.md; pure `core/similar*.py`, `core/features*.py`, `core/intent.py`; `cli/similar*.py`, `gitmode/features.py`, `ui/routes_similar.py`): every git-mode run gets a `features` log entry next to `create` (changed files, `path::name` symbols, commit subjects from the captured patches; `author` via `core/identity.py:run_author`, email git config > `VMN_EXP_AUTHOR` > first local commit's `From:`; best-effort). Git-free runs copy it from the export's `vmn_features.yml` (older exports: `vmn_experiment.yml`'s log; `vmn-exp export` adds `exported_by`), reruns copy it with the rerunner as author, forks inherit neither it nor the intent. `--intent <text>`/`--intent-file <path|->` (YAML front matter `keywords`/`generator`) on `create`/`run`/`add` (add: latest unless `-v`) and SDK `start_run(intent=, intent_keywords=)`/`run.set_intent(text, keywords=, generator=)` append an `intent` entry (latest wins, 16 KB). Neither is on lean rows: `IndexSnapshot.features_of/intent_of/hypothesis_of` side tables; rows gain `author` and `has_intent` (query fields). `vmn-exp similar <app> [-v <ref>|--latest] [--top 10] [--all-apps] [--query] [--archived] [--context <text>|--context-file <path|->] [--text-only] [--json]` (no ref = the working tree, nothing created) ranks runs with SQLite FTS5: one document per run (columns text/code/files/symbols/inputs/params/entry/objective/branch; structured tokens hashed to `t<12 hex>` words; text `porter unicode61`) in the app's index SQLite file, filed in the index save's transaction (`core/similar_fts.py`, `similar_filing.py`, `similar_doc.py`); a query is one MATCH of the target's rarest tokens (<= 64 per column) ranked by `bm25` with `experiment.similar.weights` as column weights, 500 at a time (`similar_match.py`), reasons only for the items shown (`similar_reasons.py`); FTS5 is required (no fallback scorer; the UI answers 501); a Postgres-backed server keeps a host-local SQLite corpus per workspace app (`ui/similar_corpora.py`); leaves out the target, its run tree and its reruns (rerun identity = `code_verstr` + exact params + inputs, `similar_profile.rerun_key`; forks of/from the target are kept), so same-commit runs with other params are candidates; groups by run tree, else rerun identity (best on the goal metric, `(+N runs)`); conf `experiment.similar.weights`/`hint`. `create`/`run` print up to 3 on stderr (`cli/similar_hint.py`: persisted index file only, never refreshed; a 300 ms budget interrupts its SQLite queries; never for inner runs; `--no-similar`). `vmn-exp similar-backfill <app> [--all-apps] [--dry-run]` appends missing/older features from code-object patches under this host's writer (repo lock, git-free; `backfilled N, partial P, skipped S, failed F`). SDK `reader.similar_runs`, `run.similar()`; UI `GET .../experiments/{verstr}/similar`, `GET .../apps/{app}/similar?text=`, `GET /api/v1/similar?text=`, a run-page card and a leaderboard similar mode.
- Rich logging: `run.log_table/log_image/log_histogram` (tables as `outputs/tables/<name>/<step>.json`, images as `outputs/media/<name>/<step>.png` with a stdlib PNG encoder fallback, histograms as log entries; image/table files upload on a background worker that finalization waits for). The image/table entry carries the stored bytes' `sha256`/`size` (hashed before upload) and is itself the output record (no separate `artifact` entry), so `run.use_artifact(ref, "outputs/media/<name>/<step>.png")` fetches it; the uploader appends the entry (with its log-time step/timestamp) only once the file is stored — `finish()` drains the queue before the final log flush — and a file that fails to store (or is still queued at the final wait's deadline) is never recorded, so no entry/output ever outlives a failed or killed upload; `log_histogram` also takes a precomputed `{"bins", "counts"}`; run detail exposes `media`/`tables`/`histograms_total` (`histograms` steps inlined only while <= 100 served in all), `GET .../experiments/{verstr}/histograms/{name}` serves one key's steps, `GET .../experiments/{verstr}/table/{path}` pages a table.
- Parameter importance: `core/importance.py` (pure-Python histogram random forest + Pearson/Spearman) behind `reader.param_importance`, `vmn-exp importance <app> --metric m`, `GET .../experiments-importance?metric=`, and the leaderboard's Importance view.
- Metric values: numeric only. numpy/torch scalars and numeric strings are coerced to float; bools/strings/vectors are dropped with a warning; NaN/inf are kept and served as JSON `null`. Missing/non-finite values sort last in both directions. Numeric params fold into `metrics` only when finite (bools fold as 1.0/0.0).
- Metric storage (plan 12, `core/metric_*.py`, `core/series_reader.py`): metrics are not JSONL lines but per-stream columnar files under `metrics/`: an append-only stream `<s>.vms` while live (remote sync as `<s>@<seq>.vms` segments; sealed = compacted + a new stream every 10M points), compacted at the writer's end (SDK finish within the final upload wait, `vmn-exp run` child exit, importers; `core/metric_compact.py`) into one indexed `<s>@000000-<last>.vmx` (sorted chunks + min/max LOD pyramid + footer, range-readable) that supersedes the stream's segments up to `<last>` only (later ones stay visible; widest `.vmx` wins). `vmn-exp compact <app> -v <ref>... | --all-finished` compacts writers that died (running runs refused); `vmn-exp watch --compact` does it for failed/stuck runs; `rewind` rebuilds a compacted `.vmx`. Readers rebuild `metrics` log entries for log views and still read v1 JSONL `metrics` entries.
- Streams (docs/plans/21-per-root-streams.md; `core/streams.py`, `storage/stream_owner.py`, `storage/local_streams.py`, `storage/ship_state.py`): a log/metric stream is `<writer>~<owner>` (`display_writer` = the writer id, shown in `_writer`, UI writer columns, `export-metrics`; a `~` in an explicit writer id becomes `-`). The owner is per local record copy (`<record>/.stream-owner`, identified by the dir's realpath + `(st_dev, st_ino)`, minted under `.ship/owner.lock`: symlinks/bind mounts/same-fs renames keep it; a copied, restored-in-place, cross-fs-moved or re-fetched copy mints a new one; retired once its metric stream is compacted), per process without a local copy, per log buffer in `buffered.py`; get it via `core/writer.py:own_stream`. Only a stream's owner ever appends to, ships, seals, compacts or pushes it: `.stream-owner` holds one entry per view (matched on `(st_dev, st_ino)`, else on `st_ino` + realpath for a container's other st_dev; a renamed dir's path is refreshed under the owner lock), and every other id in it is *inherited* (a `cp -r`, restore, cross-fs move, another view) — never shipped, pushed nor compacted automatically (no "is the original alive" check); sync and `vmn-exp push` note once on stderr when the remote lacks part of them (`core/adopt.py`). `vmn-exp push <app> [-v <ref>...] --adopt` (`core/adopt.py:adopt_streams`, `CachedSnapshotStorage.adopt_streams`) catches their files up with the remote (refusing the run, nothing changed, push exit 1, when a remote `.vmx` holds fewer points than the copy's files), then makes them this view's retired owners (shipped/pushed from then on) — only when the original checkout is gone. An append under an unowned stream goes to the copy's own. A metric append holds the copy's append lock (`.ship/append.lock`) from choosing its stream to writing it, and a compaction retires its stream under it. Each stream has one appending owner and its remote copy is a replica of the owner's local file: syncs ship every local stream's new bytes as the next segment with exclusive `create_log_segment`/`create_metric_segment` (taken → re-list, resume from the remote size), ship state in `<record>/.ship/` under a per-copy ship lock (2s wait while the SDK SIGTERM finalizer runs). No seed/merge of other roots' lines; S3 `read_range` raises on non-missing errors.
- - Status is derived, never stored: `created`/`running`/`stuck`/`succeeded`/`failed`. `stuck` = claims running with no exit code, and both the heartbeat timestamp (writer clock) and the store's write time of `run_state.yml` (file mtime / S3 `LastModified`, when known) are stale past `max(3 * interval, 60s)` — a fresh store write proves liveness despite writer clock skew; unknown store time falls back to the heartbeat alone. SDK attempts carry a random `attempt` id in `run_state.yml`; a heartbeat reads the stored state first and is dropped (the attempt stops heartbeating) once another attempt wrote the final state (`sdk/state_publisher.py`; best-effort read-then-write, a start is never blocked).
- - Python SDK: `from vmn_exp.sdk import start_run` — in-process alternative to `vmn-exp run` (`run.log_metric/log_metrics/log_params/log_note/log_input/log_artifact/log_artifacts/log_dict/log_text/log_figure/set_tag(s)`, `run.finish()`; `log_metrics` without `step=` records `run.step` and advances it — one per-run counter with a high-water mark (explicit steps raise it, lower ones are kept, `commit=False` shares the step with the next call, all-dropped calls consume none; seeded from `start_step` on fork/rewind, else past the highest visible metrics step on resume; `sys_*` metrics stay step-less, media keeps per-name steps, the metrics file is unchanged; `NoOpRun.step` counts too); log writes are batched and flushed at most ~1s apart, on heartbeat and on finish/SIGTERM/exit; `start_run(name=, tags=, run_id=<resume>, all_ranks=False, capture_env=None, mode=None)` — ranks > 0 get a no-op run; `mode="disabled"`/`VMN_MODE=disabled` returns a `NoOpRun` (`disabled=True`) before touching git or the store (explicit `mode="enabled"` beats the env) and makes `autolog()` patch nothing; the snapshot is captured outside the repo lock; an SDK cold start commits/tags locally without pushing; read side `vmn_exp.sdk.reader.get_run/list_runs/runs_dataframe/get_metric_history` — the last two need `vmn-exp-sdk[pandas]`). Writes the same files as the CLI and heartbeats from its own daemon thread (which also syncs the log to a remote every `sync_interval_sec`, default 30). `current_run()` returns the run the calling thread records into (its own context-bound run, else the process's only open run); runs are fork-aware (`Run.pid`); `run.pause_background(timeout=None)`/`resume_background()`/`with run.background_paused():` (module-level versions for every open run; `NoOpRun` no-ops) quiesce the run's background threads (heartbeat, flush, syncs, uploads go through its `sdk/background_gate.py` gate: in-flight calls finish, none starts until resumed, `finish()` lifts it), and `sdk/fork_safety.py`'s `os.register_at_fork` handlers pause every open run before a fork (bounded by `FORK_PAUSE_TIMEOUT_SEC` = 10 s total), resume in the parent and reset SDK locks/gates in the child, and `VMN_EXPERIMENT_ID` never parents a run to a sibling another thread opened. With `VMN_SNAPSHOT_METADATA` set, `start_run()` works without a git checkout (storage from `VMN_EXPERIMENT_DIR`). `vmn_exp/sdk/` ships in vmn-exp-sdk: it depends only on `vmn_exp.core`, `vmn_exp.storage`, `vmn_exp.registry`, `vmn_exp._base` — never on `version_stamp`, `vmn_exp.snapshot`, `vmn_exp.ui` or `vmn_exp.cli`; git-mode creation lives in `vmn_exp.gitmode` (vmn-exp) and is imported lazily. See docs/vmn-exp/sdk.md.
+ - Status is derived, never stored: `created`/`running`/`stuck`/`succeeded`/`failed`. A run handed back with `run.release()` (`state: released`, no exit code, `end_reason: released`, `released_at`) derives `created` (waiting; settled `lost` under an ended outer run). `stuck` = claims running with no exit code, and both the heartbeat timestamp (writer clock) and the store's write time of `run_state.yml` (file mtime / S3 `LastModified`, when known) are stale past `max(3 * interval, 60s)` — a fresh store write proves liveness despite writer clock skew; unknown store time falls back to the heartbeat alone. SDK attempts carry a random `attempt` id in `run_state.yml`; a heartbeat reads the stored state first and is dropped (the attempt stops heartbeating) once another attempt wrote the final state (`sdk/state_publisher.py`; best-effort read-then-write, a start is never blocked).
+ - Python SDK: `from vmn_exp.sdk import start_run` — in-process alternative to `vmn-exp run` (`run.log_metric/log_metrics/log_params/log_note/log_input/log_artifact/log_artifacts/log_dict/log_text/log_figure/set_tag(s)`, `run.finish()`, `run.release(reason=)` (hand the run back unfinished: flush/sync, stop recording, `state: released`; `start_run(on_sigterm="fail"|"release")` makes SIGTERM fail it with 143 or release it, `sdk/release.py`; reopening with `run_id=` bumps `resume_count`; rows/queries carry `end_reason`, `signal`, `released_at` (kept across reopens), `resume_count`; job timelines count `released` per minute; `NoOpRun.release()` no-ops); `log_metrics` without `step=` records `run.step` and advances it — one per-run counter with a high-water mark (explicit steps raise it, lower ones are kept, `commit=False` shares the step with the next call, all-dropped calls consume none; seeded from `start_step` on fork/rewind, else past the highest visible metrics step on resume; `sys_*` metrics stay step-less, media keeps per-name steps, the metrics file is unchanged; `NoOpRun.step` counts too); log writes are batched and flushed at most ~1s apart, on heartbeat and on finish/SIGTERM/exit; `start_run(name=, tags=, run_id=<resume>, all_ranks=False, capture_env=None, mode=None)` — ranks > 0 get a no-op run; `create_runs(app, [{name, params, tags, note, run_id}], parent=, capture_env=False, workers=16, exist_ok=False)` pre-registers waiting runs, verstrs in order (`core/create_batch.py`; `create_run(..., tags=)` is a batch of one): one parent pick, capture/export read, env capture and lock per batch, the code version's runs listed at most once (+1 when another host took a planned name), then per run a `create_exclusive` claim + parent marker + one create entry with params and tags (a scratch-dir store ships it at once); a bad item refuses the batch first, a taken `run_id` raises `RunExistsError` unless `exist_ok` and same parent + name; pods open them with `start_run(run_id=)`; `mode="disabled"`/`VMN_MODE=disabled` returns a `NoOpRun` (`disabled=True`) before touching git or the store (explicit `mode="enabled"` beats the env) and makes `autolog()` patch nothing; the snapshot is captured outside the repo lock; an SDK cold start commits/tags locally without pushing; read side `vmn_exp.sdk.reader.get_run/list_runs/runs_dataframe/get_metric_history` — the last two need `vmn-exp-sdk[pandas]`). Writes the same files as the CLI and heartbeats from its own daemon thread (which also syncs the log to a remote every `sync_interval_sec`, default 30). `current_run()` returns the run the calling thread records into (its own context-bound run, else the process's only open run); runs are fork-aware (`Run.pid`); `run.pause_background(timeout=None)`/`resume_background()`/`with run.background_paused():` (module-level versions for every open run; `NoOpRun` no-ops) quiesce the run's background threads (heartbeat, flush, syncs, uploads go through its `sdk/background_gate.py` gate: in-flight calls finish, none starts until resumed, `finish()` lifts it), and `sdk/fork_safety.py`'s `os.register_at_fork` handlers pause every open run before a fork (bounded by `FORK_PAUSE_TIMEOUT_SEC` = 10 s total), resume in the parent and reset SDK locks/gates in the child, and `VMN_EXPERIMENT_ID` never parents a run to a sibling another thread opened. With `VMN_SNAPSHOT_METADATA` set, `start_run()` works without a git checkout (storage from `VMN_EXPERIMENT_DIR`). `vmn_exp/sdk/` ships in vmn-exp-sdk: it depends only on `vmn_exp.core`, `vmn_exp.storage`, `vmn_exp.registry`, `vmn_exp._base` — never on `version_stamp`, `vmn_exp.snapshot`, `vmn_exp.ui` or `vmn_exp.cli`; git-mode creation lives in `vmn_exp.gitmode` (vmn-exp) and is imported lazily. See docs/vmn-exp/sdk.md.
+ - Out-of-pod finish (plan 28 API3, `core/run_finish.py`, `sdk/manage.py`, `cli/finish_args.py`): `manage.finish_run(app, ref, exit_code, *, error=, end_reason=, force=False)` writes the final `run_state.yml` directly (no reopen, no heartbeat; keeps `started_at`/host/cadence, replaces an earlier ending's `error`/`end_reason`/`signal`, new `attempt` so a forced-out live attempt's heartbeats are dropped); `FinishRefusedError` (a `ValueError`) while the run is live (fresh by either clock, `derive_status`) or already finished, unless `force`. `manage.log_note(app, ref, text)` / `manage.log_metrics(app, ref, values, step=None)` append under the caller's own stream; all flush to the remote at once. CLI `vmn-exp add <app> -v <ref> --finish <code> [--error <text>] [--end-reason <text>] [--force]` (after the add's other entries). The Optuna tracker (`integrations/optuna_study.py`) declares `objective`/`objective_<i>`/`best_value` with the study direction as `goal`.
- Autologging: `from vmn_exp.sdk import autolog, autolog_disable`. `autolog()` wraps framework `fit()` methods to record hyperparameters, final metrics, per-epoch series where the framework exposes one, and (with `log_models=True`; default off) the model in its native format. Frameworks are patched only once imported (`vmn_exp/sdk/import_hooks.py`), so `autolog()` never imports tensorflow/torch itself. Fits record into `current_run()`; framework worker threads without their own run are ignored while a recorded fit is in progress. Search estimators add `sklearn_best_cv_score`/`sklearn_best_<param>`; `sklearn_score` is a training-set score, computed when `training_score="auto"` (<= 10k rows). Lightning under DDP skips model saving. Supported: `sklearn`, `xgboost`, `keras`/`tensorflow`, `lightning`/`pytorch_lightning`, `transformers` (injects a `VmnCallback` via `Trainer.train`; watches `transformers.trainer` lazily). Plain `torch` has no `fit` to wrap — log metrics with `run.log_metric(...)` in your own loop; gradient/parameter histograms via `vmn_exp.integrations.torch_watch.watch(model, log=, freq=1000, bins=64)` (training-mode forward count as step, logs on the training thread, `unwatch`). Add a framework with an `_Adapter` entry in `SUPPORTED_FRAMEWORKS` in `vmn_exp/sdk/autolog.py` (only `discover` is mandatory; the other hooks default to the sklearn behaviour). Records nothing unless a `start_run()` is open (never opens one implicitly — that would stamp a version from inside `fit()`). Keys are `<framework>_<name>` with an underscore so the query language's two-part paths resolve them. Failures never break `fit()`; patching is idempotent and `autolog_disable()` restores the originals.
- - Query language (`vmn_exp/core/query.py`): filters rows for `list_runs(query=...)` and the UI's `?q=`. Comparisons `= == != < <= > >=`, `~`/`contains`/`!~` (case-insensitive substring; on list fields like `command`/`children` any element matches), numbers accept scientific notation (`1e-4`), `in`/`not in`, `and`/`or`/`not`, parens. Fields are bare row keys, `metrics.<name>` (numeric fold) and `params.<name>` (verbatim, so strings/bools work). `intent` (latest intent text) and `keywords` (its list) come from the snapshot's `intent_of` side table, never lean rows; `similar("<text>")` is true for runs whose FTS `text` column holds every word of the text (porter-stemmed AND; `core/query_similar.py`, extras built once by `core/query_extra.py:snapshot_extra`/`store_extra` for every entry point); `under("<v>")` matches a subtree. Two-valued: a comparison against a missing field is false, `= null` tests absence. Bad queries raise `QueryError` with an offset (400 over HTTP). `vmn-exp list --query` applies it on the CLI.
+ - Query language (`vmn_exp/core/query.py`): filters rows for `list_runs(query=...)` and the UI's `?q=`. Comparisons `= == != < <= > >=`, `~`/`contains`/`!~` (case-insensitive substring; on list fields like `command`/`children` any element matches), numbers accept scientific notation (`1e-4`), `in`/`not in`, `and`/`or`/`not`, parens. Fields are bare row keys (incl. plan 28 N4's `queue_sec` = `started_at` - `timestamp`, null until started, and the board-only `duration_ratio`), `metrics.<name>` (numeric fold) and `params.<name>` (verbatim, so strings/bools work). `intent` (latest intent text) and `keywords` (its list) come from the snapshot's `intent_of` side table, never lean rows; `similar("<text>")` is true for runs whose FTS `text` column holds every word of the text (porter-stemmed AND; `core/query_similar.py`, extras built once by `core/query_extra.py:snapshot_extra`/`store_extra` for every entry point); `under("<v>")` matches a subtree; `ago("30m"|"1h"|"2d")`, right of `< <= > >=` only, is the UTC ISO time that long before now (`core/query_ago.py`), pinned when a query compiles (`query.py:pin_ago`; the server pins `q` to the minute, `ui/attention.py:pinned`, so caches/ETags follow the clock; the Postgres backend compares the pinned time); `up.params|tags.<k>`/`up.name` read the nearest ancestor holding the field and `up(tags.kind = "study").params.<k>` the nearest ancestor matching one equality on `params.<k>`/`tags.<k>`/`name` (plan 28 N1, `core/query_up.py`: resolved through the snapshot's parent map, memoized per snapshot (`query_extra.rows_extra`, under `lean_extra`/`store_extra`); never the run itself; the built-in `kind` is no selector; not in the Postgres backend); `group=up.*` and `experiments-columns` `up.*` keys read the same, and a moved board order/grouping reading `up` is rebuilt when an ancestor's params/tags/name change (the generation's delta marks it, `ui/leaderboard_up.py`). Two-valued: a comparison against a missing field is false, `= null` tests absence. Bad queries raise `QueryError` with an offset (400 over HTTP). `vmn-exp list --query` applies it on the CLI.
- Lock scope: `vmn-exp run` holds the repo lock only for the create/auto-init phase and releases it before supervising the child, so long runs don't block other `vmn` commands and nesting works. The SDK scopes it the same way around create/cold-start.
- Storage (`vmn_exp/storage/`): backends are chosen by store URI — `--store` > `VMN_EXPERIMENT_STORE` > conf `experiment.storage.uri` (`s3://bucket/prefix[?endpoint_url=]`, `gs://` (`[gcs]` extra), `az://` (`[azure]` extra), `file:///dir`); `--bucket/--prefix/--endpoint-url` are shorthand for an `s3://` URI (`--prefix` has no literal default; `store_uri` supplies `vmn`). Restore safety snapshots (and `vmn goto` of them) live in the experiment store's `snapshots/` area. `vmn goto -v <dev-version>` and `vmn-exp restore` share one lookup (`cli/code_record.py`): local experiments, then the remote experiment store, then snapshots; no-code runs are refused. Every backend implements `list_apps()` (one listing of `runs/`). Schemes register in `storage/registry.py`; other packages add them via the `vmn_exp.storage` entry-point group (the contract is in docs/vmn-exp/experiments.md). Run verstrs are claimed atomically (`create_exclusive`: `O_EXCL` mkdir / S3 `If-None-Match` / GCS `if_generation_match=0` / Azure `overwrite=False`), so hosts sharing a bucket or NFS dir never collide; allocation lists names only. Local writes are atomic (temp + rename) and never resurrect a deleted record. Remote log sync uploads only new lines as segments `log/<writer>@<seq>.jsonl`; `run_state.yml`/logs are never cached locally from a remote. App keys use the tag form (`/`→`-`) on every backend; S3 `exists` is a HEAD; artifacts stream via `upload_file` and are listable/downloadable. Snapshot identity keeps the full `diff_hash` and extends the verstr's hash on a collision instead of overwriting. A run's code (patches + untracked tarball) is stored once per code identity as a record `<code_verstr>.<diff_hash>` in the `code/<app-key>/` area (`core/code_store.py`; its `metadata.yml`, written last, marks it complete); the run's metadata references it as `code:` and `storage.load` resolves it (backends implement `load_record`). A run of already-stored code builds no tarball and uploads nothing; a missing/incomplete object makes `restore`/`goto`/`export` refuse (`code_missing`); `prune` deletes an object with the last run of its code.
- - Nesting: an experiment created while `VMN_EXPERIMENT_ID` is set records it as its `parent`, so a sweep wrapped in `vmn-exp run` yields one outer job with inner jobs. `create`/`run --parent <ref>` sets it explicitly. `kind` is `outer`/`inner`/`single`; an outer job's `tree_status` rolls up its subtree (`failed > stuck > running > created > succeeded`); once an outer run has ended (own status succeeded/failed/stuck) its descendants settle (`core/settle.py`, derived, never stored: row field/query field `settled` = `never_started` for a created one, adding nothing to rollups, or `lost` for a stuck one, counted as failed; job summaries/fleet/`experiments-status` add `never_started`/`lost` counts; an ended short sweep reads its own status + `ended · N never started`); a container outer that never ran (own `created`) is left out of its own rollup, and a sweep outer whose trials number fewer than planned (grid size, else `run_cap`, from `core/sweep/spec.py:planned_trials`; the index puts it on the row as `sweep_planned`, the tree adds `sweep_trials`; none for an uncapped random/bayes sweep) adds `created`, so a finished-but-short sweep reads `created` (`vmn-exp list`: `created (16/20 incomplete)`, UI: `16/20 · incomplete`).
- - Sweeps (docs/vmn-exp/sweeps.md): `vmn-exp sweep create <app> -f sweep.yml` stores a W&B-style spec (`method: grid|random|bayes`, `metric`, `parameters`, `run_cap`, `early_terminate: {type: median}`, `command`/`program` template) on an outer run; `vmn-exp sweep agent <app> <sweep> [--count] [--retry-failed] [-- cmd]` claims trial slots with `create_exclusive` (records `t<N>`/`t<N>.a<K>` in the `sweeps/<app-key>~<sweep>/` area; never reused), runs each as an inner job with `VMN_SWEEP_PARAMS`, and median-stops laggards (recorded as `end_reason: stopped` in `run_state.yml`, which derives as succeeded; `end_reason` is a row/query field). A trial's metric is its own, else its nested SDK runs' (single one, or best by goal). `sweep status [--json]`, `GET .../experiments/{verstr}/sweep`; SDK `from vmn_exp.sdk import sweep_params`. bayes needs optuna.
+ - Nesting: an experiment created while `VMN_EXPERIMENT_ID` is set records it as its `parent`, so a sweep wrapped in `vmn-exp run` yields one outer job with inner jobs. `create`/`run --parent <ref>` sets it explicitly. `kind` is `outer`/`inner`/`single`; an outer job's `tree_status` rolls up its subtree (`failed > stuck > running > created > succeeded`); once an outer run has ended (own status succeeded/failed/stuck) its descendants settle (`core/settle.py`, derived, never stored: row field/query field `settled` = `never_started` for a created one, adding nothing to rollups, or `lost` for a stuck one, counted as failed, or `outlived` for a running one, still counted as running and keeping its children open (plan 28 T2b; flagged by the built-in `outlived` attention part, `ui/attention_built_in.py`: counted from the live rows, in `experiments-status` `attention` and `job.attention` only while non-zero; the web pill reads `running · outlived`); job summaries/fleet/`experiments-status` add `never_started`/`lost` counts; an ended short sweep reads its own status + `ended · N never started`); a container outer that never ran (own `created`) is left out of its own rollup, and a sweep outer whose trials number fewer than planned (grid size, else `run_cap`, from `core/sweep/spec.py:planned_trials`; the index puts it on the row as `sweep_planned`, the tree adds `sweep_trials`; none for an uncapped random/bayes sweep) adds `created`, so a finished-but-short sweep reads `created` (`vmn-exp list`: `created (16/20 incomplete)`, UI: `16/20 · incomplete`).
+ - Sweeps (docs/vmn-exp/sweeps.md): `vmn-exp sweep create <app> -f sweep.yml` stores a W&B-style spec (`method: grid|random|bayes`, `metric`, `parameters`, `run_cap`, `early_terminate: {type: median}`, `command`/`program` template) on an outer run; `vmn-exp sweep agent <app> <sweep> [--count] [--retry-failed] [-- cmd]` claims trial slots with `create_exclusive` (records `t<N>`/`t<N>.a<K>` in the `sweeps/<app-key>~<sweep>/` area; never reused), runs each as an inner job with `VMN_SWEEP_PARAMS`, and median-stops laggards (recorded as `end_reason: stopped` in `run_state.yml`, which derives as succeeded; `end_reason` is a row/query field). A trial's metric is its own, else its nested SDK runs' (single one, or best by goal). The sweep outer run and each trial carry a `define_metric` of the spec's `metric.goal` (`core/sweep/spec.py:metric_definition`), so rankings follow it without conf. `sweep status [--json]`, `GET .../experiments/{verstr}/sweep`; SDK `from vmn_exp.sdk import sweep_params`. bayes needs optuna.
- `import-mlflow`: `vmn-exp import-mlflow <app> (--mlruns <dir> | --tracking-uri <uri>) [--experiment <name|id>]... [--skip-artifacts] [--include-deleted] [--dry-run] [--workers 8] [--refresh-features] [--repo <path>]` — import runs from an MLflow FileStore or tracking server. Exactly one source required. Re-import is idempotent (already-present runs are skipped). No repo lock taken; no auto-init. `mlflow.user`/`mlflow.project.entryPoint` go to a `features` entry (`source: mlflow`), `mlflow.note.content`/`mlflow.source.git.branch` to metadata `note`/`branch`; `--refresh-features` gives already-imported runs those (adding `refreshed R` to the summary), `--repo` records the source commit's branch diff as `changed` (`changed_source: commit`). Summary line: `imported N, skipped M (already present), resumed K, failed F`; exit 1 if any failed. `--tracking-uri` requires `pip install mlflow-skinny`.
- - `vmn-exp ui`: Serve the web dashboard + REST API (a remote store's `/apps` counts runs from the persisted index, `experiments: null` before it has any — never by listing them) (`pip install "vmn-exp[ui]"`). `--host`, `--port` (8265), `--token`, `--data-dir`, `--repo` (repeatable), `--store <uri>` (`s3://bucket/prefix?endpoint_url=` for MinIO), `--read-only`, `--no-browser`, `--no-index`, `--allowed-host` (repeatable). `vmn-exp ui backup <path> [--config|--db]` copies the control plane online (SQLite backup API + integrity check; Postgres prints the `pg_dump` command). `vmn-exp ui diag (--pid <PID> [--data-dir] [--wait 30] | --url <base> [--token] | --summarize FILE) [-o FILE]` (`ui/diag_cli.py`, `ui/diag_summary.py`) takes a diagnostic dump (SIGUSR1 → `<data-dir>/diag/diag-<ts>-<pid>.json`, or `GET /api/v1/diag`) and prints a summary; S3 request stats come from botocore hooks on every vmn S3 client (`storage/s3_stats.py`, `add_listener` per call). Sections also cover S3 by object kind (`storage/s3_kinds.py`) and by caller (requests' `s3`, `by_thread`; pools carry the request context via `storage/thread_context.py` and are named, `tests/test_pool_names.py`), traced locks (`core/traced_lock.py`: holder/waiters), background activities (`core/activity.py`), recent warnings (`ui/diag/logs.py`) and event-loop/GIL lag (`ui/diag/latency.py`); `--summarize` reads them (`ui/diag_summary_more.py`); `ui diag` needs no checkout. Postgres access goes through one pool per DSN with timeouts/keepalives/reconnects (`ui/pg_pool.py`); API tokens are HMAC-SHA256 under a per-control-plane key; expired sessions and `audit.retention_days` (server.yml, default 365) are swept hourly (`ui/control_plane_sweep.py`); see docs/vmn-exp/server.md. Without a token, `/api` only accepts loopback/allowed `Host` headers (DNS-rebinding guard); mutations reject foreign `Origin`s and need `Content-Type: application/json`; URL app names/verstrs are validated; the SPA fallback never serves files outside `static/`. Responses are gzipped and NaN-safe. `vmn-exp ui` keeps each watched app's index fresh from a background refresher, so requests read a lock-free snapshot (list/detail stay ~ms at 100k runs); list/detail/columns answer ETag/304; `experiments-facets`, `experiments-columns` (whole-set chart data) and `POST .../series` (batched overlay series) exist. The vmn-native shell (plan 19: context root, status rollup, CLI-grammar ⌘K, sign-in page) reads `/meta` (versions, `mode`, `auth` hints, `principal`, `role`), `/workspaces` `role`, `workspaces/{ws}/git` (branch/HEAD/dirty via one lock-free `git status`) and `.../apps/{app}/experiments-status` (counts + `rollup`, carried across index generations by the row delta, `ui/status_counts.py`; plus `freshness` `{as_of, refreshing, refresh_running_sec, phase, progress, refresh_failing, last_error, stale_after_sec}`, `as_of` persisted in the index, `core/index_freshness.py`; the web shows a stale/error banner, `components/FreshnessBanner.tsx`; the refresher heals itself: every step guarded, a dead thread restarted, failure streaks logged, warming on its own `vmn-ui-warm` worker that keeps only the newest snapshot, `ui/latest_worker.py`); a 401 carries `auth` hints and `WWW-Authenticate: Bearer`; run detail has `idx`/`name`. The list endpoint pages (`offset`/`limit` <= 1000), sorts server-side (`sort`, `sort=timestamp|started_at|finished_at`, `order=asc|desc`); `group=parent|forked_from|rerun_of|code_verstr|branch|status|tags.<k>|params.<k>` answers groups instead (plan 22 K1, `ui/leaderboard_groups.py` + `_group_spec`/`_group_fields`/`_group_summary`/`_group_cache`, cached per snapshot and moved by deltas): `{groups: [{key, label, target, loose, counts, aggregates: {m: {mean,best,median,min,max,n}}, rows, rows_total, rows_unfiltered}], total_groups, total_rows, running, aggregate}` (`rows_unfiltered` = the group without `q`/`status`, `running` = running rows after the filters, `aggregate` = what `agg=` resolved to), groups ordered by `agg=mean|best|median|min|max` (default conf `experiment.ui.group_aggregate` of the checkout branch's conf (`ui/readers/branch_conf.py`: HEAD read as a file, no git process; a detached HEAD takes the branch vmn resolves it to via `version_stamp.api.checkout_branch_of`, git once per (checkout, HEAD sha)), else mean) of the sort metric (`sort=c:total|c:waiting|c:running|c:done|c:failed` orders groups by that count instead, desc unless `order=asc`, `failed` = failed + stuck, ties and rows in the default order; plan 22 F10), the no-value group last (key `∅`, a value starting with `∅` keyed doubled; label `(none)`; groups[].`loose` marks a link grouping's (`parent`/`forked_from`/`rerun_of`) no-link group, which holds only the runs heading no group (absent when empty; `ui/leaderboard_group_loose.py`; plan 22 F12)), `limit` <= 50 groups and `rows_per_group` 1..100 (<= 5000 rows a page; clamped, never refused), `group_key=` pages one group; web client `webui/src/apiGroups.ts`; run detail is bounded (`log_tail`/`log_total`, per-metric `series` downsampled to `max_points`, `include_log=1` for the full log) and `/experiments/{verstr}/log?offset&limit` pages the log. Run code (plan 20 K2-K4, `ui/routes_run_code.py`, `ui/readers/{patch_files,baselines,failure_context}.py`): `GET .../experiments/{verstr}/patch?max_bytes=` = `show --patch --json`'s `patch` + `files: [{path, added, removed}]` (ETag = code identity); `GET .../experiments/{verstr}/baselines` = `{base?, parent?, forked_from?, rerun_of?, previous?, best?}` each `{verstr, idx, name, status}` from the index snapshot (`previous` = latest earlier same branch + command, `best` = best other run on the primary declared goal); a failed/stuck run's detail adds `output_tail` (`{lines, truncated}`, last 40 lines of `outputs/output.log`, one <= 16 KB range read), `last_step` (`{step, of}`, `of` = numeric `params.epochs`/`max_steps`) and `last_good` (latest earlier success of the same command) — absent for other statuses; web client `webui/src/apiRunCode.ts`. Leaderboard default columns (plan 20 D11, `webui/src/pages/defaultColumns.ts`): `#`/status/experiment, the schema-declared goal metrics (pinned unless `pin=`; `pin=-` = none; all metrics when no goal is declared), up to 6 params that vary across loaded rows (fit to the width), `when` (plan 22 G1: every run is a row, outer and inner alike; no fold, no per-run inner count columns; an outer's experiment cell keeps `tree <status>` and an incomplete sweep's `16/20 · incomplete`). Grouped (`group=`, plan 22 W2: `hooks/useBoardRows.ts` > `useGroupedRows.ts`, `util/groupItems.ts`, `pages/LeaderboardGroupRow.tsx`) the board renders K1's groups: a header row per group (link target `@N name` + status/tree pill, the code verstr + base · branch, `field = value`; counts in words `16 runs · 2 running · 1 failed` / `16 runs · all succeeded`, `5 of 16 runs · 5 succeeded` when groups[].`rows_unfiltered` differs from `rows_total` (plan 22 F1/F2; a cut group never says `all`), with an sr-only sentence; the counts belong to the group's head, not to columns (plan 22 F13): before the count line a run bar in the status colours (finished runs fill from the left, waiting = the empty track; none for a group of one run; `pages/groupHeaderText.ts:barSegments`, `GroupBar`), on the phone section label too; the menu's "Sort groups" row (aggregate / runs / running / failed / waiting / succeeded, `util/groupCountSort.ts`) sorts groups by a count (`sort=c:<name>&order=desc`; a `sort=c:*` is dropped with `group=none`); per metric column the `agg` aggregate titled `mean of 16`, `own <v>` for a target's own value, metric headers tagged with the aggregate), up to 10 rows, a `+N more` row (`group_key` paging), `collapse=<key>`, a loose group's rows as plain unindented rows after the groups with no header (a plain `+N more` row; plain cards on the phone; no group colour in Charts), 50 groups per page; the `Group by ▾` menu next to Columns (`components/GroupByMenu.tsx`, `util/groupBy.ts`, `hooks/useBoardGrouping.ts`) offers none, Links (parent run, forked from, rerun of, same code), Fields (branch, status, `tags.<k>`, `params.<k>` with <= 20 distinct values, from facets `tag_keys`/`param_values`), Sort groups and the Aggregate; `group=` absent = `parent` when facets `parented` > 0, else `none` (flat, sorted/paged as before); `agg=` absent = the app default (the request omits it; the header tags and the menu show the payload's top-level `aggregate`, plan 22 F3, `hooks/useBoardAggregate.ts`); grouped, the board polls while the payload's top-level `running` > 0 (F4); on a phone the menu is viewport-fixed under its trigger (`hooks/useAnchorTop.ts`); saved views keep `group=`/`agg=`/`collapse=` with the rest of the URL; a header's checkbox selects its loaded rows (indeterminate when partial; Compare (N)/Archive count rows), and the Charts tab's bar/scatter/parallel colour runs by group (`experiments-columns` key `group:<field>`, `util/chartGroups.ts`, legend `components/ChartGroupLegend.tsx`); with rows selected the Charts tab plots only them (plan 22 F7, `hooks/useChartSelection.ts`: `Showing N selected · Show all`, Show all keeps the selection and lasts until it changes); note/tags/started/ended via `show=`; `show=m:|p:|c:<name>` turns a column on, `hide=` off (a schema-hidden metric keeps `hide=` = shown). The leaderboard filter is one grammar (`components/LeaderboardFilter.tsx` = `QueryBar` tokens + text, `FilterShortcuts`, `SimilarToggle`; pure `util/filterTokens.ts`): status/branch/archived shortcuts and hand-typed `status`/`branch`/`text ~` clauses become the same removable tokens, URL params unchanged (`q`, `status`, `branch`, `search`, `archived`); similar mode keeps every token. Below 640px (`hooks/useMediaQuery.ts` `NARROW_QUERY`) the leaderboard renders cards instead of the grid (plan 20 C8, `pages/LeaderboardCards.tsx`): name + status (outers: tree status), the sorted-by metric (else the primary, else the first shown) large, `@N · branch · when`, a per-card checkbox plus select-all; grouped, each group is a sticky section label above its cards (plan 22 G9, `pages/LeaderboardCardGroup.tsx`: the group checkbox (F8), fold, name, status count (wraps, F2), the group's aggregate on that metric) with a `+N more` button; the toolbar popovers span the viewport; touch screens (`TOUCH_QUERY`) get no ⌘K key hints, and the phone identity chip shows an initial only for a signed-in name. Compare (plan 20 D6, `webui/src/pages/Compare/`) is one page, `compare-runs?tab=curves|table|code&sel=…` (repeated `sel` shared by the tabs, selection chips; the code tab diffs the first run against the second or `?to=`, K6 422 reasons shown); the plan-20 `overlay?runs=` / `compare?v=&to=` pages are gone (plan 22 G8, no redirect). The run page (plan 20 D5, `webui/src/pages/RunLayout.tsx`) reads header (`@N`, status, base) > note/tags > failure card > curves > Changes > Fleet/Sweep > Params | Metrics > Similar | Lineage > outputs, then `components/Disclosure.tsx` sections (Status details, Log, Environment closed; Media, Comments open when present), each choice remembered in localStorage `vmn-exp:disclosures`. With the GIL on and a watched app of 10k-200k runs it logs a one-time hint to run it on free-threaded 3.14t (`uvx --python 3.14t --from "vmn-exp[ui]" vmn-exp ui`; ~10x the throughput at 100k runs); from 200k runs it doesn't, and on 3.14t it warns once to use a GIL build (mimalloc's per-thread heaps keep 3-5x the live data, and stop-the-world collections grow with them; `ui/gil_hint.py`). The `ui` extra has no orjson (no free-threaded wheels); the `orjson` extra adds it. The diag dump's `gc` section (`ui/diag/gc_pauses.py`) records collection pauses. See docs/vmn-exp/ui.md.
- - Hot board views: Single runs (`top=singles`) is the runs board with single-run columns (`pages/Experiments/SinglesView.tsx`); a second after a view shows, the first page of the other views is prefetched under the current filters (`useViewPrefetch.ts`, fresh 30 s); index-backed answers (first pages, facets, statuses, apps, job summaries; never per-run reads or tokens) persist in localStorage `vmn-exp:query-cache:v1` (3 MB LRU, `src/queryPersist.ts`, cleared on sign-out) and paint at startup, then revalidate; the server warms each new generation's facets and default first pages (`ui/leaderboard_warm.py`). Hovering a run fetches nothing; slow per-run requests go through a 2-in-flight gate per tab (`requestGate.ts`); one live stream per tab (`hooks/liveHub.ts`); a slow read whose client left answers 499 unread (`ui/slow_reads.py`).
- - Outer runs at scale (plan 23, docs/plans/23-experiments-at-scale.md; an *outer run* (`kind = "outer"`; *job* in code, URLs and API fields) = a run with children, thousands of runs each; every user-visible string says outer run). The experiments page opens on **Outer runs** (`view=jobs`, switch `Outer runs (N) · Single runs (N) · All runs`, last view per app in localStorage; `webui/src/pages/Experiments/`): one row per top-level outer run with run bar, `done/expected · failed · stuck · rate · ETA` (`finished`/`ended · N never started`/`stalled`), best (`—` without a goal), spread; `job:*` sorts. Server: per-job rollups over each subtree moved by index deltas (`ui/job_rollup*.py`, `job_tally.py`, removable `quantile_sketch.py`, `failure_signature.py`; no goal → the app's most common non-param metrics, `ranked: false`); rows gain `root`/`failure_sig`, query `under("<v>")`; `experiments?top=1|jobs|singles` (+ `match_subtree=1` → `matched`, `jobs_matched`/`runs_matched`), `scope=<v>` on list/columns/facets/importance (`ui/leaderboard_scope.py`), groups carry `clause` (+ job aggregates with `top`), `experiments/{v}/summary` and `/failures` (`ui/routes_job.py`, `job_views.py`; a job's run detail has no `fleet.children`), `search?text=` (⌘K; `ui/search_index.py` also prefilters `~` clauses), `events` SSE (`ui/live_events.py`, `routes_events.py`; web `hooks/useLiveScope.ts`, polling fallback), bulk `actions/exp_archive|exp_unarchive|exp_tag` by `{scope,q,status}` with `dry_run` (`ui/bulk_actions.py`). The outer run's page (`webui/src/pages/Job/`): header + run bar + `outer · N runs` badge, tabs `overview|runs|charts|failures|code|log`; the Runs tab is the scoped board with a facet sidebar and select-all-matching; inner run pages get an outer run context bar (rank, prev/next via `from=`/`at=`, `[`/`]`). Boards: `density=compact|comfortable`, keys `j/k x shift+x Enter / Esc ?`; the selection is app-wide (sessionStorage per ws/app, `hooks/useRunSelection.ts`, URL `sel=` only seeds it) in one tray the shell mounts under every app page (`pages/SelectionTrayHost.tsx`; boards register their actions via `useTrayBoard`; `pages/SelectionTray.tsx`: `N runs selected · from M outer runs ▾`, picks by outer run, a picked outer run = one run); a Runs tab's select-all-matching adds a query group `{scope,q,status,count,members}` (`util/selectionGroups.ts`) beside the picks (archive = one by-query action per group + one by verstrs, Compare = picks + each group's top 10 loaded runs, unticking a member turns the group back into picks). `components/OuterBadge.tsx` (`outer · N runs`, N from the row's `job.total` > fleet counts > `children`, `util/outerRun.ts`) marks an outer run in All runs' name cell, the cards, its page header, Compare chips and ⌘K; Compare takes outer runs, and one that logs no metrics itself gets a note on Curves/Table (`pages/Compare/OuterRunNotes.tsx`: `Compare its best run` = summary `top[0]`, else its first run; `Add its N runs`, capped at 20). Index warm restart serves persisted rows first and revalidates in the background (`core/index.py`, `core/gc_pause.py`, `storage/listing_workers.py`); a remote store's large (cold) load publishes and persists every `_REMOTE_CHUNK` (500) records, newest `metadata.yml` first, the snapshot's `loading` = `(loaded, total)` meanwhile (`experiments-status` `indexing` `{loaded, total}`; the web shows `indexing 12,000 / 45,386 runs` on the Outer runs count and the app tile, status polled every 2 s), so the board answers after the first chunk and a cut-short load resumes, its reads `_REMOTE_WORKERS` (32) at a time on a 64-connection S3 pool (`storage/s3_base.py:MAX_POOL_CONNECTIONS`). A journaled (store) sweep re-lists a live run the journal did not name once per its heartbeat interval (`index_sweep.live_relist_sec`: at least `LIVE_RELIST_SEC` = 10 s apart, at most half its stuck window; heartbeats are not journaled), and a journal tick lists the last two minute partitions, its whole skew window every `rescan_sec` (30 s, `core/journal_reader.py`). The per-host index (plan 25) persists lean rows + side tables (`exp_rows`/`exp_sides`, `core/index_lean_store.py`) and folds lazily (`core/index_lazy.py`): a warm start decodes no full record, `show -v`/`reader.get_run`/ref resolution are indexed point reads (`core/index_point*.py`, `SqliteStore.resolve_one`), `list` answers from the persisted rows (`core/index_list.py`); revalidation lists record names only, with persisted listing baselines (`core/index_baseline.py`) and a full stat sweep at most every 300 s (`DEFAULT_FULL_SWEEP_SEC`), so a finished run edited behind vmn's back, on pure S3, or by another host on a cached store shows up within 300 s (the rolling consistency slice is capped per refresh at `SLICE_BUDGET` records and, for journaled store sweeps, at `MAX_RECHECK_PER_SEC`, so a huge store's round just takes longer; `core/index_sweep.py`); a journaled index resumes from its persisted journal mark after a restart (`core/index_journal_mark.py`; past retention it reconciles with a parallel background listing, `core/index_relist.py`, while journal-announced runs keep publishing); the store's `-wal` is capped (`journal_size_limit`, TRUNCATE checkpoints after large saves), saves are chunked (`SAVE_CHUNK`) and superseded index generations are swept once unheld (`ui/cache_tidy.py`); `sys_*` metrics live in a side table off the lean rows (`core/system_metrics.py`: boards/facets/sorts don't carry them; run detail, row copies, compare, `experiments-columns` and queries do); every vmn SQLite file goes through `core/sqlite_conn.py` (WAL, `synchronous=NORMAL`, `BEGIN IMMEDIATE`) and rebuilds flip a `<file>.current` pointer (`core/sqlite_files.py`), never deleting a file another process has open.
+ - `vmn-exp ui`: Serve the web dashboard + REST API (a remote store's `/apps` counts runs from the persisted index, `experiments: null` before it has any — never by listing them) (`pip install "vmn-exp[ui]"`). `--host`, `--port` (8265), `--token`, `--data-dir`, `--repo` (repeatable), `--store <uri>` (`s3://bucket/prefix?endpoint_url=` for MinIO), `--read-only`, `--no-browser`, `--no-index`, `--allowed-host` (repeatable), `--keep-warm`/`--no-keep-warm` (server.yml `server.keep_warm`; unset = store workspaces' indexes keep refreshing while the server runs, watched or not, checkouts' idle out 60 s after their last request, `ui/refresher.py`; the startup warm-up loads every app), `--metrics-ended-hours` (server.yml `server.metrics_ended_hours`, 24). `GET /metrics` (viewer; plan 28 T2a, `ui/metrics_feed.py`, `ui/rollup_gauges.py`, `ui/prometheus.py`) serves `vmn_cache_drift_total` and per `workspace`/`app` gauges: `vmn_runs{status}` (experiments-status counts + `never_started`/`lost`), `vmn_attention{key}` (built-in `stuck`/`failed` (24 h)/`running`/`outlived`, then the attention queries), for top-level outer runs live or ended within the window (<= 200 per app, live first then newest) `vmn_job_runs{outer_run,name,status}`/`vmn_job_progress`/`vmn_job_eta_seconds` (their summary rollups), and `vmn_index_freshness_seconds`; it reads only what an index already published (`ExperimentSource.served_apps`/`published`/`as_of`: never loads, refreshes or waits), off the board's caches (`LeaderboardCache.conf_of` for conf/settings), computed once per published snapshot and minute. `vmn-exp ui backup <path> [--config|--db]` copies the control plane online (SQLite backup API + integrity check; Postgres prints the `pg_dump` command). `vmn-exp ui diag (--pid <PID> [--data-dir] [--wait 30] | --url <base> [--token] | --summarize FILE) [-o FILE]` (`ui/diag_cli.py`, `ui/diag_summary.py`) takes a diagnostic dump (SIGUSR1 → `<data-dir>/diag/diag-<ts>-<pid>.json`, or `GET /api/v1/diag`) and prints a summary; S3 request stats come from botocore hooks on every vmn S3 client (`storage/s3_stats.py`, `add_listener` per call). Sections also cover S3 by object kind (`storage/s3_kinds.py`) and by caller (requests' `s3`, `by_thread`; pools carry the request context via `storage/thread_context.py` and are named, `tests/test_pool_names.py`), traced locks (`core/traced_lock.py`: holder/waiters), background activities (`core/activity.py`), recent warnings (`ui/diag/logs.py`) and event-loop/GIL lag (`ui/diag/latency.py`); `--summarize` reads them (`ui/diag_summary_more.py`); `ui diag` needs no checkout. Postgres access goes through one pool per DSN with timeouts/keepalives/reconnects (`ui/pg_pool.py`); API tokens are HMAC-SHA256 under a per-control-plane key; expired sessions and `audit.retention_days` (server.yml, default 365) are swept hourly (`ui/control_plane_sweep.py`); see docs/vmn-exp/server.md. Without a token, `/api` only accepts loopback/allowed `Host` headers (DNS-rebinding guard); mutations reject foreign `Origin`s and need `Content-Type: application/json`; URL app names/verstrs are validated; the SPA fallback never serves files outside `static/`. Responses are gzipped and NaN-safe. `vmn-exp ui` keeps each watched app's index fresh from a background refresher, so requests read a lock-free snapshot (list/detail stay ~ms at 100k runs); list/detail/columns answer ETag/304; `experiments-facets`, `experiments-columns` (whole-set chart data) and `POST .../series` (batched overlay series) exist. The vmn-native shell (plan 19: context root, status rollup, CLI-grammar ⌘K, sign-in page) reads `/meta` (versions, `mode`, `auth` hints, `principal`, `role`), `/workspaces` `role`, `workspaces/{ws}/git` (branch/HEAD/dirty via one lock-free `git status`) and `.../apps/{app}/experiments-status` (counts + `rollup`, carried across index generations by the row delta, `ui/status_counts.py`; plan 28 P1: `attention: [{key, label, level, count, q}]` per settings/conf `experiment.ui.attention` query (at most 8, `core/app_settings.py`; counts = the board's cached order lengths, `ui/attention.py`; a bad query counts `null` + `error`) and `?scope=<v>` = the whole payload over that subtree from its job rollup; the web strip (`util/attention.ts`, `components/AttentionStrip.tsx`) shows stuck, failed in 24h, the configured parts by level, then running, on All runs, Outer runs and an outer run's page; plan 28 P4: the unscoped payload lists `views: [{name, page: jobs|runs|job, search, default}]` per settings/conf `experiment.ui.views` (at most 20, one name per page, one default per page, `core/app_settings.py:views_entries`; absent when none; in the ETag), the web's team views: `components/BoardViews.tsx` lists the page's above "my views" in `SavedViews.tsx` (`TeamViews.tsx`; applying keeps the page's `view`/`kind`/`tab`, `util/teamViews.ts`), applies a page's default on a landing without view params once per page per tab (`hooks/useDefaultView.ts`, sessionStorage), and gives editors "Save for team" (settings GET → `withTeamView` → `util/yamlDump.ts` → settings PUT, comments not kept); plus `freshness` `{as_of, refreshing, refresh_running_sec, phase, progress, refresh_failing, last_error, stale_after_sec, unreadable?}`, `as_of` persisted in the index, `core/index_freshness.py`; `unreadable` = `{count, store_errors, last_error}` of the records the index set aside unread (`core/index_unreadable.py:Unreadable.summary`; the I/O helper's for a local store), absent when none; the web shows a stale/error banner, `components/FreshnessBanner.tsx`, plus a `N runs could not be read (…); retrying.` warning unless the refresh is failing; the refresher heals itself: every step guarded, a dead thread restarted, failure streaks logged, warming on its own `vmn-ui-warm` worker that keeps only the newest snapshot, `ui/latest_worker.py`); a 401 carries `auth` hints and `WWW-Authenticate: Bearer`; run detail has `idx`/`name`. The list endpoint pages (`offset`/`limit` <= 1000), sorts server-side (`sort`, `sort=timestamp|started_at|finished_at`, `order=asc|desc`; plan 28 FS: `sort=<any other row field>|tags.<k>|params.<k>`, `core/field_sort.py`, numbers before strings, ascending by default, valueless rows last, cached/moved like metric orders; `vmn-exp list --sort` too); board rows carry `duration_ratio` (plan 28 N4, `ui/sibling_durations.py`: `duration_sec` over the median of the finished unarchived siblings with the same parent and `experiment.ui.split_by` value (via `LeaderboardCache(conf_of=)`), from per-(parent, value) sorted durations moved by deltas, P3's rank snapped to its sketch bucket; running = elapsed; null under 3 siblings and for waiting/stuck runs; a group whose median moves gets its members' rows again, exempt from the moved orders' delta limit); `group=parent|forked_from|rerun_of|code_verstr|branch|status|root|tags.<k>|params.<k>` answers groups instead (plan 22 K1, `ui/leaderboard_groups.py` + `_group_spec`/`_group_fields`/`_group_summary`/`_group_cache`, cached per snapshot and moved by deltas): `{groups: [{key, label, target, loose, counts, aggregates: {m: {mean,best,median,min,max,n}}, rows, rows_total, rows_unfiltered}], total_groups, total_rows, running, aggregate}` (`rows_unfiltered` = the group without `q`/`status`, `running` = running rows after the filters, `aggregate` = what `agg=` resolved to), groups ordered by `agg=mean|best|median|min|max` (default conf `experiment.ui.group_aggregate` of the checkout branch's conf (`ui/readers/branch_conf.py`: HEAD read as a file, no git process; a detached HEAD takes the branch vmn resolves it to via `version_stamp.api.checkout_branch_of`, git once per (checkout, HEAD sha)), else mean) of the sort metric (`sort=c:total|c:waiting|c:running|c:done|c:failed` orders groups by that count instead, desc unless `order=asc`, `failed` = failed + stuck, ties and rows in the default order; plan 22 F10), the no-value group last (key `∅`, a value starting with `∅` keyed doubled; label `(none)`; groups[].`loose` marks a link grouping's (`parent`/`forked_from`/`rerun_of`) no-link group, which holds only the runs heading no group (absent when empty; `ui/leaderboard_group_loose.py`; plan 22 F12)), `limit` <= 50 groups and `rows_per_group` 1..100 (<= 5000 rows a page; clamped, never refused), `group_key=` pages one group; web client `webui/src/apiGroups.ts`; run detail is bounded (`log_tail`/`log_total`, per-metric `series` downsampled to `max_points`, `include_log=1` for the full log) and `/experiments/{verstr}/log?offset&limit` pages the log. Run code (plan 20 K2-K4, `ui/routes_run_code.py`, `ui/readers/{patch_files,baselines,failure_context}.py`): `GET .../experiments/{verstr}/patch?max_bytes=` = `show --patch --json`'s `patch` + `files: [{path, added, removed}]` (ETag = code identity); `GET .../experiments/{verstr}/baselines` = `{base?, parent?, forked_from?, rerun_of?, previous?, best?}` each `{verstr, idx, name, status}` from the index snapshot (`previous` = latest earlier same branch + command, `best` = best other run on the primary declared goal); a failed/stuck run's detail adds `output_tail` (`{lines, truncated}`, last 40 lines of `outputs/output.log`, one <= 16 KB range read), `last_step` (`{step, of}`, `of` = numeric `params.epochs`/`max_steps`) and `last_good` (latest earlier success of the same command) — absent for other statuses; web client `webui/src/apiRunCode.ts`. Leaderboard default columns (plan 20 D11, `webui/src/pages/defaultColumns.ts`): `#`/status/experiment, the schema-declared goal metrics (pinned unless `pin=`; `pin=-` = none; all metrics when no goal is declared), up to 6 params that vary across loaded rows (fit to the width), `when` (plan 22 G1: every run is a row, outer and inner alike; no fold, no per-run inner count columns; an outer's experiment cell keeps `tree <status>` and an incomplete sweep's `16/20 · incomplete`). Grouped (`group=`, plan 22 W2: `hooks/useBoardRows.ts` > `useGroupedRows.ts`, `util/groupItems.ts`, `pages/LeaderboardGroupRow.tsx`) the board renders K1's groups: a header row per group (link target `@N name` + status/tree pill, the code verstr + base · branch, `field = value`; counts in words `16 runs · 2 running · 1 failed` / `16 runs · all succeeded`, `5 of 16 runs · 5 succeeded` when groups[].`rows_unfiltered` differs from `rows_total` (plan 22 F1/F2; a cut group never says `all`), with an sr-only sentence; the counts belong to the group's head, not to columns (plan 22 F13): before the count line a run bar in the status colours (finished runs fill from the left, waiting = the empty track; none for a group of one run; `pages/groupHeaderText.ts:barSegments`, `GroupBar`), on the phone section label too; the menu's "Sort groups" row (aggregate / runs / running / failed / waiting / succeeded, `util/groupCountSort.ts`) sorts groups by a count (`sort=c:<name>&order=desc`; a `sort=c:*` is dropped with `group=none`); per metric column the `agg` aggregate titled `mean of 16`, `own <v>` for a target's own value, metric headers tagged with the aggregate), up to 10 rows, a `+N more` row (`group_key` paging), `collapse=<key>`, a loose group's rows as plain unindented rows after the groups with no header (a plain `+N more` row; plain cards on the phone; no group colour in Charts), 50 groups per page; the `Group by ▾` menu next to Columns (`components/GroupByMenu.tsx`, `util/groupBy.ts`, `hooks/useBoardGrouping.ts`) offers none, Links (parent run, forked from, rerun of, same code), Fields (branch, status, `tags.<k>`, `params.<k>` with <= 20 distinct values, from facets `tag_keys`/`param_values`), Sort groups and the Aggregate; `group=` absent = `parent` when facets `parented` > 0, else `none` (flat, sorted/paged as before); `agg=` absent = the app default (the request omits it; the header tags and the menu show the payload's top-level `aggregate`, plan 22 F3, `hooks/useBoardAggregate.ts`); grouped, the board polls while the payload's top-level `running` > 0 (F4); on a phone the menu is viewport-fixed under its trigger (`hooks/useAnchorTop.ts`); saved views keep `group=`/`agg=`/`collapse=` with the rest of the URL; a header's checkbox selects its loaded rows (indeterminate when partial; Compare (N)/Archive count rows), and the Charts tab's bar/scatter/parallel colour runs by group (`experiments-columns` key `group:<field>`, `util/chartGroups.ts`, legend `components/ChartGroupLegend.tsx`); with rows selected the Charts tab plots only them (plan 22 F7, `hooks/useChartSelection.ts`: `Showing N selected · Show all`, Show all keeps the selection and lasts until it changes); note/tags/started/ended and `c:queue`/`c:duration`/`c:ratio` (`3.1× siblings`, sortable) via `show=`, and a column per tag key (plan 28 P4: `t:<key>`, the picker's tags section from facets `tag_keys` + the rows', `leaderboardColumns.ts:tagColumns`; header sorts `sort=tags.<k>`, a to z first); `show=m:|p:|c:|t:<name>` turns a column on, `hide=` off (a schema-hidden metric keeps `hide=` = shown). The leaderboard filter is one grammar (`components/LeaderboardFilter.tsx` = `QueryBar` tokens + text, `FilterShortcuts`, `SimilarToggle`; pure `util/filterTokens.ts`): status/branch/archived shortcuts and hand-typed `status`/`branch`/`text ~` clauses become the same removable tokens, URL params unchanged (`q`, `status`, `branch`, `search`, `archived`); similar mode keeps every token. Below 640px (`hooks/useMediaQuery.ts` `NARROW_QUERY`) the leaderboard renders cards instead of the grid (plan 20 C8, `pages/LeaderboardCards.tsx`): name + status (outers: tree status), the sorted-by metric (else the primary, else the first shown) large, `@N · branch · when`, a per-card checkbox plus select-all; grouped, each group is a sticky section label above its cards (plan 22 G9, `pages/LeaderboardCardGroup.tsx`: the group checkbox (F8), fold, name, status count (wraps, F2), the group's aggregate on that metric) with a `+N more` button; the toolbar popovers span the viewport; touch screens (`TOUCH_QUERY`) get no ⌘K key hints, and the phone identity chip shows an initial only for a signed-in name. Compare (plan 20 D6, `webui/src/pages/Compare/`) is one page, `compare-runs?tab=curves|table|code&sel=…` (repeated `sel` shared by the tabs, selection chips; the code tab diffs the first run against the second or `?to=`, K6 422 reasons shown); the plan-20 `overlay?runs=` / `compare?v=&to=` pages are gone (plan 22 G8, no redirect). The run page (plan 20 D5, `webui/src/pages/RunLayout.tsx`) reads header (`@N`, status, base) > note/tags > failure card > curves > Changes > Fleet/Sweep > Params | Metrics > Similar | Lineage > outputs, then `components/Disclosure.tsx` sections (Status details, Log, Environment closed; Media, Comments open when present), each choice remembered in localStorage `vmn-exp:disclosures`. With the GIL on and a watched app of 10k-200k runs it logs a one-time hint to run it on free-threaded 3.14t (`uvx --python 3.14t --from "vmn-exp[ui]" vmn-exp ui`; ~10x the throughput at 100k runs); from 200k runs it doesn't, and on 3.14t it warns once to use a GIL build (mimalloc's per-thread heaps keep 3-5x the live data, and stop-the-world collections grow with them; `ui/gil_hint.py`). The `ui` extra has no orjson (no free-threaded wheels); the `orjson` extra adds it. The diag dump's `gc` section (`ui/diag/gc_pauses.py`) records collection pauses. See docs/vmn-exp/ui.md.
+ - Pivot heatmap (plan 28 N2; `ui/pivot.py`, `ui/pivot_cache.py`, `ui/routes_pivot.py`): `GET .../experiments-pivot?rows=<field>&cols=<field>[&cell=count|share&match=<query>|agg&metric=<m>&agg=][&scope=&q=&status=&archived=]` crosses the runs by two group fields (`key_of`/`value_of`, so `tags.`/`params.`/`up.*`/`branch`/`status`/`root` (now a board group field too)/links): `{rows, cols: [{key, label, n, clause}], cells: [[{n, value, clause, hits?} | null]], total, rows_total, cols_total, cell, match, metric, agg}`, each axis its 200/50 most common values ordered by value, `(none)` last; memoized per (base, bucket), and a scope with nothing live under it is kept across generations while its subtree's rows (and the chain above it) are the same objects, under an ETag of its own. Web: `components/PivotHeatmap.tsx` (one-hue accent fill, `util/pivotCells.ts`, scale legend, cell/header links), the outer run's Overview card `pages/Job/JobPivot.tsx` (field pickers from facets) and the `pivot` report panel (`reports/PivotPanel.tsx`; spec `rows`/`cols`/`cell`/`match`/`metric`/`agg`/`scope` in `core/panel_spec.py`).
+ - Baseline compare (plan 28 N3; pure `ui/baseline_choice.py`, `ui/baseline_compare.py`, cached `ui/baseline_views.py`, `ui/routes_baseline.py`; settled helpers and `SettledMemo` shared with pivots in `ui/settled_subtree.py`): conf/settings `experiment.ui.baseline: {match: [params.<k>|tags.<k>...] (<= 8), children_by: <field>}` (`core/app_settings.py:baseline_conf`, checked like `ui.attention`). `GET .../experiments/{v}/baseline[?to=&match=a,b&by=]` (404 not an outer run / unknown `to`, 400 bad field / `to` = itself): the baseline is `to`, else the latest peer (same parent) tagged `baseline=true`, else the latest peer that ended before it was created, equal on the match fields; answers `{baseline: {verstr, idx, name, status, source}, metrics: {m: {best, median}}, failed_pct, duration_p90, attention (shares under each subtree, pinned), own (metrics both log), pairs (direct children by `children_by`, key order, <= 200 listed, main metric = own else subtree best, `done`/`worse`), pairs_total, chip}`, each delta `{value, baseline, delta, better}`; the chip counts finished pairs worse than the baseline children's p10 (p90 for a min goal): `▼ best_value from 2024-12 (5 of 41 trade_date values worse than the baseline's p10)`, `pending` until one finished, without `children_by` the best's change. `GET .../experiments-baselines?v=...` (<= 200) = `{chips: {v: chip + baseline | null}}` for a Jobs view page in one request (not on board rows). Memoized per (base, bucket); a pair with both subtrees settled is kept across generations (own ETag) until either's rows change; attention shares (`attention_payload` per side) are memoized per generation apart, so `ago()` never rebuilds a settled pair; peers come from `JobTree.peers`. Web: `apiBaseline.ts`, `components/BaselineChip.tsx`, the Overview card + header chip `pages/Job/JobBaseline.tsx`, Jobs view rows/cards via `pages/Experiments/useBaselineChips.ts` (`JobContext.baselines`), `styles/baseline.css`.
+ - Hot board views: Single runs (`top=singles`) is the runs board with single-run columns (`pages/Experiments/SinglesView.tsx`); a second after a view shows, the first page of the other views is prefetched under the current filters (`useViewPrefetch.ts`, fresh 30 s); index-backed answers (first pages, facets, statuses, apps, job summaries; never per-run reads or tokens) persist in localStorage `vmn-exp:query-cache:v1` (3 MB LRU, `src/queryPersist.ts`, cleared on sign-out) and paint at startup, then revalidate; the server warms each new generation's facets and default first pages (`ui/leaderboard_warm.py`; under the app's conf/store settings schema + its runs' declarations and `ui.progress`, as `LeaderboardCache.conf_of` resolves them). Hovering a run fetches nothing; slow per-run requests go through a 2-in-flight gate per tab (`requestGate.ts`); one live stream per tab (`hooks/liveHub.ts`); a slow read whose client left answers 499 unread (`ui/slow_reads.py`).
+ - Outer runs at scale (plan 23, docs/plans/23-experiments-at-scale.md; an *outer run* (`kind = "outer"`; *job* in code, URLs and API fields) = a run with children, thousands of runs each; every user-visible string says outer run). The experiments page opens on **Outer runs** (`view=jobs`, switch `Outer runs (N) · Single runs (N) · All runs`, last view per app in localStorage; `webui/src/pages/Experiments/`): one row per top-level outer run with run bar, `done/expected · failed · stuck · rate · ETA` (`finished`/`ended · N never started`/`stalled`), best (`—` without a goal), spread; `job:*` sorts (`job:attention`: each row's `job.attention` `{key: n}` and `attention_rank`, error-level matches + stuck first, `ui/attention.py:AttentionCounts`). Declared progress (plan 28 P2, `core/progress.py`): an outer run whose own row has finite `progress`/`progress_total` (`run.set_progress(done, total, unit)`: step-less metrics + `progress_unit` param; `NoOpRun` too) gets `job.declared_progress` `{done,total,unit,rate_per_min}` and its `progress`/`eta_sec` (rate since its own start), while `done`/`expected`/`counts`/`throughput_per_min` stay runs; the bar, line (`41/60 dates · 1,415 runs · …`), rate cell and done tile (+ `runs done`) read it (`util/jobProgress.ts`, `pages/Job/jobData.ts`); other names come from conf/store settings `experiment.ui.progress: {done, total, unit}` (validated in `core/app_settings.py`; `RollupSpec.progress` via `JobRollupCache.progress_names`, which reads `LeaderboardCache.conf_of(snapshot)` — the server resolves a snapshot's (workspace, app) by `ui/snapshot_owners.py` — and keys the spec/answer memos and every board ETag by it, so a settings change shows at once). Server: per-job rollups over each subtree moved by index deltas (`ui/job_rollup*.py`, `job_tally.py`, removable `quantile_sketch.py`, `failure_signature.py`; no goal → the app's most common non-param metrics, `ranked: false`); rows gain `root`/`failure_sig`, query `under("<v>")`; `experiments?top=1|jobs|singles` (+ `match_subtree=1` → `matched`, `jobs_matched`/`runs_matched`), `scope=<v>` on list/columns/facets/importance (`ui/leaderboard_scope.py`), groups carry `clause` (+ job aggregates with `top`), `experiments/{v}/summary` and `/failures` (`ui/routes_job.py`, `job_views.py`; a job's run detail has no `fleet.children`; plan 28 P3: summary `?split=tags.<k>|params.<k>` (default conf `experiment.ui.split_by`; store workspaces: none until A0's settings) adds `split` + `by` = per value counts/done/failed_pct/duration p50/p90 + `clause`, at most 12 by runs then `other`, from one `ui/job_split.py` tally per (parent, value) — ≤ 12 values a parent in key order, the rest its `other` — moved by the same `StaticRollups` deltas; `job_split_payload.py` never lists a value a parent may hold under `other`; web `pages/Job/JobSplitTable.tsx` under the KPI tiles, page `?split=` passed through), `search?text=` (⌘K; `ui/search_index.py` also prefilters `~` clauses), `events` SSE (`ui/live_events.py`, `routes_events.py`; web `hooks/useLiveScope.ts`, polling fallback), bulk `actions/exp_archive|exp_unarchive|exp_tag` by `{scope,q,status}` with `dry_run` (`ui/bulk_actions.py`). The outer run's page (`webui/src/pages/Job/`): header + run bar + `outer · N runs` badge, tabs `overview|runs|charts|failures|code|log`; the Runs tab is the scoped board with a facet sidebar and select-all-matching; inner run pages get an outer run context bar (rank, prev/next via `from=`/`at=`, `[`/`]`). Boards: `density=compact|comfortable`, keys `j/k x shift+x Enter / Esc ?`; the selection is app-wide (sessionStorage per ws/app, `hooks/useRunSelection.ts`, URL `sel=` only seeds it) in one tray the shell mounts under every app page (`pages/SelectionTrayHost.tsx`; boards register their actions via `useTrayBoard`; `pages/SelectionTray.tsx`: `N runs selected · from M outer runs ▾`, picks by outer run, a picked outer run = one run); a Runs tab's select-all-matching adds a query group `{scope,q,status,count,members}` (`util/selectionGroups.ts`) beside the picks (archive = one by-query action per group + one by verstrs, Compare = picks + each group's top 10 loaded runs, unticking a member turns the group back into picks). `components/OuterBadge.tsx` (`outer · N runs`, N from the row's `job.total` > fleet counts > `children`, `util/outerRun.ts`) marks an outer run in All runs' name cell, the cards, its page header, Compare chips and ⌘K; Compare takes outer runs, and one that logs no metrics itself gets a note on Curves/Table (`pages/Compare/OuterRunNotes.tsx`: `Compare its best run` = summary `top[0]`, else its first run; `Add its N runs`, capped at 20). Index warm restart serves persisted rows first and revalidates in the background (`core/index.py`, `core/gc_pause.py`, `storage/listing_workers.py`); a remote store's large (cold) load publishes and persists every `_REMOTE_CHUNK` (500) records, newest `metadata.yml` first, the snapshot's `loading` = `(loaded, total)` meanwhile (`experiments-status` `indexing` `{loaded, total}`; the web shows `indexing 12,000 / 45,386 runs` on the Outer runs count and the app tile, status polled every 2 s), so the board answers after the first chunk and a cut-short load resumes, its reads `_REMOTE_WORKERS` (32) at a time on a 64-connection S3 pool (`storage/s3_base.py:MAX_POOL_CONNECTIONS`). A journaled (store) sweep re-lists a live run the journal did not name once per its heartbeat interval (`index_sweep.live_relist_sec`: at least `LIVE_RELIST_SEC` = 10 s apart, at most half its stuck window; heartbeats are not journaled), and a journal tick lists the last two minute partitions, its whole skew window every `rescan_sec` (30 s, `core/journal_reader.py`). The per-host index (plan 25) persists lean rows + side tables (`exp_rows`/`exp_sides`, `core/index_lean_store.py`) and folds lazily (`core/index_lazy.py`): a warm start decodes no full record, `show -v`/`reader.get_run`/ref resolution are indexed point reads (`core/index_point*.py`, `SqliteStore.resolve_one`), `list` answers from the persisted rows (`core/index_list.py`); revalidation lists record names only, with persisted listing baselines (`core/index_baseline.py`) and a full stat sweep at most every 300 s (`DEFAULT_FULL_SWEEP_SEC`), so a finished run edited behind vmn's back, on pure S3, or by another host on a cached store shows up within 300 s (the rolling consistency slice is capped per refresh at `SLICE_BUDGET` records and, for journaled store sweeps, at `MAX_RECHECK_PER_SEC`, so a huge store's round just takes longer; `core/index_sweep.py`); a journaled index resumes from its persisted journal mark after a restart (`core/index_journal_mark.py`; past retention it reconciles with a parallel background listing, `core/index_relist.py`, while journal-announced runs keep publishing); the store's `-wal` is capped (`journal_size_limit`, TRUNCATE checkpoints after large saves), saves are chunked (`SAVE_CHUNK`) and superseded index generations are swept once unheld (`ui/cache_tidy.py`); `sys_*` metrics live in a side table off the lean rows (`core/system_metrics.py`: boards/facets/sorts don't carry them; run detail, row copies, compare, `experiments-columns` and queries do); every vmn SQLite file goes through `core/sqlite_conn.py` (WAL, `synchronous=NORMAL`, `BEGIN IMMEDIATE`) and rebuilds flip a `<file>.current` pointer (`core/sqlite_files.py`), never deleting a file another process has open.
+ - `vmn-exp settings <app> [show|set -f <file>] [--store|--dir|--bucket]` (plan 28 A0, `core/app_settings.py` pure parse/validate/overlay, `core/app_settings_store.py`, `cli/settings.py`, `ui/routes_settings.py`): an app's store settings, `settings/<app-key>/app/experiment.yml` (the file as written; `metadata.yml` has `updated_at`/`updated_by`), git-free. Keys = the reader-facing part of conf's `experiment:` block: `metrics.<name|glob>.{goal,summary,hidden,primary,step_metric}` and `ui.*` (`group_aggregate`/`stale_after_min`/`progress`/`attention`/`views` validated, other `ui` keys kept for later items); unknown/bad keys refused on write, a broken stored file reads as `{}` with a warning. Read/written on the shared copy (`store_marker.authoritative`: a local-first storage's remote); both writes journaled. Precedence: checkout conf (per metric name / `ui` key) > store settings > run declarations — the server's `_app_schema`/`_group_aggregate`/stale window (`ExperimentSource.app_settings`, a `ui/settings_cache.py:SettingsCache` dropped by the journal follower's `on_settings`, else re-read every `SETTINGS_RECHECK_SEC` = 60 s; inline refresh re-reads per request; a git workspace reads its checkout store's settings below conf), `list_runs()`/`get_run`/`similar_runs` and the CLI readers (`cli/views.py:metrics_schema`: `list`/`show`/`compare`/`diff`/`prune`). API `GET/PUT .../apps/{app}/settings` (`{text}`; PUT editor, refused read-only; `{text, settings, error, updated_at, updated_by}`). Web editor: route `app-settings` at `/ws/{ws}/app/{app}/settings` (`webui/src/pages/AppSettings.tsx`, `apiSettings.ts`, `http.ts:put`): a YAML textarea + Save (a 400's reason shown, the edit kept; a stored file that does not parse is flagged), read-only with the page scope's reason for viewers / read-only servers / stores without edit access; crumbs `<app> / settings`, CLI twin `vmn-exp settings <app> show`, ⌘K `settings [<app>]` (no sidebar or Go to item yet). See docs/vmn-exp/server.md#app-settings.
- `vmn-exp model <action> [model] [flags]`: Model registry — link named, versioned model identifiers to experiment runs. Actions: `register` (attach a run ref + artifact path to a new version; `--kind dataset` with `--uri <uri> [--digest]` registers a reference dataset — local paths hashed, same digest deduped), `alias` (point a mutable alias at a version number), `list` (`--kind model|dataset`; UI API `?kind=`), `show`, `resolve` (print version metadata for a ref), `deprecate`, `delete` (refused while an alias points at it; warns when runs used the version). All actions are git-free; storage resolved from `--dir`/`--store`/`--bucket` or env. Refs: `model`, `model@latest`, `model@N`, `model@alias`. `register --alias <name>` sets an alias on create; `alias --expect <N>` is a CAS guard. Prune refuses registered runs even with `--force`; delete the model version first. SDK: `from vmn_exp.sdk.models import register_model, set_alias, remove_alias, get_model_version, list_models, download_model`; `run.register_model(name, artifact_path=, alias=)`. Datasets share the namespace (header `kind: model|dataset`, absent = model): `register_dataset(name, uri)` (reference; a file's sha256 / a dir's manifest digest, `dedupe=True`) or `register_dataset(name, run=, artifact_path=)` (copied, prune-protected), `get_dataset_version(ref)`. Usage: `use_model(ref)`/`use_dataset(ref)` (also `run.use_*`; `download_model(record=True)` inside a run) log an input named `<name>@<N>` with the producer artifact's `vmn://` URI (reference datasets: `vmn-registry://<name>@<N>`) and append a best-effort `use` entry to the record `registry/<name>/uses` (never the model's log); `get_model_version` never records; `NoOpRun.use_*` only resolve. See docs/vmn-exp/models.md.
- `vmn skill`: Print the AI-agent skill block to stdout. `--install` writes it instead (`--target claude` → `.claude/skills/vmn/SKILL.md`, `cursor` → `.cursorrules`, `agents` → `AGENTS.md`); `--force` overwrites an existing Claude SKILL.md. Cursor/agents targets only rewrite vmn's marker block and preserve surrounding text. Core-only: it ends with a one-line pointer to `vmn-exp skill`.
- `vmn-exp skill [--install] [--target claude|cursor|agents] [--force]`: the experiment/UI-fleet/similar-experiments/model-registry skill block (`vmn_exp/cli/skill.py`, handled in `vmn_exp/cli/main.py` before the vmn-command refusal). Installs through `version_stamp.api.run_skill` (vmn's machinery, parameterized by name/description/body/markers): claude → `.claude/skills/vmn-exp/SKILL.md`; cursor/agents use distinct `vmn-exp skill` markers, so vmn's block in the same file is untouched.
- `vmn config <name>`: TUI config editor. `--vim` for $EDITOR, `--global` for repo-level config. `--branch` edits the current branch's canonical branch conf (seeded from the effective conf).
- `vmn config gen <name>`: Non-interactively create a config file (no TTY needed, for CI/scripting). Default creates `conf.yml`; `--branch` (± `--root`) creates the canonical branch conf seeded from the existing effective conf. Never overwrites an existing file.
- `vmn worktrees create <name>` (alias `vmn wt`): Create an island (git worktrees for main repo + all deps). `--island-name`, `-fv`/`--from-version`, `-fb`/`--from-branch`, `--base-path` (default `../vmn-islands`), `--shallow-deps`, `--carry-changes` (apply the app's and deps' uncommitted tracked edits as a patch and copy their untracked non-ignored files into the island; a checkout not at the source's commit is skipped with a warning; without the flag, `create` only warns). `create` is the default action, so `vmn wt <name>` works.
- `vmn worktrees list`: List active islands.
- `vmn worktrees remove <island>`: Clean up an island (removes its worktrees, private branches, and the `vmn-readonly` remote once no island uses it).
- `vmn worktrees freeze <name>`: In the app checkout, pin every dep that is on a real (non-`island/`) branch to that branch in the current branch's canonical branch conf. Refuses on an `island/` branch, on detached HEAD, or while a dep has commits only on its private branch. Never commits or pushes.
- `vmn worktrees pull [island]`: `git pull --rebase` in every island checkout still on its private branch (skips repos moved to a real branch; dirty repos and conflicts exit 1). Without a name, finds `island.json` above the cwd.
- `vmn --completion [SHELL]`: Print shell completion setup script (bash/zsh/fish/tcsh). Auto-detects shell.
- `vmn --completion-install [SHELL]`: Append completion to shell rc file. Idempotent.
- `vmn --completion-uninstall [SHELL]`: Remove completion from the shell rc file. Idempotent.
### Islands (worktrees)
- Layout mirrors the source: every checkout sits at its path relative to the common parent of the app and its deps, so configured dep paths like `../libs/B` resolve inside the island.
- Source branch = the branch each repo was on (or `--from-branch`); `--from-version` islands have none. Each repo with a source branch gets private branch `island/{name}/{source}` tracking `vmn-readonly/{source}` with `pushRemote=vmn-readonly`. `vmn-readonly` is a per-repo remote with origin's fetch URL, an unusable push URL, and refspecs only for islands' source branches; vmn never selects it as its remote.
- Dep start point comes from the conf pin: hash/tag → that ref (detached); branch X → the local checkout if it is on X, else `vmn-readonly/X` fetched; no pin → the local checkout.
- `stamp`/`release`/`add`/`init-app` are refused on an `island/` branch (checked before the lock). A real branch checked out inside an island stamps normally once pushed. A dep on `island/{name}/X` with no commits beyond its upstream counts as synced with a `branch: X` pin.
- `island.json` in the island root is the machine-readable manifest (paths, private `branch`, `source_branch`, `upstream`, dep hashes, `shallow_deps`).
- vmn never borrows another branch's upstream: a branch whose upstream is missing (or on another remote) needs `git push -u origin <branch>` before `vmn stamp`.
- Islands are for app + deps work next to `goto`, not the way to spawn parallel worktrees — the worktree methodology uses plain `git worktree`.
## Environment Variables
- `VMN_WORKING_DIR`: Override the working directory for vmn
- `VMN_LOCK_FILE_PATH`: Custom lock file path (default `.vmn/vmn.lock`, a per-repo lock preventing concurrent vmn commands). Purely a user-facing override — vmn never injects it into a child process's environment. `exp run` and the SDK hold this lock only for their create/auto-init phase, not for the run's lifetime.
- `GITHUB_TOKEN` / `GH_TOKEN`: Required for GitHub Releases feature
- `VMN_GIT_PUSH_USER` / `VMN_GIT_PUSH_TOKEN`: Fallbacks for `stamp`/`release` `--git-push-user`/`--git-push-token`
- `VMN_UI_TOKEN`: Fallback for `vmn-exp ui --token`
- `VMN_EXP_MIN_STALE_SEC`: Overrides the 60s floor of the stuck window (`max(3 * interval, floor)`), read wherever status is derived (CLI, `vmn-exp ui`). Non-positive/invalid values keep 60.
- `VMN_EXP_KILL_GRACE_SEC`: Fallback for `vmn-exp run --kill-grace-sec` (seconds a forwarded signal waits before SIGKILL)
- `VMN_EXP_CACHE_DIR`: Base dir for vmn-exp's per-host state (index cache, push ledger) instead of the XDG cache/state dirs (containers with a read-only home)
- `VMN_EXP_FINAL_UPLOAD_TIMEOUT_SEC`: Seconds the SDK waits for a run's last remote uploads when it is finalized (`finish()`, SIGTERM, interpreter exit; default 60). Several runs finalized together share this one wait (each drains its media uploads before its final log flush, then all final states are written before the remaining uploads are awaited). Non-positive/invalid values keep 60.
- `VMN_SNAPSHOT_MAX_FILE_MB` / `VMN_SNAPSHOT_MAX_TOTAL_MB`: caps on untracked files captured into a snapshot/experiment tarball (defaults 50 / 200; skipped paths are recorded as `untracked_skipped`)
- `VMN_EXP_AUTHOR`: The email a new run's `features` author gets when git config has no `user.email` (git mode: before the first local commit's `From:`; git-free: after the export's `exported_by`).
- `VMN_CAPTURE_ENV`: Set to `0`/`false`/`no`/`off` to disable automatic environment capture (Python version, platform, installed packages) on `vmn-exp create`, `vmn-exp run`, and `start_run()`. CLI `--no-env` and `start_run(capture_env=False)` also opt out; `start_run(capture_env=True)` overrides this variable and conf.yml but cannot override the CLI flag. Opt-out precedence: CLI flag > `VMN_CAPTURE_ENV` > conf `experiment.capture_env`.
- `VMN_EXPERIMENT_DIR` / `VMN_SNAPSHOT_METADATA`: git-free experiment mode (container images built from `vmn-exp export`) for both the CLI and `start_run()`
- `VMN_MODE`: `disabled` makes `start_run()` return a no-op run (no git/store access), `autolog()` patch nothing, and `vmn-exp run <app> -- cmd` exec `cmd` directly (no lock/snapshot/run_state; `VMN_METRICS_FILE=/dev/null`). Explicit `start_run(mode="enabled")` wins; other CLI actions ignore it.
- `VMN_EXPERIMENT_STORE`: fallback for `--store` (a store URI; beats the bucket shorthand)
- `VMN_EXP_OFFLINE`: `1`/`true`/`yes`/`on` drops the remote store everywhere (CLI, SDK, `vmn snapshot`; beats flags, env and conf) so runs record to the local root only, named `<code_verstr>.<writer_id>[.N]`; upload later with `vmn-exp push` (which ignores it)
- `VMN_SYSTEM_METRICS`: `0`/`false`/`no`/`off` disables default `sys_*` sampling
- `VMN_EXP_OUTPUT_CAP_MB`: fallback for `vmn-exp run --output-cap-mb`
- `VMN_EXP_ALERT_WEBHOOK_URL` / `VMN_EXP_ALERT_SLACK_URL` / `VMN_EXP_ALERT_COMMAND` / `VMN_EXP_ALERT_ON`: add alert sinks / set triggers
- `VMN_EXPERIMENT_BUCKET` / `VMN_EXPERIMENT_PREFIX` / `VMN_EXPERIMENT_ENDPOINT_URL`: fallbacks for the experiment `--bucket`/`--prefix`/`--endpoint-url` (flags > env > conf.yml), honoured by the CLI and `start_run()`; with a bucket and no local dir, runs record straight to S3
- Set *by* vmn for the `vmn-exp run` child process: `VMN_EXPERIMENT_ID`, `VMN_APP_NAME`, `VMN_METRICS_FILE`, `VMN_EXP_SUPERVISOR_SAMPLES` (when it samples system metrics); sweep trials also get `VMN_SWEEP_PARAMS`/`VMN_SWEEP_ID`/`VMN_SWEEP_TRIAL`. `VMN_EXPERIMENT_ID` also drives auto-parenting — any experiment created while it is set becomes an inner job of that run.
## Docs Layout
Core vmn and vmn-exp are documented separately (vmn-exp will become a separate product): core docs never carry experiment material beyond one link to `docs/vmn-exp/README.md`.
- `README.md`: core vmn only. Skimmable top level with reference material (commands, flags, conf.yml keys, backends, env vars) inside `<details>` blocks; deep guides live in `docs/` and are linked, not inlined. Keep it that way — don't paste long reference back into it.
- `docs/agent-skill.md`: **generated** from `vmn skill`. Regenerate it (don't hand-edit) whenever `packages/vmn/src/version_stamp/cli/skill.py` changes.
- `docs/vmn-exp/agent-skill.md`: **generated** from `vmn-exp skill`. Regenerate it (don't hand-edit) whenever `packages/vmn-exp/src/vmn_exp/cli/skill.py` changes.
- `docs/snapshots.md`: the `vmn snapshot` guide (core vmn; vmn-exp stores in one closing section). `docs/packaging.md`: the three-package split, import rules, and how releases version each package. `docs/agent-methodology.md`: optional dev rules for AI agents.
- `docs/vmn-vs-*.md`, `docs/migrating-from-*.md`: comparison/migration guides for core vmn; they link to README anchors instead of repeating setup.
- `docs/vmn-exp/`: all vmn-exp docs, indexed by `docs/vmn-exp/README.md`. One canonical home per topic; other docs summarize and link:
- `client-guide.md`: task walkthrough (store choice, submit/sweep, Kubernetes git-free recipe, in-job SDK, watch, compare, restore/goto/rerun/export, resume/rewind/fork, models, prune).
- `experiments.md`: `vmn-exp` CLI reference, status/alerts, nesting, storage + plugin contract, offline + `push`, env vars.
- `sdk.md`: the `vmn_exp.sdk` Python SDK (`start_run`, logging, autolog, readers, query language, integrations).
- `ui.md`: `vmn-exp ui` deployment + HTTP API. `models.md`: model/dataset registry. `sweeps.md`: sweeps.
- `server.md`: `vmn-exp ui` as a central server (`server.yml`, SQLite cache, change journal, resync/rebuild, OIDC/API tokens/roles). `byo-bucket.md`: store areas, the server's permission sets, journal lifecycle rule. `reports.md`: reports, panels, publish/export, comments.
- `ai-fleet-tracking.md`: for AI agents — which SDK/CLI call moves each UI fleet count (the parent group header). `vmn-vs-mlflow.md`, `migrating-from-mlflow.md`.