CLAUDE.md git:20261016.f268a43A

Immutable. This exact content never changes and is served at /api/v1/blob/a8a65d7f0ef9909f.

# CLAUDE.md

# Claude Code instructions

When generating commit messages, pull request text, patches, or any code-related output, never include this line:

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Omit any Claude co-author trailer unless I explicitly ask for it.

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

# Splitting tasks
Split big tasks into separate worktrees and do in parallel, but TDD takes precedence.
Each worktree agent must follow TDD internally: write its tests first (red), then implement (green).
If multiple worktrees touch independent features, each worktree owns its own red-green-refactor cycle.
Each worktree agent should write ~200-300 lines max per file. If writing more, split into additional worktrees.
When finished - always try /simplify on the change.

## Timeout handling
If you see API timeouts, split the current task into smaller parallel worktrees rather than retrying.

## Worktree hygiene
- Never push worktree branches to remote.
- When a worktree agent finishes, immediately remove the worktree (`git worktree remove --force`) and delete its local branch (`git branch -D`).
- Before starting new work, check for and clean up any stale worktrees from previous sessions (`git worktree list`).

## Permissions hygiene
- Do not accumulate one-off permission rules in `.claude/settings.local.json`.
- Prefer broad wildcards (e.g., `Bash(git:*)`) over specific subcommand rules.
- Keep the allow list under 30 entries.

## Code practices
- Keep functions small and single-purpose; prefer clear names over comments.
- Don't add abstractions, config flags, or error handling for cases that can't happen — match the scope of the change to what was actually asked.
- Reuse existing helpers/patterns in the codebase instead of duplicating logic.
- Keep diffs minimal and focused; don't refactor unrelated code in the same change.
- Run the relevant tests (see Running Tests) before considering a change done.

## Test-driven development (required)
All feature work and bug fixes must follow strict TDD:
1. Write the test first. It must fail for the right reason (red).
2. Write the minimum implementation code needed to make it pass (green). Do not modify the test to make it pass.
3. Refactor implementation code only, keeping tests green.

Rules:
- Never edit a test to force a passing result — if the test seems wrong, stop and ask before touching it.
- Do not write implementation code before its test exists.
- Each new behavior gets a test asserting it before any code implements it.

## Project Overview

vmn is a CLI tool and Python library for automatic semantic versioning. Versions live in git annotated tags — zero lock-in, zero databases.

Key differentiators vs semantic-release/release-please:
- Language-agnostic (not JS-centric)
- Multi-repo dependency tracking with `vmn goto` state recovery
- Microservice topology (root apps with independent service versions)
- 4-segment hotfix versioning (`major.minor.patch.hotfix`)
- Auto-init on first `vmn stamp` — no separate `vmn init` required
- Works offline, with shallow clones, in air-gapped environments

## Development Setup

The repo is a uv workspace of three packages (see docs/packaging.md):
`packages/vmn` (`version_stamp`), `packages/vmn-exp-sdk` (the job-side
`vmn_exp.sdk`/`storage`/`core`/`registry`/`integrations`/`_base`) and
`packages/vmn-exp` (`vmn_exp.cli`/`ui`/`snapshot`/`importers`/`gitmode`, the
`vmn-exp` command and `webui/`). `vmn_exp` is a PEP 420 namespace package split
across the last two — never add a `vmn_exp/__init__.py`.

```sh
python3 -m venv ./venv
source ./venv/bin/activate
pip install -U pip "setuptools>=64"
pip install -r ./tests/requirements.txt
pip install -r ./packages/vmn-exp/tests/test_requirements.txt  # core + vmn-exp test deps
pip install -e packages/vmn -e packages/vmn-exp-sdk -e "packages/vmn-exp[ui]"
vmn --version  # e.g. 0.10.2-rc.11+dev: a source checkout adds +dev
```

`uv sync` also works from the repo root. Build the wheels with
`uv build --all-packages` (or `make _build NAME=vmn|vmn_exp`).

Import rules (packages/vmn-exp/tests/test_packaging_split.py): `version_stamp` never imports
`vmn_exp`; the vmn-exp-sdk subpackages import neither `version_stamp` nor the
vmn-exp subpackages (except a lazy `vmn_exp.gitmode`); vmn-exp reaches
`version_stamp` only through `version_stamp.api`. vmn loads `vmn goto` of dev
versions (`register_dev_version`) from the `vmn.plugins` entry point that
vmn-exp declares.

## Running Tests

Two suites (vmn-exp will become a separate product):
- **core** — `tests/`: vmn alone. `tests/conftest.py` puts only `packages/vmn/src`
  on `sys.path` and makes `vmn_exp` unimportable (`tests/no_vmn_exp.py`), so it
  passes without vmn-exp installed; deps in `tests/test_requirements.txt`.
- **exp** — `packages/vmn-exp/tests/`: vmn-exp. Its conftest adds all three
  `src/` roots plus `tests/`, reusing the core fixtures (`tests/vmn_fixtures.py`)
  and helpers (`exp_helpers.py` = `tests/helpers.py` + `_experiment`/`_exp`/
  `_storage`/`cli_module`); deps in `packages/vmn-exp/tests/test_requirements.txt`
  (includes the core file). A test that needs `vmn_exp` or a vmn-exp command
  belongs here; one asserting vmn works *without* vmn-exp belongs in core.

Tests require Docker. Activate the test venv first — the suites run through the
active interpreter's `coverage`/`pytest`, and the release-notes tests need the
`git-cliff` binary that `tests/test_requirements.txt` installs into it.
```sh
source ./venv/bin/activate
./tests/run_pytest.sh                        # core suite (= --suite core)
./packages/vmn-exp/tests/run_pytest.sh       # exp suite (= tests/run_pytest.sh --suite exp)
```

Run a specific test:
```sh
./tests/run_pytest.sh --specific_test <test_name>
./packages/vmn-exp/tests/run_pytest.sh --specific_test <test_name>
```

UI load harness (real `vmn-exp ui` + live job processes; see packages/vmn-exp/tests/uiload/README.md):
```sh
python packages/vmn-exp/tests/uiload/run.py live --profile smoke --duration 0   # watch in a browser
VMN_UILOAD_PROFILE=smoke python -m pytest -s packages/vmn-exp/tests/test_uiload_profile.py
```

CI is the local Muster pipeline in `ci/pipeline.py` (`./ci/start.sh`, see
`ci/README.md`) — there is no GitHub Actions workflow.

Skip a test:
```sh
./tests/run_pytest.sh --skip_test <test_name>
```

Tests run in parallel (29 workers by default) using pytest-xdist. Timing benchmarks carry
`@pytest.mark.serial`: `run_pytest.sh` (and the CI `tests_exp` stage) run them alone in a
second, one-process pass, since under 29 busy workers they measure contention.

### Key Concepts

- **App name**: Identifier for a versioned app (e.g., `my_app` or `root_app/service1`). Cannot contain `-` or start with `/`
- **Root app**: Parent container for microservices, format `root_app/service_name`. Root version is an auto-incrementing integer.
- **Version format**: `major.minor.patch[.hotfix][-prerelease.rcn][+buildmetadata]`
- **Tag format**: `{app_name}_{version}` where `/` in app names becomes `-`

### Data Flow

1. Version info stored in git annotated tag messages as YAML (`vmn_info` + `stamping` sections with changesets)
2. Local state tracked in `.vmn/{app_name}/last_known_app_version.yml`
3. `stamp` command: increments version → writes to backends → commits → tags → pushes

### Configuration

Per-app config in `.vmn/{app_name}/conf.yml`. Key fields:
- `template`: Version display format (e.g., `[{major}][.{minor}]`)
- `conventional_commits`: Auto-detect release mode from commit messages (`fix:` → patch, `feat:` → minor, `BREAKING CHANGE` → major). When enabled, `-r` flag is optional.
- `release_mode_policy`: `optional` (--orm behavior) or `strict` (-r behavior) — controls how detected release mode is applied
- `default_release_mode`: Fallback release mode (`patch`/`minor`/`major`/`hotfix`) when none is resolved from CLI or conventional commits
- `changelog.path`: Generate CHANGELOG.md on stamp (requires conventional_commits)
- `github_release.draft`: Create GitHub Release on stamp (requires `gh` CLI + `GITHUB_TOKEN`)
- `deps`: External repository dependencies for multi-repo tracking
- `version_backends`: Auto-embed version into package.json, Cargo.toml, pyproject.toml, or any file via regex/Jinja2
- `policies.whitelist_release_branches`: Restrict which branches can stamp/release
- Branch-specific overrides: canonical layout `.vmn/{app}/branch_conf/{branch}/conf.yml` (branch slashes become real directories; root apps use `root_conf.yml`). Legacy flat `<branch>_conf.yml` and nested `{branch}/conf.yml` files are still read (precedence canonical > flat > legacy) and are auto-migrated to the canonical layout on the next `vmn stamp`.

### Test Infrastructure

- `tests/vmn_fixtures.py`: Pytest fixtures including `FSAppLayoutFixture` for creating isolated git repos, registered by both suites' `conftest.py`
- Tests create temporary git repos with remotes to simulate real workflows

## CLI Commands

- `vmn stamp -r <mode> <name>`: Stamp a new version (mode: major/minor/patch/hotfix). Auto-inits repo/app. Idempotent.
  - `--pr <id>`: Create prerelease (e.g., `--pr rc` → `0.0.1-rc.1`)
  - `--orm`: Optional release mode — only advances if no prerelease exists at target
  - `--pull`: Pull remote first, retry on conflict
  - `--dry-run`: Preview without committing
  - Without `-r`: works during prerelease sequence, or always with `conventional_commits` enabled
  - `--git-push-user` / `--git-push-token`: push credentials for hosts where the checkout has none (fall back to `VMN_GIT_PUSH_USER` / `VMN_GIT_PUSH_TOKEN`). Both required together; injected into an ephemeral HTTPS push URL only, git remote config is left untouched.
- `vmn release <name>`: Promote prerelease to final. `-v <version>` for explicit, `--stamp` for full stamp flow. Also takes the `--git-push-*` flags.
- `vmn show <name>`: Display version info. `--verbose` for full YAML, `--raw`, `--base` (`major.minor.patch[.hotfix]` only), `--type`, `-u` for unique ID, `--dev`, `--conf`, `--from-file`, `--ignore-dirty`, `-t <template>`.
- `vmn goto -v <version> <name>`: Checkout repo + all deps to exact state at version. Without `-v`, returns to a branch tip via `ensure_on_branch()` (`backends/branch_checkout.py`); a detached HEAD's branch comes from `backends/branch_resolver.py` (CI env hint > branch last checked out > unique tip > HEAD version's `stamped_on_remote_branch` > unique container > alphabetical + warning). Building a backend never creates/switches branches (branch state is resolved lazily, `backends/branch_state.py`); commands that commit declare `commits` in their `CommandSpec` (`cli/constants.py`, `cli/plugin_api.py`; built-ins `stamp`/`init`/`init-app`/`release --stamp`) and the CLI then calls the stamper's `vcs.ensure_on_branch()` (reloads conf when the branch changes), as do goto-back and the vmn-exp cold start / `vmn-exp` auto-init. Real branch names only; a legacy `vmn_tracking_remote__*` branch is created only when the local branch diverged. `active_branch` is the branch identity (a legacy branch counts as its upstream), `local_branch` the git ref (push refspec, checkout, outgoing range). `release`/`add` push only their tag. `tests/test_git_process_budget.py` caps git processes for detached stamp/goto. `--deps-only`, `--root`, `--pull` (fetch first when the version is not local), `--force` (dev versions: restore despite untracked files over the snapshot caps). A dev-version goto and `vmn-exp restore` restore through `version_stamp.snapshot.restore.restore_record` (via `version_stamp.api`), like `vmn snapshot restore`: dirty work is saved as a thin snapshot record first (hint `vmn goto -v <saved> <app>`).
- `vmn gen -t <template> -o <output> <name>`: Generate file from Jinja2 template.
- `vmn add -v <version> --bm <metadata> <name>`: Attach build metadata. `--vmp` for a metadata YAML path, `--vmu` for an associated URL.
- `vmn snapshot [create|list|show|note|delete|restore|export|diff] <name>` (built into vmn, `cli/snapshot_cmd.py` + `version_stamp/snapshot/`; `create` is the default; app must be stamped): save/restore uncommitted work (tracked diff, local commits, untracked files, deps) as a deterministic dev verstr. Refs: `-v` verstr/unique prefix/`@N`/`latest`, or `--latest`; `show`/`export`/`diff` default to the latest, `note`/`delete`/`restore` require a ref. Flags: `--note`, `--meta k=v` (repeatable) / `--meta-file` (create), `--filter k=v`/`--last N`/`--verbose` (list), `--json` (list/show), `--to <snapshot ref|stamped version>` (diff; default current working state) and `--tool` (default git `diff.tool`), `-o` (export: dir or `.tar.gz`/`.tgz`, default `<verstr>.tar.gz`), `--force` (restore). Records are thin (`.vmn/store/snapshots/<app-key>/<verstr>/metadata.yml` with `code:`), referencing code objects shared with experiment runs (`.vmn/store/code/<app-key>/`); re-creating a saved state keeps its verstr/timestamp; `delete` drops the code object only when no snapshot or run uses it. `restore` auto-saves dirty work first (note `auto-saved before restore`, prints the restore command), refuses without `--force` when untracked files over the size caps would be deleted, then resets detached to the base commit and applies patches (missing deps are skipped with a warning). `list`/`show`/`diff`/`export` are lock-free (`constants.READ_ONLY_ACTIONS`). With vmn-exp installed: `--store <uri>` > `VMN_EXPERIMENT_STORE` > conf `experiment.storage.uri` put records in that store's `snapshots/` area and code objects in its `code/` area (`vmn_exp/snapshot/stores.py`, registered via `plugin_api.register_snapshot_store_opener`); `--local` overrides; `VMN_EXP_OFFLINE` keeps them local; without vmn-exp `--store` errors. `vmn goto -v <snapshot verstr>` works (vmn-exp's dev-version loader, `cli/code_record.py`). `vmn-exp snapshot` exits 2 pointing at `vmn snapshot`; the stamp dirty-tree hint points at `vmn snapshot create`. See docs/snapshots.md.
- `vmn-exp [action] <name>` (the `vmn-exp` command; a leading `exp`/`experiment` is accepted): Local-first experiment tracking (a snapshot + an append-only metrics/notes log). Actions: `create` (default), `run`, `add`, `list`, `show`, `compare`, `diff`, `restore`, `export`, `prune`, `tag`, `archive`, `unarchive`, `import-mlflow`, `watch`, `importance`, `lineage`, `rewind`, `rerun`, `push`, `migrate`, `compact`, `export-metrics`, `similar`, `similar-backfill` (sweeps are the separate `vmn-exp sweep` command). `create`/`run --name` names a run; `list --query <expr>` filters, `list`/`show --json` print machine-readable output; archived runs are hidden unless `list --archived`. `vmn-exp run <name> -- <cmd>` runs a command and ingests `key=value` lines the child writes to `$VMN_METRICS_FILE`. See docs/vmn-exp/experiments.md.
  - `run` publishes `run_state.yml` (state/pid/host/heartbeat/exit_code, `runner` `exp run`|`sdk`, `cwd` repo-relative or null) next to `metadata.yml` and refreshes a heartbeat while the child lives; `--heartbeat-interval <sec>` (default 30).
  - `run` tees the child's stdout/stderr (pipes, not a pty; `PYTHONUNBUFFERED=1` unless set) into a size-capped `outputs/output.log` (head + tail kept past the cap; `--output-cap-mb` > `VMN_EXP_OUTPUT_CAP_MB` > 10), uploaded after the log sync, throttled to ~64 KB/s of upload bandwidth, and once unthrottled at the end; `--no-capture-output` opts out. SDK: `start_run(capture_output=True)` (off by default) tees fds 1/2 the same way.
  - System metrics (`sys_*`: CPU/RAM, GPU with pynvml) are sampled on the heartbeat by default. Opt out: `--no-system-metrics` / `start_run(system_metrics=False)` > `VMN_SYSTEM_METRICS=0` > conf `experiment.system_metrics: false`. psutil is a hard dependency of vmn-exp-sdk. When `vmn-exp run` samples the child's tree it sets `VMN_EXP_SUPERVISOR_SAMPLES=1`, and `start_run()` inside the child then defaults `system_metrics` off (explicit `True` wins).
  - Alerts: `run.alert(title, text, level)` (rate-limited per title), plus `failed`/`stuck` transition alerts to sinks in conf `experiment.alerts` (`on: [failed, stuck, alert]`, `sinks: webhook|slack|command`) or `VMN_EXP_ALERT_WEBHOOK_URL`/`_SLACK_URL`/`_COMMAND`/`_ON`. `failed` fires from the run itself; `vmn-exp watch <app> [--interval] [--within]` (cron-friendly) detects `stuck` and missed `failed`, deduped via the run's `alerts_sent.yml`. Sink errors never break a run.
  - `vmn-exp run` supervision is crash-proof: the metrics tailer reads bytes (non-ASCII lines never kill it), ingestion/heartbeat/sync errors are best-effort, SIGTERM/SIGINT/SIGHUP are forwarded to the child (SIGKILL after `--kill-grace-sec`, default 30 / `VMN_EXP_KILL_GRACE_SEC`) and the final state is always written (`exit_code` = `128+N` when signal N ended the child; `signal`/`received_signal` fields). The child runs in the invocation cwd (or `$VMN_WORKING_DIR`). Remote sync runs off the heartbeat loop.
  - Read-only actions (`list`/`show`/`compare`/`diff`/`export`/`watch`/`importance`/`lineage`/`export-metrics`/`similar`) never take the repo lock. `list` row numbers are the storage index `@N` resolves, whatever `--sort`/`--last`/`--top` do. `show` prints the last 50 log entries (`--full-log` for all); `--patch [--max-bytes N]` (default 200000) adds the run's own text patches, never the untracked tarball (`--json`: `patch` = `{text, truncated}` or `{text: null, reason: no_code|code_missing|untracked_only}`; no patch = a stderr note, exit 0). `prune` never deletes `running` or `stuck` runs, or a run carrying a `--protect-tag`-named tag key (`--force` overrides either), or a run with a kept descendant; `--dry-run` previews, `--local-only` keeps remote copies, `-v <ref>` (repeatable, not combined with `--keep`/`--older-than`) deletes exactly the named run(s) instead of a bulk policy. `--query <expr>` selects candidates via the query language (same rows as `list --query`); is a dry-run preview unless `--yes`/`-y` is given; `--dry-run` beats `--yes`; `--keep`/`--older-than` apply within the query scope; cannot be combined with `-v`.
  - Metric summaries: the fold keeps last, first (earliest by fold key), finite min/max and an exact finite sum/count per repeated metric (`core/metric_summary.py`; index `SCHEMA_VERSION` bumps on any fold change). `row["metrics"][k]` is the summary per policy: run `define_metric(name, summary=min|max|last|first|mean, goal=, step_metric=, hidden=)` (exact name beats glob) > conf `experiment.metrics.<name|glob>.summary|goal|step_metric|hidden` > `last`; `goal` derives the summary; `summary="none"` is rejected. Sorting, queries, prune `--keep`, compare and the UI all rank on it; `row["metric_summary"][k]` holds `{last,min,max,first,mean}` for metrics logged more than once. The *effective schema* (`core/metric_schema.py`: conf + run-declared `goal`/`hidden` for names conf lacks, latest run wins; `IndexSnapshot.declared_schema()`) drives sort direction (`list_runs`, `vmn-exp list`, UI leaderboard; glob goals count) and UI hidden columns/`/metrics-schema`, never summaries. `hidden` is display-only; run detail carries `hidden_metrics`. CLI: `vmn-exp add <app> -v <ref> --define-metric NAME [--goal] [--summary] [--step-metric] [--hidden]`. `step_metric` joins a metric's series on another metric's value at the same step (reader `get_run(x=)`, series endpoints `x=`, UI x-axis picker).
  - Record format: new runs and registry records carry `format_version` (`core/record_format.py`, currently 2); readers skip records from a newer format, and v1 records without it, with a warning.
  - Store layout v2 (docs/plans/14-store-layout.md): every store root (repo-local `<repo>/.vmn/store/` with one `.gitignore` of `*`, `file://`, buckets; URI without a path = root `vmn`) holds `store.yml` (`layout: 2`, created by the first writer) and areas `runs/ snapshots/ code/ sweeps/ registry/ reports/ comments/ journal/` (`storage/areas.py`), scoped by tag-form app keys (`/`→`-`). Records: `log/<stream>[@<seq>].jsonl` (stream = `<writer>~<owner>`, see Streams below), `outputs/` (`output.log`, `media/`, `tables/`), `artifacts/` (user files only). Unknown layouts and v1 stores (records, no `store.yml`) are refused with a migrate hint (`storage/store_marker.py`); writers also refuse while `migrating`. Per-host state lives outside the store (`storage/host_dirs.py`): index cache `$XDG_CACHE_HOME/vmn-exp/index/<store-id>/`, push ledger `$XDG_STATE_HOME/vmn-exp/push/<store-id>/` (macOS `~/Library/Caches|Application Support/vmn-exp`; `VMN_EXP_CACHE_DIR` replaces both bases). `vmn-exp migrate [--store <uri>|--dir <path>] [--dry-run] [--skip-live]` (`cli/migrate*.py`) converts v1 stores record by record (resumable; `migrating: true` meanwhile; running/stuck runs stop it unless `--skip-live`); without a flag it moves the checkout's `.vmn/<app>/experiments/` etc. into `.vmn/store/`.
  - Fork/rewind: `start_run(fork_from="<ref>", fork_step=N)` (or `"<ref>?_step=N"`; CLI `--fork-from/--fork-step`) makes a new run whose log starts with the source's entries up to N (marked `inherited`) and records `forked_from` (not a child). `start_run(run_id=<ref>, rewind_to_step=N)` appends a `rewind` marker; every reader hides earlier entries with step > N. CLI `vmn-exp rewind <app> -v <ref> --step N` appends the same marker without reopening the run (repo lock like `tag`; works with `VMN_SNAPSHOT_METADATA`; ui job action `exp_rewind`); `vmn-exp run` has no resume, so continue via SDK `run_id=` or `--fork-from`. A running run can't be rewound.
  - Rerun (`cli/rerun.py`, `cli/rerun_report.py`, `cli/rerun_workdir.py`, pure `core/rerun.py`): `vmn-exp rerun <app> -v <ref>|--latest [-- <cmd>]` restores the run's code strictly into throwaway detached worktrees beside the repo (`$TMPDIR/vmn-rerun-<app>-*` or `--worktree-dir`; any restore failure = exit 1, nothing created), claims a record under the original's `code_verstr` that copies its code identity (shares the code object, keeps it alive through prune; ) with `rerun_of`, releases the repo lock and supervises like `run` in the recorded cwd (`--cwd` overrides; child gets `VMN_EXPERIMENT_DIR`=original store, `VMN_WORKING_DIR`=workspace cwd, sweep trials `VMN_SWEEP_PARAMS`). Workspace removed afterwards unless `--keep-worktree` (recorded as `workdir`). `--dry-run` prints the plan; `--print [--json]` prints command/cwd/code_verstr/code + `recipe`/`export_recipe` for cluster jobs (vmn does not schedule). Refused: no code, no recorded command, SDK runs without `--`, `--fork-from`, git-free mode. Not read-only. `rerun_of` is a row/query/detail field; `show` prints `Rerun of:`; the UI has links, no Rerun button.
  - Offline + push: `VMN_EXP_OFFLINE` (`core/storage_resolve.py:drop_remote_if_offline`, `core/writer.py`) makes every storage factory (CLI, SDK, `vmn snapshot`) record to the local root only (error without one), naming new runs `<code_verstr>.<writer_id>[.N]`. `vmn-exp push <app> [-v <ref>...] [--store|--bucket/--prefix/--endpoint-url] [--dry-run] [--json] [--adopt]` (`cli/push.py`, `core/push_*.py`) uploads local runs (all, or the refs) to the remote store (flags > `VMN_EXPERIMENT_STORE` > conf; offline never hides it; `file://` and no remote are refused) under the same name, parents first: code object if missing, claim with the run identity as `create_exclusive(claim_token=)` (a crashed claim of ours resumes), other files by sha, logs per stream from the remote offset (exclusive `create_log_segment`), missing/resized metric objects (`core/push_metrics.py`: a `.vmx` via `put_indexed(covers=)`, dropping the remote segments it covers; else the local blocks past the remote ones as the next `create_metric_segment`), missing/resized artifacts, three-way `archived`/`note` merge. Resumable via a per-host ledger at `$XDG_STATE_HOME/vmn-exp/push/<store-id>/<remote_id>/` (`storage/host_dirs.py`). A remote record of another run under the name → rename on both sides to the next `<code_verstr>.rN` (`renamed_from`, children's `parent` follow); running/stuck, locally registered and sweep-outer runs are skipped instead. Prints `<verstr>  <status>[ -> new][ (detail)]` lines then `pushed N, up-to-date M, renamed R, skipped S, failed F`; exit 1 if any failed. Takes the repo lock; works git-free. UI job action `exp_push` (body `{"verstrs": [...]}`).
  - Lineage: rows carry `outputs` (path/digest/size of every artifact and logged image/table, queryable as `outputs."<path>".digest`). The index keeps them off its lean snapshot rows (`IndexSnapshot.outputs_of(verstr)`, like `metric_summary`), so UI list/leaderboard/facets/columns payloads never carry them; row copies (`list_runs`, `get_run`, `list --json`), UI run detail, `LineageIndex(rows, outputs_of)` and queries (`filter_rows(..., extra={"outputs": snap.outputs_of})`) do. `run.use_artifact(ref, path)` records a `vmn://<app>/<verstr>/<path>` input and returns the file; inputs link to producers by that URI (any app) or by digest (same app). `reader.get_lineage`, `vmn-exp lineage <app> -v <ref> [--depth]`, `GET .../experiments/{verstr}/lineage` (all via `registry/lineage.py:run_lineage`). Registry join: an upstream `uri` link to a live version's artifact gains `model`/`version`/`kind` (the `<name>@<N>` input's version, else the first); `vmn-registry://` inputs are never nodes or digest matches, they fill the payload's `datasets` (`found` = version live). `registry.lineage.version_lineage(storage, name, n)` = `{model, version, kind, status, producer, consumers}` (consumers from `registry/<name>/uses`, pruned ones `found: false`), served at `GET .../models/{name}/versions/{n}/lineage`; `core/lineage.py` stays pure (the registry is injected as `registry=`).
  - Similar runs (docs/plans/18-similar-runs.md, docs/plans/24-persistent-similar-index.md; pure `core/similar*.py`, `core/features*.py`, `core/intent.py`; `cli/similar*.py`, `gitmode/features.py`, `ui/routes_similar.py`): every git-mode run gets a `features` log entry next to `create` (changed files, `path::name` symbols, commit subjects from the captured patches; `author` via `core/identity.py:run_author`, email git config > `VMN_EXP_AUTHOR` > first local commit's `From:`; best-effort). Git-free runs copy it from the export's `vmn_features.yml` (older exports: `vmn_experiment.yml`'s log; `vmn-exp export` adds `exported_by`), reruns copy it with the rerunner as author, forks inherit neither it nor the intent. `--intent <text>`/`--intent-file <path|->` (YAML front matter `keywords`/`generator`) on `create`/`run`/`add` (add: latest unless `-v`) and SDK `start_run(intent=, intent_keywords=)`/`run.set_intent(text, keywords=, generator=)` append an `intent` entry (latest wins, 16 KB). Neither is on lean rows: `IndexSnapshot.features_of/intent_of/hypothesis_of` side tables; rows gain `author` and `has_intent` (query fields). `vmn-exp similar <app> [-v <ref>|--latest] [--top 10] [--all-apps] [--query] [--archived] [--context <text>|--context-file <path|->] [--text-only] [--json]` (no ref = the working tree, nothing created) ranks runs with SQLite FTS5: one document per run (columns text/code/files/symbols/inputs/params/entry/objective/branch; structured tokens hashed to `t<12 hex>` words; text `porter unicode61`) in the app's index SQLite file, filed in the index save's transaction (`core/similar_fts.py`, `similar_filing.py`, `similar_doc.py`); a query is one MATCH of the target's rarest tokens (<= 64 per column) ranked by `bm25` with `experiment.similar.weights` as column weights, 500 at a time (`similar_match.py`), reasons only for the items shown (`similar_reasons.py`); FTS5 is required (no fallback scorer; the UI answers 501); a Postgres-backed server keeps a host-local SQLite corpus per workspace app (`ui/similar_corpora.py`); leaves out the target, its run tree and its reruns (rerun identity = `code_verstr` + exact params + inputs, `similar_profile.rerun_key`; forks of/from the target are kept), so same-commit runs with other params are candidates; groups by run tree, else rerun identity (best on the goal metric, `(+N runs)`); conf `experiment.similar.weights`/`hint`. `create`/`run` print up to 3 on stderr (`cli/similar_hint.py`: persisted index file only, never refreshed; a 300 ms budget interrupts its SQLite queries; never for inner runs; `--no-similar`). `vmn-exp similar-backfill <app> [--all-apps] [--dry-run]` appends missing/older features from code-object patches under this host's writer (repo lock, git-free; `backfilled N, partial P, skipped S, failed F`). SDK `reader.similar_runs`, `run.similar()`; UI `GET .../experiments/{verstr}/similar`, `GET .../apps/{app}/similar?text=`, `GET /api/v1/similar?text=`, a run-page card and a leaderboard similar mode.
  - Rich logging: `run.log_table/log_image/log_histogram` (tables as `outputs/tables/<name>/<step>.json`, images as `outputs/media/<name>/<step>.png` with a stdlib PNG encoder fallback, histograms as log entries; image/table files upload on a background worker that finalization waits for). The image/table entry carries the stored bytes' `sha256`/`size` (hashed before upload) and is itself the output record (no separate `artifact` entry), so `run.use_artifact(ref, "outputs/media/<name>/<step>.png")` fetches it; the uploader appends the entry (with its log-time step/timestamp) only once the file is stored — `finish()` drains the queue before the final log flush — and a file that fails to store (or is still queued at the final wait's deadline) is never recorded, so no entry/output ever outlives a failed or killed upload; `log_histogram` also takes a precomputed `{"bins", "counts"}`; run detail exposes `media`/`tables`/`histograms_total` (`histograms` steps inlined only while <= 100 served in all), `GET .../experiments/{verstr}/histograms/{name}` serves one key's steps, `GET .../experiments/{verstr}/table/{path}` pages a table.
  - Parameter importance: `core/importance.py` (pure-Python histogram random forest + Pearson/Spearman) behind `reader.param_importance`, `vmn-exp importance <app> --metric m`, `GET .../experiments-importance?metric=`, and the leaderboard's Importance view.
  - Metric values: numeric only. numpy/torch scalars and numeric strings are coerced to float; bools/strings/vectors are dropped with a warning; NaN/inf are kept and served as JSON `null`. Missing/non-finite values sort last in both directions. Numeric params fold into `metrics` only when finite (bools fold as 1.0/0.0).
  - Metric storage (plan 12, `core/metric_*.py`, `core/series_reader.py`): metrics are not JSONL lines but per-stream columnar files under `metrics/`: an append-only stream `<s>.vms` while live (remote sync as `<s>@<seq>.vms` segments; sealed = compacted + a new stream every 10M points), compacted at the writer's end (SDK finish within the final upload wait, `vmn-exp run` child exit, importers; `core/metric_compact.py`) into one indexed `<s>@000000-<last>.vmx` (sorted chunks + min/max LOD pyramid + footer, range-readable) that supersedes the stream's segments up to `<last>` only (later ones stay visible; widest `.vmx` wins). `vmn-exp compact <app> -v <ref>... | --all-finished` compacts writers that died (running runs refused); `vmn-exp watch --compact` does it for failed/stuck runs; `rewind` rebuilds a compacted `.vmx`. Readers rebuild `metrics` log entries for log views and still read v1 JSONL `metrics` entries.
  - Streams (docs/plans/21-per-root-streams.md; `core/streams.py`, `storage/stream_owner.py`, `storage/local_streams.py`, `storage/ship_state.py`): a log/metric stream is `<writer>~<owner>` (`display_writer` = the writer id, shown in `_writer`, UI writer columns, `export-metrics`; a `~` in an explicit writer id becomes `-`). The owner is per local record copy (`<record>/.stream-owner`, identified by the dir's realpath + `(st_dev, st_ino)`, minted under `.ship/owner.lock`: symlinks/bind mounts/same-fs renames keep it; a copied, restored-in-place, cross-fs-moved or re-fetched copy mints a new one; retired once its metric stream is compacted), per process without a local copy, per log buffer in `buffered.py`; get it via `core/writer.py:own_stream`. Only a stream's owner ever appends to, ships, seals, compacts or pushes it: `.stream-owner` holds one entry per view (matched on `(st_dev, st_ino)`, else on `st_ino` + realpath for a container's other st_dev; a renamed dir's path is refreshed under the owner lock), and every other id in it is *inherited* (a `cp -r`, restore, cross-fs move, another view) — never shipped, pushed nor compacted automatically (no "is the original alive" check); sync and `vmn-exp push` note once on stderr when the remote lacks part of them (`core/adopt.py`). `vmn-exp push <app> [-v <ref>...] --adopt` (`core/adopt.py:adopt_streams`, `CachedSnapshotStorage.adopt_streams`) catches their files up with the remote (refusing the run, nothing changed, push exit 1, when a remote `.vmx` holds fewer points than the copy's files), then makes them this view's retired owners (shipped/pushed from then on) — only when the original checkout is gone. An append under an unowned stream goes to the copy's own. A metric append holds the copy's append lock (`.ship/append.lock`) from choosing its stream to writing it, and a compaction retires its stream under it. Each stream has one appending owner and its remote copy is a replica of the owner's local file: syncs ship every local stream's new bytes as the next segment with exclusive `create_log_segment`/`create_metric_segment` (taken → re-list, resume from the remote size), ship state in `<record>/.ship/` under a per-copy ship lock (2s wait while the SDK SIGTERM finalizer runs). No seed/merge of other roots' lines; S3 `read_range` raises on non-missing errors.
  - Status is derived, never stored: `created`/`running`/`stuck`/`succeeded`/`failed`. `stuck` = claims running with no exit code, and both the heartbeat timestamp (writer clock) and the store's write time of `run_state.yml` (file mtime / S3 `LastModified`, when known) are stale past `max(3 * interval, 60s)` — a fresh store write proves liveness despite writer clock skew; unknown store time falls back to the heartbeat alone.
  - Python SDK: `from vmn_exp.sdk import start_run` — in-process alternative to `vmn-exp run` (`run.log_metric/log_metrics/log_params/log_note/log_input/log_artifact/log_artifacts/log_dict/log_text/log_figure/set_tag(s)`, `run.finish()`; `log_metrics` without `step=` records `run.step` and advances it — one per-run counter with a high-water mark (explicit steps raise it, lower ones are kept, `commit=False` shares the step with the next call, all-dropped calls consume none; seeded from `start_step` on fork/rewind, else past the highest visible metrics step on resume; `sys_*` metrics stay step-less, media keeps per-name steps, the metrics file is unchanged; `NoOpRun.step` counts too); log writes are batched and flushed at most ~1s apart, on heartbeat and on finish/SIGTERM/exit; `start_run(name=, tags=, run_id=<resume>, all_ranks=False, capture_env=None, mode=None)` — ranks > 0 get a no-op run; `mode="disabled"`/`VMN_MODE=disabled` returns a `NoOpRun` (`disabled=True`) before touching git or the store (explicit `mode="enabled"` beats the env) and makes `autolog()` patch nothing; the snapshot is captured outside the repo lock; an SDK cold start commits/tags locally without pushing; read side `vmn_exp.sdk.reader.get_run/list_runs/runs_dataframe/get_metric_history` — the last two need `vmn-exp-sdk[pandas]`). Writes the same files as the CLI and heartbeats from its own daemon thread (which also syncs the log to a remote every `sync_interval_sec`, default 30). `current_run()` returns the run the calling thread records into (its own context-bound run, else the process's only open run); runs are fork-aware (`Run.pid`), and `VMN_EXPERIMENT_ID` never parents a run to a sibling another thread opened. With `VMN_SNAPSHOT_METADATA` set, `start_run()` works without a git checkout (storage from `VMN_EXPERIMENT_DIR`). `vmn_exp/sdk/` ships in vmn-exp-sdk: it depends only on `vmn_exp.core`, `vmn_exp.storage`, `vmn_exp.registry`, `vmn_exp._base` — never on `version_stamp`, `vmn_exp.snapshot`, `vmn_exp.ui` or `vmn_exp.cli`; git-mode creation lives in `vmn_exp.gitmode` (vmn-exp) and is imported lazily. See docs/vmn-exp/sdk.md.
  - Autologging: `from vmn_exp.sdk import autolog, autolog_disable`. `autolog()` wraps framework `fit()` methods to record hyperparameters, final metrics, per-epoch series where the framework exposes one, and (with `log_models=True`; default off) the model in its native format. Frameworks are patched only once imported (`vmn_exp/sdk/import_hooks.py`), so `autolog()` never imports tensorflow/torch itself. Fits record into `current_run()`; framework worker threads without their own run are ignored while a recorded fit is in progress. Search estimators add `sklearn_best_cv_score`/`sklearn_best_<param>`; `sklearn_score` is a training-set score, computed when `training_score="auto"` (<= 10k rows). Lightning under DDP skips model saving. Supported: `sklearn`, `xgboost`, `keras`/`tensorflow`, `lightning`/`pytorch_lightning`, `transformers` (injects a `VmnCallback` via `Trainer.train`; watches `transformers.trainer` lazily). Plain `torch` has no `fit` to wrap — log metrics with `run.log_metric(...)` in your own loop; gradient/parameter histograms via `vmn_exp.integrations.torch_watch.watch(model, log=, freq=1000, bins=64)` (training-mode forward count as step, logs on the training thread, `unwatch`). Add a framework with an `_Adapter` entry in `SUPPORTED_FRAMEWORKS` in `vmn_exp/sdk/autolog.py` (only `discover` is mandatory; the other hooks default to the sklearn behaviour). Records nothing unless a `start_run()` is open (never opens one implicitly — that would stamp a version from inside `fit()`). Keys are `<framework>_<name>` with an underscore so the query language's two-part paths resolve them. Failures never break `fit()`; patching is idempotent and `autolog_disable()` restores the originals.
  - Query language (`vmn_exp/core/query.py`): filters rows for `list_runs(query=...)` and the UI's `?q=`. Comparisons `= == != < <= > >=`, `~`/`contains`/`!~` (case-insensitive substring; on list fields like `command`/`children` any element matches), numbers accept scientific notation (`1e-4`), `in`/`not in`, `and`/`or`/`not`, parens. Fields are bare row keys, `metrics.<name>` (numeric fold) and `params.<name>` (verbatim, so strings/bools work). `intent` (latest intent text) and `keywords` (its list) come from the snapshot's `intent_of` side table, never lean rows; `similar("<text>")` is true for runs whose FTS `text` column holds every word of the text (porter-stemmed AND; `core/query_similar.py`, extras built once by `core/query_extra.py:snapshot_extra`/`store_extra` for every entry point); `under("<v>")` matches a subtree. Two-valued: a comparison against a missing field is false, `= null` tests absence. Bad queries raise `QueryError` with an offset (400 over HTTP). `vmn-exp list --query` applies it on the CLI.
  - Lock scope: `vmn-exp run` holds the repo lock only for the create/auto-init phase and releases it before supervising the child, so long runs don't block other `vmn` commands and nesting works. The SDK scopes it the same way around create/cold-start.
  - Storage (`vmn_exp/storage/`): backends are chosen by store URI — `--store` > `VMN_EXPERIMENT_STORE` > conf `experiment.storage.uri` (`s3://bucket/prefix[?endpoint_url=]`, `gs://` (`[gcs]` extra), `az://` (`[azure]` extra), `file:///dir`); `--bucket/--prefix/--endpoint-url` are shorthand for an `s3://` URI (`--prefix` has no literal default; `store_uri` supplies `vmn`). Restore safety snapshots (and `vmn goto` of them) live in the experiment store's `snapshots/` area. `vmn goto -v <dev-version>` and `vmn-exp restore` share one lookup (`cli/code_record.py`): local experiments, then the remote experiment store, then snapshots; no-code runs are refused. Every backend implements `list_apps()` (one listing of `runs/`). Schemes register in `storage/registry.py`; other packages add them via the `vmn_exp.storage` entry-point group (the contract is in docs/vmn-exp/experiments.md). Run verstrs are claimed atomically (`create_exclusive`: `O_EXCL` mkdir / S3 `If-None-Match` / GCS `if_generation_match=0` / Azure `overwrite=False`), so hosts sharing a bucket or NFS dir never collide; allocation lists names only. Local writes are atomic (temp + rename) and never resurrect a deleted record. Remote log sync uploads only new lines as segments `log/<writer>@<seq>.jsonl`; `run_state.yml`/logs are never cached locally from a remote. App keys use the tag form (`/`→`-`) on every backend; S3 `exists` is a HEAD; artifacts stream via `upload_file` and are listable/downloadable. Snapshot identity keeps the full `diff_hash` and extends the verstr's hash on a collision instead of overwriting. A run's code (patches + untracked tarball) is stored once per code identity as a record `<code_verstr>.<diff_hash>` in the `code/<app-key>/` area (`core/code_store.py`; its `metadata.yml`, written last, marks it complete); the run's metadata references it as `code:` and `storage.load` resolves it (backends implement `load_record`). A run of already-stored code builds no tarball and uploads nothing; a missing/incomplete object makes `restore`/`goto`/`export` refuse (`code_missing`); `prune` deletes an object with the last run of its code.
  - Nesting: an experiment created while `VMN_EXPERIMENT_ID` is set records it as its `parent`, so a sweep wrapped in `vmn-exp run` yields one outer job with inner jobs. `create`/`run --parent <ref>` sets it explicitly. `kind` is `outer`/`inner`/`single`; an outer job's `tree_status` rolls up its subtree (`failed > stuck > running > created > succeeded`); a container outer that never ran (own `created`) is left out of its own rollup, and a sweep outer whose trials number fewer than planned (grid size, else `run_cap`, from `core/sweep/spec.py:planned_trials`; the index puts it on the row as `sweep_planned`, the tree adds `sweep_trials`; none for an uncapped random/bayes sweep) adds `created`, so a finished-but-short sweep reads `created` (`vmn-exp list`: `created (16/20 incomplete)`, UI: `16/20 · incomplete`).
  - Sweeps (docs/vmn-exp/sweeps.md): `vmn-exp sweep create <app> -f sweep.yml` stores a W&B-style spec (`method: grid|random|bayes`, `metric`, `parameters`, `run_cap`, `early_terminate: {type: median}`, `command`/`program` template) on an outer run; `vmn-exp sweep agent <app> <sweep> [--count] [--retry-failed] [-- cmd]` claims trial slots with `create_exclusive` (records `t<N>`/`t<N>.a<K>` in the `sweeps/<app-key>~<sweep>/` area; never reused), runs each as an inner job with `VMN_SWEEP_PARAMS`, and median-stops laggards (recorded as `end_reason: stopped` in `run_state.yml`, which derives as succeeded; `end_reason` is a row/query field). A trial's metric is its own, else its nested SDK runs' (single one, or best by goal). `sweep status [--json]`, `GET .../experiments/{verstr}/sweep`; SDK `from vmn_exp.sdk import sweep_params`. bayes needs optuna.
  - `import-mlflow`: `vmn-exp import-mlflow <app> (--mlruns <dir> | --tracking-uri <uri>) [--experiment <name|id>]... [--skip-artifacts] [--include-deleted] [--dry-run] [--workers 8] [--refresh-features] [--repo <path>]` — import runs from an MLflow FileStore or tracking server. Exactly one source required. Re-import is idempotent (already-present runs are skipped). No repo lock taken; no auto-init. `mlflow.user`/`mlflow.project.entryPoint` go to a `features` entry (`source: mlflow`), `mlflow.note.content`/`mlflow.source.git.branch` to metadata `note`/`branch`; `--refresh-features` gives already-imported runs those (adding `refreshed R` to the summary), `--repo` records the source commit's branch diff as `changed` (`changed_source: commit`). Summary line: `imported N, skipped M (already present), resumed K, failed F`; exit 1 if any failed. `--tracking-uri` requires `pip install mlflow-skinny`.
- `vmn-exp ui`: Serve the web dashboard + REST API (`pip install "vmn-exp[ui]"`). `--host`, `--port` (8265), `--token`, `--data-dir`, `--repo` (repeatable), `--store <uri>` (`s3://bucket/prefix?endpoint_url=` for MinIO), `--read-only`, `--no-browser`, `--no-index`, `--allowed-host` (repeatable). `vmn-exp ui backup <path> [--config|--db]` copies the control plane online (SQLite backup API + integrity check; Postgres prints the `pg_dump` command). Postgres access goes through one pool per DSN with timeouts/keepalives/reconnects (`ui/pg_pool.py`); API tokens are HMAC-SHA256 under a per-control-plane key; expired sessions and `audit.retention_days` (server.yml, default 365) are swept hourly (`ui/control_plane_sweep.py`); see docs/vmn-exp/server.md. Without a token, `/api` only accepts loopback/allowed `Host` headers (DNS-rebinding guard); mutations reject foreign `Origin`s and need `Content-Type: application/json`; URL app names/verstrs are validated; the SPA fallback never serves files outside `static/`. Responses are gzipped and NaN-safe. `vmn-exp ui` keeps each watched app's index fresh from a background refresher, so requests read a lock-free snapshot (list/detail stay ~ms at 100k runs); list/detail/columns answer ETag/304; `experiments-facets`, `experiments-columns` (whole-set chart data) and `POST .../series` (batched overlay series) exist. The vmn-native shell (plan 19: context root, status rollup, CLI-grammar ⌘K, sign-in page) reads `/meta` (versions, `mode`, `auth` hints, `principal`, `role`), `/workspaces` `role`, `workspaces/{ws}/git` (branch/HEAD/dirty via one lock-free `git status`) and `.../apps/{app}/experiments-status` (counts + `rollup`); a 401 carries `auth` hints and `WWW-Authenticate: Bearer`; run detail has `idx`/`name`. The list endpoint pages (`offset`/`limit` <= 1000), sorts server-side (`sort`, `sort=timestamp|started_at|finished_at`, `order=asc|desc`); `group=parent|forked_from|rerun_of|code_verstr|branch|status|tags.<k>|params.<k>` answers groups instead (plan 22 K1, `ui/leaderboard_groups.py` + `_group_spec`/`_group_fields`/`_group_summary`/`_group_cache`, cached per snapshot and moved by deltas): `{groups: [{key, label, target, loose, counts, aggregates: {m: {mean,best,median,min,max,n}}, rows, rows_total, rows_unfiltered}], total_groups, total_rows, running, aggregate}` (`rows_unfiltered` = the group without `q`/`status`, `running` = running rows after the filters, `aggregate` = what `agg=` resolved to), groups ordered by `agg=mean|best|median|min|max` (default conf `experiment.ui.group_aggregate` of the checkout branch's conf (`ui/readers/branch_conf.py`: HEAD read as a file, no git process; a detached HEAD takes the branch vmn resolves it to via `version_stamp.api.checkout_branch_of`, git once per (checkout, HEAD sha)), else mean) of the sort metric (`sort=c:total|c:waiting|c:running|c:done|c:failed` orders groups by that count instead, desc unless `order=asc`, `failed` = failed + stuck, ties and rows in the default order; plan 22 F10), the no-value group last (key `∅`, a value starting with `∅` keyed doubled; label `(none)`; groups[].`loose` marks a link grouping's (`parent`/`forked_from`/`rerun_of`) no-link group, which holds only the runs heading no group (absent when empty; `ui/leaderboard_group_loose.py`; plan 22 F12)), `limit` <= 50 groups and `rows_per_group` 1..100 (<= 5000 rows a page; clamped, never refused), `group_key=` pages one group; web client `webui/src/apiGroups.ts`; run detail is bounded (`log_tail`/`log_total`, per-metric `series` downsampled to `max_points`, `include_log=1` for the full log) and `/experiments/{verstr}/log?offset&limit` pages the log. Run code (plan 20 K2-K4, `ui/routes_run_code.py`, `ui/readers/{patch_files,baselines,failure_context}.py`): `GET .../experiments/{verstr}/patch?max_bytes=` = `show --patch --json`'s `patch` + `files: [{path, added, removed}]` (ETag = code identity); `GET .../experiments/{verstr}/baselines` = `{base?, parent?, forked_from?, rerun_of?, previous?, best?}` each `{verstr, idx, name, status}` from the index snapshot (`previous` = latest earlier same branch + command, `best` = best other run on the primary declared goal); a failed/stuck run's detail adds `output_tail` (`{lines, truncated}`, last 40 lines of `outputs/output.log`, one <= 16 KB range read), `last_step` (`{step, of}`, `of` = numeric `params.epochs`/`max_steps`) and `last_good` (latest earlier success of the same command) — absent for other statuses; web client `webui/src/apiRunCode.ts`. Leaderboard default columns (plan 20 D11, `webui/src/pages/defaultColumns.ts`): `#`/status/experiment, the schema-declared goal metrics (pinned unless `pin=`; `pin=-` = none; all metrics when no goal is declared), up to 6 params that vary across loaded rows (fit to the width), `when` (plan 22 G1: every run is a row, outer and inner alike; no fold, no per-run inner count columns; an outer's experiment cell keeps `tree <status>` and an incomplete sweep's `16/20 · incomplete`). Grouped (`group=`, plan 22 W2: `hooks/useBoardRows.ts` > `useGroupedRows.ts`, `util/groupItems.ts`, `pages/LeaderboardGroupRow.tsx`) the board renders K1's groups: a header row per group (link target `@N name` + status/tree pill, the code verstr + base · branch, `field = value`; counts in words `16 runs · 2 running · 1 failed` / `16 runs · all succeeded`, `5 of 16 runs · 5 succeeded` when groups[].`rows_unfiltered` differs from `rows_total` (plan 22 F1/F2; a cut group never says `all`), with an sr-only sentence; the counts belong to the group's head, not to columns (plan 22 F13): before the count line a run bar in the status colours (finished runs fill from the left, waiting = the empty track; none for a group of one run; `pages/groupHeaderText.ts:barSegments`, `GroupBar`), on the phone section label too; the menu's "Sort groups" row (aggregate / runs / running / failed / waiting / succeeded, `util/groupCountSort.ts`) sorts groups by a count (`sort=c:<name>&order=desc`; a `sort=c:*` is dropped with `group=none`); per metric column the `agg` aggregate titled `mean of 16`, `own <v>` for a target's own value, metric headers tagged with the aggregate), up to 10 rows, a `+N more` row (`group_key` paging), `collapse=<key>`, a loose group's rows as plain unindented rows after the groups with no header (a plain `+N more` row; plain cards on the phone; no group colour in Charts), 50 groups per page; the `Group by ▾` menu next to Columns (`components/GroupByMenu.tsx`, `util/groupBy.ts`, `hooks/useBoardGrouping.ts`) offers none, Links (parent run, forked from, rerun of, same code), Fields (branch, status, `tags.<k>`, `params.<k>` with <= 20 distinct values, from facets `tag_keys`/`param_values`), Sort groups and the Aggregate; `group=` absent = `parent` when facets `parented` > 0, else `none` (flat, sorted/paged as before); `agg=` absent = the app default (the request omits it; the header tags and the menu show the payload's top-level `aggregate`, plan 22 F3, `hooks/useBoardAggregate.ts`); grouped, the board polls while the payload's top-level `running` > 0 (F4); on a phone the menu is viewport-fixed under its trigger (`hooks/useAnchorTop.ts`); saved views keep `group=`/`agg=`/`collapse=` with the rest of the URL; a header's checkbox selects its loaded rows (indeterminate when partial; Compare (N)/Archive count rows), and the Charts tab's bar/scatter/parallel colour runs by group (`experiments-columns` key `group:<field>`, `util/chartGroups.ts`, legend `components/ChartGroupLegend.tsx`); with rows selected the Charts tab plots only them (plan 22 F7, `hooks/useChartSelection.ts`: `Showing N selected · Show all`, Show all keeps the selection and lasts until it changes); note/tags/started/ended via `show=`; `show=m:|p:|c:<name>` turns a column on, `hide=` off (a schema-hidden metric keeps `hide=` = shown). The leaderboard filter is one grammar (`components/LeaderboardFilter.tsx` = `QueryBar` tokens + text, `FilterShortcuts`, `SimilarToggle`; pure `util/filterTokens.ts`): status/branch/archived shortcuts and hand-typed `status`/`branch`/`text ~` clauses become the same removable tokens, URL params unchanged (`q`, `status`, `branch`, `search`, `archived`); similar mode keeps every token. Below 640px (`hooks/useMediaQuery.ts` `NARROW_QUERY`) the leaderboard renders cards instead of the grid (plan 20 C8, `pages/LeaderboardCards.tsx`): name + status (outers: tree status), the sorted-by metric (else the primary, else the first shown) large, `@N · branch · when`, a per-card checkbox plus select-all; grouped, each group is a sticky section label above its cards (plan 22 G9, `pages/LeaderboardCardGroup.tsx`: the group checkbox (F8), fold, name, status count (wraps, F2), the group's aggregate on that metric) with a `+N more` button; the toolbar popovers span the viewport; touch screens (`TOUCH_QUERY`) get no ⌘K key hints, and the phone identity chip shows an initial only for a signed-in name. Compare (plan 20 D6, `webui/src/pages/Compare/`) is one page, `compare-runs?tab=curves|table|code&sel=…` (repeated `sel` shared by the tabs, selection chips; the code tab diffs the first run against the second or `?to=`, K6 422 reasons shown); the plan-20 `overlay?runs=` / `compare?v=&to=` pages are gone (plan 22 G8, no redirect). The run page (plan 20 D5, `webui/src/pages/RunLayout.tsx`) reads header (`@N`, status, base) > note/tags > failure card > curves > Changes > Fleet/Sweep > Params | Metrics > Similar | Lineage > outputs, then `components/Disclosure.tsx` sections (Status details, Log, Environment closed; Media, Comments open when present), each choice remembered in localStorage `vmn-exp:disclosures`. With the GIL on and a watched app of >= 10k runs it logs a one-time hint to run it on free-threaded 3.14t (`uvx --python 3.14t --from "vmn-exp[ui]" vmn-exp ui`; ~10x the throughput at 100k runs). See docs/vmn-exp/ui.md.
  - Outer runs at scale (plan 23, docs/plans/23-experiments-at-scale.md; an *outer run* (`kind = "outer"`; *job* in code, URLs and API fields) = a run with children, thousands of runs each; every user-visible string says outer run). The experiments page opens on **Outer runs** (`view=jobs`, switch `Outer runs (N) · Single runs (N) · All runs`, last view per app in localStorage; `webui/src/pages/Experiments/`): one row per top-level outer run with run bar, `done/expected · failed · stuck · rate · ETA` (`finished`/`ended · N never started`/`stalled`), best (`—` without a goal), spread; `job:*` sorts. Server: per-job rollups over each subtree moved by index deltas (`ui/job_rollup*.py`, `job_tally.py`, removable `quantile_sketch.py`, `failure_signature.py`; no goal → the app's most common non-param metrics, `ranked: false`); rows gain `root`/`failure_sig`, query `under("<v>")`; `experiments?top=1|jobs|singles` (+ `match_subtree=1` → `matched`, `jobs_matched`/`runs_matched`), `scope=<v>` on list/columns/facets/importance (`ui/leaderboard_scope.py`), groups carry `clause` (+ job aggregates with `top`), `experiments/{v}/summary` and `/failures` (`ui/routes_job.py`, `job_views.py`; a job's run detail has no `fleet.children`), `search?text=` (⌘K; `ui/search_index.py` also prefilters `~` clauses), `events` SSE (`ui/live_events.py`, `routes_events.py`; web `hooks/useLiveScope.ts`, polling fallback), bulk `actions/exp_archive|exp_unarchive|exp_tag` by `{scope,q,status}` with `dry_run` (`ui/bulk_actions.py`). The outer run's page (`webui/src/pages/Job/`): header + run bar + `outer · N runs` badge, tabs `overview|runs|charts|failures|code|log`; the Runs tab is the scoped board with a facet sidebar and select-all-matching; inner run pages get an outer run context bar (rank, prev/next via `from=`/`at=`, `[`/`]`). Boards: `density=compact|comfortable`, keys `j/k x shift+x Enter / Esc ?`; the selection is app-wide (sessionStorage per ws/app, `hooks/useRunSelection.ts`, URL `sel=` only seeds it) in one tray the shell mounts under every app page (`pages/SelectionTrayHost.tsx`; boards register their actions via `useTrayBoard`; `pages/SelectionTray.tsx`: `N runs selected · from M outer runs ▾`, picks by outer run, a picked outer run = one run); a Runs tab's select-all-matching adds a query group `{scope,q,status,count,members}` (`util/selectionGroups.ts`) beside the picks (archive = one by-query action per group + one by verstrs, Compare = picks + each group's top 10 loaded runs, unticking a member turns the group back into picks). `components/OuterBadge.tsx` (`outer · N runs`, N from the row's `job.total` > fleet counts > `children`, `util/outerRun.ts`) marks an outer run in All runs' name cell, the cards, its page header, Compare chips and ⌘K; Compare takes outer runs, and one that logs no metrics itself gets a note on Curves/Table (`pages/Compare/OuterRunNotes.tsx`: `Compare its best run` = summary `top[0]`, else its first run; `Add its N runs`, capped at 20). Index warm restart serves persisted rows first and revalidates in the background (`core/index.py`, `core/gc_pause.py`, `storage/listing_workers.py`). The per-host index (plan 25) persists lean rows + side tables (`exp_rows`/`exp_sides`, `core/index_lean_store.py`) and folds lazily (`core/index_lazy.py`): a warm start decodes no full record, `show -v`/`reader.get_run`/ref resolution are indexed point reads (`core/index_point*.py`, `SqliteStore.resolve_one`), `list` answers from the persisted rows (`core/index_list.py`); revalidation lists record names only, with persisted listing baselines (`core/index_baseline.py`) and a full stat sweep at most every 300 s (`DEFAULT_FULL_SWEEP_SEC`), so a finished run edited behind vmn's back, on pure S3, or by another host on a cached store shows up within 300 s; every vmn SQLite file goes through `core/sqlite_conn.py` (WAL, `synchronous=NORMAL`, `BEGIN IMMEDIATE`) and rebuilds flip a `<file>.current` pointer (`core/sqlite_files.py`), never deleting a file another process has open.
- `vmn-exp model <action> [model] [flags]`: Model registry — link named, versioned model identifiers to experiment runs. Actions: `register` (attach a run ref + artifact path to a new version; `--kind dataset` with `--uri <uri> [--digest]` registers a reference dataset — local paths hashed, same digest deduped), `alias` (point a mutable alias at a version number), `list` (`--kind model|dataset`; UI API `?kind=`), `show`, `resolve` (print version metadata for a ref), `deprecate`, `delete` (refused while an alias points at it; warns when runs used the version). All actions are git-free; storage resolved from `--dir`/`--store`/`--bucket` or env. Refs: `model`, `model@latest`, `model@N`, `model@alias`. `register --alias <name>` sets an alias on create; `alias --expect <N>` is a CAS guard. Prune refuses registered runs even with `--force`; delete the model version first. SDK: `from vmn_exp.sdk.models import register_model, set_alias, remove_alias, get_model_version, list_models, download_model`; `run.register_model(name, artifact_path=, alias=)`. Datasets share the namespace (header `kind: model|dataset`, absent = model): `register_dataset(name, uri)` (reference; a file's sha256 / a dir's manifest digest, `dedupe=True`) or `register_dataset(name, run=, artifact_path=)` (copied, prune-protected), `get_dataset_version(ref)`. Usage: `use_model(ref)`/`use_dataset(ref)` (also `run.use_*`; `download_model(record=True)` inside a run) log an input named `<name>@<N>` with the producer artifact's `vmn://` URI (reference datasets: `vmn-registry://<name>@<N>`) and append a best-effort `use` entry to the record `registry/<name>/uses` (never the model's log); `get_model_version` never records; `NoOpRun.use_*` only resolve. See docs/vmn-exp/models.md.
- `vmn skill`: Print the AI-agent skill block to stdout. `--install` writes it instead (`--target claude` → `.claude/skills/vmn/SKILL.md`, `cursor` → `.cursorrules`, `agents` → `AGENTS.md`); `--force` overwrites an existing Claude SKILL.md. Cursor/agents targets only rewrite vmn's marker block and preserve surrounding text. Core-only: it ends with a one-line pointer to `vmn-exp skill`.
- `vmn-exp skill [--install] [--target claude|cursor|agents] [--force]`: the experiment/UI-fleet/similar-experiments/model-registry skill block (`vmn_exp/cli/skill.py`, handled in `vmn_exp/cli/main.py` before the vmn-command refusal). Installs through `version_stamp.api.run_skill` (vmn's machinery, parameterized by name/description/body/markers): claude → `.claude/skills/vmn-exp/SKILL.md`; cursor/agents use distinct `vmn-exp skill` markers, so vmn's block in the same file is untouched.
- `vmn config <name>`: TUI config editor. `--vim` for $EDITOR, `--global` for repo-level config. `--branch` edits the current branch's canonical branch conf (seeded from the effective conf).
- `vmn config gen <name>`: Non-interactively create a config file (no TTY needed, for CI/scripting). Default creates `conf.yml`; `--branch` (± `--root`) creates the canonical branch conf seeded from the existing effective conf. Never overwrites an existing file.
- `vmn worktrees create <name>` (alias `vmn wt`): Create an island (git worktrees for main repo + all deps). `--island-name`, `-fv`/`--from-version`, `-fb`/`--from-branch`, `--base-path` (default `../vmn-islands`), `--shallow-deps`, `--carry-changes` (apply the app's and deps' uncommitted tracked edits as a patch and copy their untracked non-ignored files into the island; a checkout not at the source's commit is skipped with a warning; without the flag, `create` only warns). `create` is the default action, so `vmn wt <name>` works.
- `vmn worktrees list`: List active islands.
- `vmn worktrees remove <island>`: Clean up an island (removes its worktrees, private branches, and the `vmn-readonly` remote once no island uses it).
- `vmn worktrees freeze <name>`: In the app checkout, pin every dep that is on a real (non-`island/`) branch to that branch in the current branch's canonical branch conf. Refuses on an `island/` branch, on detached HEAD, or while a dep has commits only on its private branch. Never commits or pushes.
- `vmn worktrees pull [island]`: `git pull --rebase` in every island checkout still on its private branch (skips repos moved to a real branch; dirty repos and conflicts exit 1). Without a name, finds `island.json` above the cwd.
- `vmn --completion [SHELL]`: Print shell completion setup script (bash/zsh/fish/tcsh). Auto-detects shell.
- `vmn --completion-install [SHELL]`: Append completion to shell rc file. Idempotent.
- `vmn --completion-uninstall [SHELL]`: Remove completion from the shell rc file. Idempotent.

### Islands (worktrees)

- Layout mirrors the source: every checkout sits at its path relative to the common parent of the app and its deps, so configured dep paths like `../libs/B` resolve inside the island.
- Source branch = the branch each repo was on (or `--from-branch`); `--from-version` islands have none. Each repo with a source branch gets private branch `island/{name}/{source}` tracking `vmn-readonly/{source}` with `pushRemote=vmn-readonly`. `vmn-readonly` is a per-repo remote with origin's fetch URL, an unusable push URL, and refspecs only for islands' source branches; vmn never selects it as its remote.
- Dep start point comes from the conf pin: hash/tag → that ref (detached); branch X → the local checkout if it is on X, else `vmn-readonly/X` fetched; no pin → the local checkout.
- `stamp`/`release`/`add`/`init-app` are refused on an `island/` branch (checked before the lock). A real branch checked out inside an island stamps normally once pushed. A dep on `island/{name}/X` with no commits beyond its upstream counts as synced with a `branch: X` pin.
- `island.json` in the island root is the machine-readable manifest (paths, private `branch`, `source_branch`, `upstream`, dep hashes, `shallow_deps`).
- vmn never borrows another branch's upstream: a branch whose upstream is missing (or on another remote) needs `git push -u origin <branch>` before `vmn stamp`.
- Islands are for app + deps work next to `goto`, not the way to spawn parallel worktrees — the worktree methodology uses plain `git worktree`.

## Environment Variables

- `VMN_WORKING_DIR`: Override the working directory for vmn
- `VMN_LOCK_FILE_PATH`: Custom lock file path (default `.vmn/vmn.lock`, a per-repo lock preventing concurrent vmn commands). Purely a user-facing override — vmn never injects it into a child process's environment. `exp run` and the SDK hold this lock only for their create/auto-init phase, not for the run's lifetime.
- `GITHUB_TOKEN` / `GH_TOKEN`: Required for GitHub Releases feature
- `VMN_GIT_PUSH_USER` / `VMN_GIT_PUSH_TOKEN`: Fallbacks for `stamp`/`release` `--git-push-user`/`--git-push-token`
- `VMN_UI_TOKEN`: Fallback for `vmn-exp ui --token`
- `VMN_EXP_MIN_STALE_SEC`: Overrides the 60s floor of the stuck window (`max(3 * interval, floor)`), read wherever status is derived (CLI, `vmn-exp ui`). Non-positive/invalid values keep 60.
- `VMN_EXP_KILL_GRACE_SEC`: Fallback for `vmn-exp run --kill-grace-sec` (seconds a forwarded signal waits before SIGKILL)
- `VMN_EXP_CACHE_DIR`: Base dir for vmn-exp's per-host state (index cache, push ledger) instead of the XDG cache/state dirs (containers with a read-only home)
- `VMN_EXP_FINAL_UPLOAD_TIMEOUT_SEC`: Seconds the SDK waits for a run's last remote uploads when it is finalized (`finish()`, SIGTERM, interpreter exit; default 60). Several runs finalized together share this one wait (each drains its media uploads before its final log flush, then all final states are written before the remaining uploads are awaited). Non-positive/invalid values keep 60.
- `VMN_SNAPSHOT_MAX_FILE_MB` / `VMN_SNAPSHOT_MAX_TOTAL_MB`: caps on untracked files captured into a snapshot/experiment tarball (defaults 50 / 200; skipped paths are recorded as `untracked_skipped`)
- `VMN_EXP_AUTHOR`: The email a new run's `features` author gets when git config has no `user.email` (git mode: before the first local commit's `From:`; git-free: after the export's `exported_by`).
- `VMN_CAPTURE_ENV`: Set to `0`/`false`/`no`/`off` to disable automatic environment capture (Python version, platform, installed packages) on `vmn-exp create`, `vmn-exp run`, and `start_run()`. CLI `--no-env` and `start_run(capture_env=False)` also opt out; `start_run(capture_env=True)` overrides this variable and conf.yml but cannot override the CLI flag. Opt-out precedence: CLI flag > `VMN_CAPTURE_ENV` > conf `experiment.capture_env`.
- `VMN_EXPERIMENT_DIR` / `VMN_SNAPSHOT_METADATA`: git-free experiment mode (container images built from `vmn-exp export`) for both the CLI and `start_run()`
- `VMN_MODE`: `disabled` makes `start_run()` return a no-op run (no git/store access), `autolog()` patch nothing, and `vmn-exp run <app> -- cmd` exec `cmd` directly (no lock/snapshot/run_state; `VMN_METRICS_FILE=/dev/null`). Explicit `start_run(mode="enabled")` wins; other CLI actions ignore it.
- `VMN_EXPERIMENT_STORE`: fallback for `--store` (a store URI; beats the bucket shorthand)
- `VMN_EXP_OFFLINE`: `1`/`true`/`yes`/`on` drops the remote store everywhere (CLI, SDK, `vmn snapshot`; beats flags, env and conf) so runs record to the local root only, named `<code_verstr>.<writer_id>[.N]`; upload later with `vmn-exp push` (which ignores it)
- `VMN_SYSTEM_METRICS`: `0`/`false`/`no`/`off` disables default `sys_*` sampling
- `VMN_EXP_OUTPUT_CAP_MB`: fallback for `vmn-exp run --output-cap-mb`
- `VMN_EXP_ALERT_WEBHOOK_URL` / `VMN_EXP_ALERT_SLACK_URL` / `VMN_EXP_ALERT_COMMAND` / `VMN_EXP_ALERT_ON`: add alert sinks / set triggers
- `VMN_EXPERIMENT_BUCKET` / `VMN_EXPERIMENT_PREFIX` / `VMN_EXPERIMENT_ENDPOINT_URL`: fallbacks for the experiment `--bucket`/`--prefix`/`--endpoint-url` (flags > env > conf.yml), honoured by the CLI and `start_run()`; with a bucket and no local dir, runs record straight to S3
- Set *by* vmn for the `vmn-exp run` child process: `VMN_EXPERIMENT_ID`, `VMN_APP_NAME`, `VMN_METRICS_FILE`, `VMN_EXP_SUPERVISOR_SAMPLES` (when it samples system metrics); sweep trials also get `VMN_SWEEP_PARAMS`/`VMN_SWEEP_ID`/`VMN_SWEEP_TRIAL`. `VMN_EXPERIMENT_ID` also drives auto-parenting — any experiment created while it is set becomes an inner job of that run.

## Docs Layout

Core vmn and vmn-exp are documented separately (vmn-exp will become a separate product): core docs never carry experiment material beyond one link to `docs/vmn-exp/README.md`.

- `README.md`: core vmn only. Skimmable top level with reference material (commands, flags, conf.yml keys, backends, env vars) inside `<details>` blocks; deep guides live in `docs/` and are linked, not inlined. Keep it that way — don't paste long reference back into it.
- `docs/agent-skill.md`: **generated** from `vmn skill`. Regenerate it (don't hand-edit) whenever `packages/vmn/src/version_stamp/cli/skill.py` changes.
- `docs/vmn-exp/agent-skill.md`: **generated** from `vmn-exp skill`. Regenerate it (don't hand-edit) whenever `packages/vmn-exp/src/vmn_exp/cli/skill.py` changes.
- `docs/snapshots.md`: the `vmn snapshot` guide (core vmn; vmn-exp stores in one closing section). `docs/packaging.md`: the three-package split, import rules, and how releases version each package. `docs/agent-methodology.md`: optional dev rules for AI agents.
- `docs/vmn-vs-*.md`, `docs/migrating-from-*.md`: comparison/migration guides for core vmn; they link to README anchors instead of repeating setup.
- `docs/vmn-exp/`: all vmn-exp docs, indexed by `docs/vmn-exp/README.md`. One canonical home per topic; other docs summarize and link:
  - `client-guide.md`: task walkthrough (store choice, submit/sweep, Kubernetes git-free recipe, in-job SDK, watch, compare, restore/goto/rerun/export, resume/rewind/fork, models, prune).
  - `experiments.md`: `vmn-exp` CLI reference, status/alerts, nesting, storage + plugin contract, offline + `push`, env vars.
  - `sdk.md`: the `vmn_exp.sdk` Python SDK (`start_run`, logging, autolog, readers, query language, integrations).
  - `ui.md`: `vmn-exp ui` deployment + HTTP API. `models.md`: model/dataset registry. `sweeps.md`: sweeps.
  - `server.md`: `vmn-exp ui` as a central server (`server.yml`, SQLite cache, change journal, resync/rebuild, OIDC/API tokens/roles). `byo-bucket.md`: store areas, the server's permission sets, journal lifecycle rule. `reports.md`: reports, panels, publish/export, comments.
  - `ai-fleet-tracking.md`: for AI agents — which SDK/CLI call moves each UI fleet count (the parent group header). `vmn-vs-mlflow.md`, `migrating-from-mlflow.md`.