inference_optimizer · diff

git:20261006.26557fd to git:20261007.f2b32cd

1 added, 14 removed. Audit B to B.

---

name: inference_optimizer

description: |

Launches and monitors Hyperloom's multi-agent inference optimizer for LLM

serving on AMD GPUs. Use when the user asks to optimize an inference model,

run Magpie benchmarks/profiles, resume an inference_optimizer session, tune

SGLang/vLLM serving parameters, run TraceLens/kernel-agent, or validate

end-to-end throughput gains in a new inference environment.

globs:

- "**/inference*optim*"

- "**/inference_optimizer*"

# The Coordinator/orchestrator lives under src/hyperloom/orchestrator/;

# keep this skill triggering on it since it still owns the launcher's runtime story.

- "**/hyperloom/orchestrator/**"

---

# Inference Optimizer Skill

You are the launcher and monitor. The optimizer itself is the Python

`inference_optimizer` runtime under this repository. Do not manually optimize

inside chat unless debugging; launch the CLI, poll persisted state, and report

objective progress.

## What This Skill Runs

The CLI starts a Python Coordinator that coordinates:

- Orchestration: decides next actions (`baseline`, `explore`, `specialist`, `integrate_patch`, `sweep`, Kernel requests). In CLOSE, the Coordinator auto-enqueues `report` at session end; Orchestration may also propose it.

- Kernel (programmatic, not LLM): the Coordinator dispatches `trace_analyze`, `integrate`, and related request kinds directly to Python handlers without an LLM turn. The `run_gemm_tuning` and `run_fusion` lanes are Coordinator-owned: they run inside the `kernel_agent` task at KERNEL entry and PolicyGate rejects an agent request for either.

- Critic: proposal review (default `--critic-agent`; see

[Critic Backend Selection](#critic-backend-selection) for modes).

State lives under a **session directory** (per optimization run).

The **workspace root** is ``$USER_DATA_PATH`` (default

``/workspace/hyperloom``) — it holds shared ``runtime/`` and ``logs/``.

### Layout (N17 default: ``per_model_ts``)

```text

$USER_DATA_PATH/ # workspace_root — set by operator / Claw / SaFE

├── runtime/ # workspace-shared (install.sh, Magpie, kernel-agent.env.sh)

│ ├── kernel-agent.env.sh

│ ├── Magpie/

│ └── source-mirrors/{InferenceX,TraceLens[,TraceLens-internal]}/

│ # Open-source deps are installed by install.sh.

├── logs/ # workspace-shared launcher stdout

└── <model_basename>/ # e.g. DeepSeek-R1-0528, deepseek-ai-DeepSeek-V3

└── <UTC_YYYYMMDDTHHMMSSZ>-<rand8>/ # session_dir — manifest.json, state.json, runs/, …

├── manifest.json

├── state.json

├── storage/coordinator.db

├── agents/{orchestration,kernel,critic}/

├── runs/{baseline,profile,roofline,explore,sweep,...}/<task_id>/

├── kernel-agent/runs/<session_id>/

├── kernel-agent-workspace/<kernel_id>/

├── optimizer_runs/ # per-session launcher logs / PID

├── reports/

└── …

```

**Claw / SaFE pods:** the launcher often sets ``$USER_DATA_PATH`` to a

run-scoped path *before* the optimizer starts, e.g.

``/hyperloom/users/<uid>/deepseek-ai-DeepSeek-V3-20260522_034024/``.

That outer directory is **platform isolation** (one Claw job). The

optimizer then creates ``<model_basename>/<UTC_ts>-<rand8>/`` inside it. Full

session path example::

/hyperloom/users/<uid>/deepseek-ai-DeepSeek-V3-20260522_034024/ ← USER_DATA_PATH (Claw)

deepseek-ai-DeepSeek-V3/20260522T035359Z-9f3c1a04/ ← session_dir (optimizer)

### Path resolution (do not guess)

`session/paths.py` is the single authority for Hyperloom paths. The launching

agent does not need to recreate that logic in shell; it only needs to run

`install.sh` and read the session dir printed by the CLI. CLI preflight loads

the generated `runtime/kernel-agent.env.sh` in-process.

| Concept | Env / helper | Meaning |

|---|---|---|

| Workspace root | ``$USER_DATA_PATH`` → ``session.paths.workspace_root()`` | Shared ``runtime/`` + ``logs/`` and parent of all sessions |

| Session dir | ``$INFERENCE_OPTIMIZER_CURRENT_SESSION_DIR`` → ``session.paths.session_dir()`` | Per-run directory containing ``manifest.json`` / ``state.json`` / ``storage/coordinator.db`` |

**Launcher rule:** do not hand-build, create, delete, or repair paths

under ``$USER_DATA_PATH/runtime/`` (especially ``source-mirrors/``).

Those are workspace-shared assets owned by `install.sh`, including

Magpie, InferenceX, GEAK, TraceLens mirrors, env files, and config.

Manual edits there can corrupt another run's checkout. If install state looks wrong,

rerun `install.sh` or follow the Recovery section; do not clone or clean

the mirrors by hand.

**Session rule:** never treat ``$USER_DATA_PATH`` as the session dir when

``$INFERENCE_OPTIMIZER_CURRENT_SESSION_DIR`` is set. Read

``manifest.json`` / ``state.json`` / ``coordinator.db`` from the

**session dir**. For monitoring after launch, learn the session dir from

the **launch-info JSON** written by ``--launch-info-file`` (``jq -r

.session_dir <file>``, or the ``read_json`` python3 one-liner used in the

health check below — ``jq`` is not installed on every node) or,

equivalently, from the single

``HYPERLOOM_LAUNCH key=value …`` sentinel line the CLI prints to stdout

(``session_dir=…``). Those are the authoritative, machine-readable

sources. Never guess by walking ``$USER_DATA_PATH/<model_basename>/`` for

the latest ``*T*Z/`` timestamp dir — overlapping sessions on the same

host make "latest" pick the wrong run.

Inputs that stay outside `$USER_DATA_PATH` by design (read-only sources

or warm-start caches): **TraceLens** — `$TRACELENS_ROOT` (default

`${HYPERLOOM_CACHE_DIR:-$REPO_ROOT/.cache}/TraceLens`; when unset,

`src/hyperloom/agents/kernel/scripts/install.sh` clones

[AMD-AGI/TraceLens](https://github.com/AMD-AGI/TraceLens) there and pins

it to a fixed SHA. A pre-existing checkout you maintain is only used as

an explicit operator override — export `TRACELENS_ROOT=<path>` to opt

in, which skips both the clone and the SHA pin) with an **optional**

internal

extension at `$TRACELENS_INTERNAL_ROOT` (no default; internal users set

it to their own existing checkout to opt in,

otherwise open-source-only; rehydration module — Hyperloom keeps no internal

URL/path). The per-version

`sglang_roofline_patches/sglang_<minor>_<patch>/` layout under

TraceLens is required by `_server_patcher`),

`/sgl-workspace/{aiter,sglang,vllm}/`,

`~/.cache/amd-ai-devtool/semantic-index/`

(GEAK RAG embedding cache), `/shared/hyperloom/geak-memory/memory.db`

(GEAK cross-session memory). Each is overridable via its own env if

you want a fully self-contained session.

Artefact paths emitted by agents must resolve under the **session dir**;

PolicyGate enforces that. `source_file` and `framework_source_root` are exempt —

they name framework source, which lives outside the session dir by construction,

and where a patch may land is decided when `integrate_patch` applies it.

`hyperloom.inference_optimizer.framework_paths.resolve_framework_tree` names the tree a

session optimises and `resolve_kernel_search_roots` the trees worth searching;

`$INFERENCE_OPTIMIZER_FRAMEWORK_SOURCE_ROOTS` (colon-separated) supplements the

latter and is auto-probed by

`src/hyperloom/inference_optimizer/assets/install.sh`.

Always prefer `manifest.json` / `state.json` / `coordinator.db` under the

**session dir** over guessing from terminal logs.

## Iron Rules

SKILL-level constraints the launcher MUST satisfy before `Coordinator`

is allowed to boot. These IronRULEs are the gate

that runs **before** `python -m hyperloom.inference_optimizer.cli optimize` is even spawned.

### IR-1 — GPU MUST be unoccupied before every launch

Before every `python -m hyperloom.inference_optimizer.cli optimize` invocation (fresh start OR

`--resume-from`), verify that every visible GPU on this pod has **zero

foreign serving PIDs and VRAM usage below 1% of each card's total capacity**. A leftover

`sglang.launch_server` / `vllm.entrypoints` / `Magpie` from a previous

run silently degrades the next `baseline` by 5–30 % (shares VRAM +

schedules on the same XCD); `current_best` cannot detect this

pollution after the fact.

> Inside a running session, the equivalent guard is enforced in

> `orchestrator/kernel/request_handlers.py` via

> `_multi_node_server_lifecycle.py::restart_server_for_round`, which kills

> stale servers before every restart. IR-1 above is the *outer* gate that

> fires before the optimizer process exists.

#### Prior workload cleanup gate (error recovery)

**Trigger (MUST):** whenever a run fails, is abandoned, or you are about to

start a **replacement** workload after any error — credential failures,

optimizer crash, install/preflight failure, user retry/restart, or any recovery

where you would run `docker run`, `install.sh`, or a new/fresh `optimize`.

Applies in bare-metal and docker mode, **whether reusing the existing container

or starting a new one.** Leftover optimizer/serving processes or occupied VRAM

are the usual cause of misleading 0% validated gain (#1314).

**Exception:** `--resume-from "$SESSION_DIR"` against the **same** session

immediately after a clean crash (no credential change, user explicitly wants

resume) may skip — but if the probe finds live or ambiguous leftover workload,

run it anyway and ask the user.

**Probe on the docker host** (bare-metal: current host). **Never** rely on

`docker exec` for process or VRAM checks — container PID namespaces hide

processes in *other* containers; the host namespace is the superset (#1314).

```bash

set -e

export REPO_ROOT="${REPO_ROOT:-$(pwd -P)}"

INSTALL_SH="${REPO_ROOT}/hyperloom/inference_optimizer/assets/install.sh"

if [ ! -f "$INSTALL_SH" ]; then

INSTALL_SH="${REPO_ROOT}/src/hyperloom/inference_optimizer/assets/install.sh"

. "${INSTALL_SH%/*}/runtime_env.sh"

load_dotenv_no_clobber

export USER_DATA_PATH

export RUN_DIR="${USER_DATA_PATH:-/workspace/hyperloom}/optimizer_runs"

# Prior launch handles from canonical artifacts (no last_launch.env — never written)

LATEST_PID_FILE="$(ls -t "$RUN_DIR"/run_*.pid 2>/dev/null | head -1 || true)"

LATEST_LAUNCH_INFO="$(ls -t "$RUN_DIR"/launch_*.json 2>/dev/null | head -1 || true)"

PRIOR_SESSION=""

if [ -n "$LATEST_LAUNCH_INFO" ]; then

PRIOR_SESSION="$(python3 -c 'import json,sys; d=json.load(open(sys.argv[1])); print(d.get("session_dir") or "")' "$LATEST_LAUNCH_INFO" 2>/dev/null || true)"

PRIOR_PID=""

PRIOR_PID_LIVE=false

if [ -n "$LATEST_PID_FILE" ] && [ -f "$LATEST_PID_FILE" ]; then

PRIOR_PID="$(tr -d '[:space:]' < "$LATEST_PID_FILE" 2>/dev/null || true)"

if [ -n "$PRIOR_PID" ] && kill -0 "$PRIOR_PID" 2>/dev/null; then

PRIOR_PID_LIVE=true

echo "prior_pid_file=${LATEST_PID_FILE:-none}"

echo "prior_pid=${PRIOR_PID:-none}"

echo "prior_pid_live=${PRIOR_PID_LIVE}"

echo "prior_launch_info=${LATEST_LAUNCH_INFO:-none}"

echo "prior_session=${PRIOR_SESSION:-none}"

# Foreign processes — host-level pgrep (patterns match preflight_optimizer.py).

# Both framework spellings are needed: Magpie launches `vllm serve`, and vLLM

# then renames its own processes to VLLM::APIServer / VLLM::EngineCore /

# VLLM::Worker_TP<n>, which no `vllm\.entrypoints` scan can see. An orphan that

# is still loading weights also holds no VRAM yet, so the VRAM check below does

# not cover for a missed process match.

pgrep -af 'hyperloom\.inference_optimizer\.cli.*optimize' || true

pgrep -af 'sglang\.launch_server|sglang::|vllm\.entrypoints|vllm serve|VLLM::|Magpie' || true

# VRAM — stdlib-only rocm-smi parse (must run on docker host; no hyperloom import)

python3 - <<'PY'

import json, shutil, subprocess

def unreadable():

print("gpu_vram=unreadable")

if not shutil.which("rocm-smi"):

unreadable()

raise SystemExit(0)

try:

proc = subprocess.run(

["rocm-smi", "--showmeminfo", "vram", "--json"],

capture_output=True,

text=True,

timeout=20,

)

except (OSError, subprocess.SubprocessError):

unreadable()

raise SystemExit(0)

if proc.returncode != 0:

unreadable()

raise SystemExit(0)

try:

data = json.loads(proc.stdout)

except (json.JSONDecodeError, ValueError):

unreadable()

raise SystemExit(0)

if not isinstance(data, dict):

unreadable()

raise SystemExit(0)

rows: list[tuple[float, float]] = []

for fields in data.values():

if not isinstance(fields, dict):

continue

raw: dict[str, float] = {}

for key, val in fields.items():

kl = key.lower()

if "vram" not in kl:

continue

if "used" in kl:

raw["used"] = float(val)

elif "total" in kl:

raw["total"] = float(val)

if not raw:

continue

try:

used_mib = raw["used"] / 1024**2

total_mib = raw["total"] / 1024**2

except (KeyError, TypeError, ValueError):

unreadable()

raise SystemExit(0)

if total_mib <= 0.0:

unreadable()

raise SystemExit(0)

rows.append((used_mib, total_mib))

if not rows:

unreadable()

raise SystemExit(0)

for i, (used_mib, total_mib) in enumerate(rows):

pct = used_mib / total_mib

busy = pct > 0.01

print(

f"gpu{i}_vram_used={used_mib:.1f}/{total_mib:.1f} MiB "

f"({pct:.2%}) {'BUSY' if busy else 'idle'}"

)

# Other hyperloom-named containers — exclude the one we will reuse

CURRENT_CONTAINER="${HYPERLOOM_CONTAINER_NAME:-hyperloom-local}"

docker ps --filter "name=hyperloom" --format '{{.Names}}' 2>/dev/null \

| grep -v "^${CURRENT_CONTAINER}$" || true

```

**If anything is found** (`prior_pid_live=true`, foreign process from pgrep,

any GPU line marked `BUSY` or `gpu_vram=unreadable`, or another

hyperloom-named running container), stop and ask the user explicitly — e.g.

*"The previous run may still be active (session …, PID …, GPU VRAM …). Stop

it before we continue?"* Wait for yes/no.

- **Yes:** stop in this order: serving PIDs from pgrep, then the optimizer

(`kill "$PRIOR_PID"` when `prior_pid_live=true`, else the pgrep match), then

**only** other hyperloom-named containers from the filtered list above

(`docker stop <name>`). Only then continue with `--resume-from "$PRIOR_SESSION"`

or a fresh launch as appropriate.

- **No:** do **not** treat results as clean. Continue only if the user insists;

**MUST** add this caveat verbatim to the final report / session summary:

> **Contaminated baseline warning (#1314):** Prior GPU workload was not

> stopped per user choice. Validated gain and benchmark numbers may be

> unreliable.

Never kill processes or stop containers without explicit user approval.

### IR-2 — install.sh MUST succeed before every launch

Run `bash "$INSTALL_SH"` using the workspace's wheel/source entrypoint resolved

in [Step 1](#step-1--install-one-time-per-pod--venv-rebuild). CLI preflight reads

the regenerated

`${KERNEL_AGENT_ENV:-${USER_DATA_PATH:-/workspace/hyperloom}/runtime/kernel-agent.env.sh}`

in-process; the launch shell must not source it. Skipping install can leave

TraceLens/GEAK unavailable or Ray unprepared. A missing runtime env fails

preflight before optimization. `install.sh --check-only` is a *diagnostic*,

never a substitute.

**Resume carve-out.** `... optimize --resume-from` may skip install only when

ALL hold: (1) `install.sh` exited 0 earlier in the *same shell*; (2)

`kernel-agent.env.sh` remains available under the same workspace root; (3)

`manifest.json` exists under the session dir passed to `--resume-from`.

Any failure → treat as fresh launch and re-run `install.sh`.

> The in-loop equivalent is `_preflight()` steps 1–12 (drift repair, not

> a substitute for this outer gate).

### IR-3 — PR Monitor reachability (in-loop, soft degrade)

The PR Monitor endpoint is co-hosted by KB Store. Hyperloom derives

`${KB_STORE_URL}/pr-monitor/v1/healthz` for this probe and

`${KB_STORE_URL}/pr-monitor/mcp/` for specialist tools; there are no separate

PR endpoint flags or Cortex endpoint variable.

`_preflight()` invokes:

```

bash "$REPO_ROOT/src/hyperloom/inference_optimizer/assets/preflight_kb.sh"

```

Exit codes (soft degrade — IR-3 never aborts launch):

- `0` → PR Monitor reachable (or probe skipped). `pr_monitor_enabled` stays `True`.

- `1` → PR Monitor unreachable. The cli auto-enables `--degraded-pr` and

continues; `manifest.json` records `pr_degraded_reason=ir3_auto`.

Recipe KB enablement is independent: `--degraded-kb` sets `recipe_kb_enabled=False`

(T0/T2/T3/T4 no-ops) without affecting PR Monitor.

Operator opt-out: pass `--degraded-pr` to skip the PR Monitor probe (one

round-trip saved); `manifest.json` then records `reason=explicit_flag`.

Pass `--degraded-kb` and `--degraded-pr` together to short-circuit the entire

IR-3 step.

### IR-4 — OPTIMIZE phase contracts (Coordinator-internal)

These govern the optimizer's OPTIMIZE phase, not the launcher; the full

contract lives in `src/hyperloom/orchestrator/prompts/orchestration.md`. In

brief:

- **IR-4 — OPTIMIZE is specialist-informed**: prefer specialist- or

research-backed variants when available, but `llm_direct`,

`default_grid`, `specialist:<domain-or-tag>`, and `dynamic` provenance

values are all accepted audit labels when phase and sequence gates pass.

Specialist- and dynamic-sourced variants are not grid-size capped;

per-round breadth is bounded by the `research_lane` / GPU pool leases

(the `research_lane` scales with the `2 × visible GPU count` ceiling).

Specialists author patches into an isolated worktree; `integrate_patch`

does the actual `git apply` + throughput/accuracy gate after Critic

review.

GPU specialists are **on by default at whole-machine capacity** (WS2):

`--gpu-specialist-capacity` defaults to the visible GPU count on the launch

host (`_default_gpu_specialist_capacity()`), so Orchestration may dispatch

`delegate{action_name='specialist', params={needs_gpu: true, gpu_count: ...}}`

without any extra flag. Pass `--gpu-specialist-capacity N` to clamp the pool,

and `--gpu-specialist-capacity 0` to disable GPU specialists entirely. The

legacy `INFERENCE_OPTIMIZER_GPU_SPECIALIST_CAPACITY` env is ignored by the CLI

default resolver; use the explicit flag for operator control. When enabled, GPU

specialists serialize against serving through `gpu_research_lane` and

exclusively own their leased cards: they may start/stop their own servers

(any port that is not the production serving port 8888), profile, autotune,

and run real benchmark loops. The one invariant is that they must not touch

the production serving process, its cards, or port 8888.

- **Plateau**: both arms' signals are computed every tick and rendered in the

orchestration prompt. One arm dry is advisory — the phase stays open on the

other lever. **Both arms dry advances the phase** via

`optimize_no_more_leverage`. The LLM may also emit

`escalate_strategy_change{hint='skip_to_kernel'}` when it judges

further effort unproductive. `skip_to_close` is not a phase advance: it abandons

the remaining budget and is reserved for genuine early abandonment.

### FRAMEWORK_AGENT phase — the optimisation phase

One phase, two arms (`--no-framework-agent` skips it entirely).

The **configuration arm** runs server-arg / env grids, sourced by specialist

fan-out. The **source arm** lands upstream PRs and specialist-authored

patches. They are worked in parallel and rotate on their own plateau

judgement, not on a wall-clock split.

The source arm's supply is the `candidate_discovery_specialist`: it surveys

the allowlisted repos, ranks what it finds against the stack and the tried

ledger, and judges each entry — already present, not applicable, or worth a

bench and by which route. The pump takes that batch in the order given; it

does not re-rank or re-audit. When discovery comes back empty its full retry

budget, the local-exploration arm authors against profile evidence instead;

with that off too, the source arm reports itself dry.

Every diff lands through `integrate_patch`, with `patch_source` naming where

it came from. KEEP commits to the live tree so the next candidate stacks on

top; REVERT does `git reset --hard`. Resume skips completed candidates by

idempotency key.

The phase exits when both arms are dry, when its budget is spent, or at the

absolute per-phase cap.

### IR-8 — `--framework atom` is single-node only

`--framework atom` (Magpie `atom_mi*x.sh` against

`atom.entrypoints.openai_server`) reaches full parity with sglang/vllm

EXCEPT multi-node: `_apply_atom_auto_tighten` in `cli.py` rejects

`--nodes >= 2` with `SystemExit(2)` (atom upstream has no multi-node TP

wiring). No other flag is auto-flipped — kernel-agent, framework-agent,

profile / roofline / TraceLens all run on atom. The atom-specific

behaviors (configs, cold-start seed grid, source roots) are summarized

under **Framework Selection** below.

## Retired modules and rules (do not re-introduce)

The live runtime uses `protocol/action_surfaces.ACTION_CATALOGUE`,

`_grid_runner.py`, and the unified specialist-informed `explore` flow. Do not

recreate the retired `backends` / `params` / `validate_stack` / scoring

modules, nor the `actions/_meta/*.yaml` catalogue and its ActionRegistry

loader, nor the `vendor_kernel_config` / `operator_tuning` /

`deep_kernel_analysis` actions (they never had an implementation).

Rules that look reasonable but break the current flow:

- **No "source lever before configuration" rule** in

`prompts/orchestration.md` — the two are arms of one phase,

worked in parallel and ranked by what the bottleneck calls for.

Upstream diffs land through `integrate_patch` with

`patch_source='upstream_pr'`; there is no separate

`framework_agent` action for the LLM to propose or be denied.

Use `--no-framework-agent` to skip the phase entirely.

- **`kernel_opt` sequencing** is no longer gated by an

explore-minimum check (the

`explore_attempts_minimum_before_kernel_opt` rule was retired

in loosen_plan P1_06). KERNEL_AGENT phase may propose `kernel_opt`

directly; the `trace_analyze → run_optimization` data

dependency (P2_11 handler-level check) and the reusable

`kernel_id` validation still keep the inputs valid.

## Setup

Two commands: Step 1 implements **IR-2** (install gate), Step 2 launches.

Both are idempotent; do not replicate them inside chat.

### Credentials

The common single-gateway setup uses `OPENAI_API_KEY` and `OPENAI_BASE_URL`.

Split-gateway deployments may provide provider-specific `ANTHROPIC_*` /

`OPENAI_*` credentials instead. Shell-exported values win; `$REPO_ROOT/.env`

is loaded only to fill missing or empty values. The recipes and `install.sh`

use `assets/runtime_env.sh` without executing dotenv contents. Docker mode

excludes dotenv-provided `PYTHON`, `VIRTUAL_ENV`, and

`INFERENCE_OPTIMIZER_FORCE_PYTHON`; explicit shell selections remain intact.

A readonly `USER_DATA_PATH` is not reassigned; reconcile a conflict with the

setup-selected workspace root before continuing. CLI preflight loads generated

`kernel-agent.env.sh` in-process, not through shell `source`.

### Step 1 — Install (one-time per pod / venv rebuild)

```bash

set -e

export REPO_ROOT="$(pwd -P)" # workspace containing .env and a wheel or source install

INSTALL_SH="${REPO_ROOT}/hyperloom/inference_optimizer/assets/install.sh"

if [ ! -f "$INSTALL_SH" ]; then

INSTALL_SH="${REPO_ROOT}/src/hyperloom/inference_optimizer/assets/install.sh"

. "${INSTALL_SH%/*}/runtime_env.sh"

load_dotenv_no_clobber

export USER_DATA_PATH

bash "$INSTALL_SH"

```

`src/hyperloom/inference_optimizer/assets/install.sh` is the only install entrypoint for

full inference optimization. It installs the optimizer / Magpie / InferenceX

first, then chains to `src/hyperloom/agents/kernel/scripts/install.sh` for the kernel

optimization environment. `src/hyperloom/agents/kernel/scripts/install.sh` remains valid for

standalone kernel-agent debugging, but should not be the main entrypoint for a

full inference optimizer session.

The install phase always initializes the full Hyperloom runtime. Even if the

user later passes `--no-kernel` at runtime, the installer still prepares

kernel-agent / TraceLens / GEAK; `--no-kernel` only means

that this `optimize` run skips the kernel optimization phase.

`install.sh` installs everything in one shot (no `--with-*` flags to

remember). Direct steps in `src/hyperloom/inference_optimizer/assets/install.sh`:

| Component | Provided by |

|---|---|

| `inference_optimizer` pkg + `claude_agent_sdk` extras (`pip install -e .[test]`) | `ensure_inference_optimizer` |

| **Magpie** (`pip install "$MAGPIE_PACKAGE_SPEC"`; default spec pins `magpie-eval` to `$MAGPIE_REF`) | `ensure_magpie` |

| `INFERENCEX_PATH` resolution (honours a pre-existing `$INFERENCEX_PATH`, else clones `$INFERENCEX_REPO` pinned to `$INFERENCEX_REF` into `$INFERENCEX_DEFAULT_DIR` = `${HYPERLOOM_CACHE_DIR:-$REPO_ROOT/.cache}/InferenceX@<sha>`, reusing an existing checkout there on re-runs) | `ensure_inferencex` |

| `INFERENCE_OPTIMIZER_FRAMEWORK_SOURCE_ROOTS` appended to `kernel-agent.env.sh` | `_probe_framework_source_roots` |

Chained from `src/hyperloom/agents/kernel/scripts/install.sh` (single chain at the end

of `src/hyperloom/inference_optimizer/assets/install.sh`):

| Component | Provided by |

|---|---|

| `ray==2.44.1` + `click<8.3.0`, or the lowest published version above it that has a wheel for this interpreter (then without the `click` ceiling, which only guards 2.44.1's CLI); `RAY_VERSION` is honoured verbatim | pip |

| TraceLens public (editable install) | `ensure_tracelens` (`pip install -e` at `$TRACELENS_ROOT`; skills, patches, CLI, analysis orchestrator) |

| TraceLens-internal (editable install, **optional**) | `ensure_tracelens` (`pip install -e` at `$TRACELENS_INTERNAL_ROOT` only when set; mirrors read-only checkout to `${HYPERLOOM_ROOT}/TraceLens-internal`; rehydration module). Unset => open-source-only. |

| GEAKv4 Claude Code workflow checkout + SDK deps | `ensure_geak` |

`${KERNEL_AGENT_ENV:-${USER_DATA_PATH:-/workspace/hyperloom}/runtime/kernel-agent.env.sh}` is

regenerated by `install.sh` and contains gateway URLs, auth aliases,

GEAK runtime variables, and InferenceX path. CLI preflight reads it; do not derive

these by hand or source it in the launch shell. Generated env/config state is written to the pod-local runtime directory,

not back into a shared WekaFS source checkout.

### Experience KB service

Every workspace runs its own local Experience KB service; it ships inside

Hyperloom. `hyperloom-setup` writes the loopback `HYPERLOOM_KB_URL` and a

generated `HYPERLOOM_KB_TOKEN` to the workspace `.env`. Load `.env` before

launching; do not ask the user for these values again.

Each optimize launch starts the service when nothing serves that URL, in the

same environment as the optimizer (inside the container in docker mode). Its

data and `service.log` live under `$USER_DATA_PATH/experience-kb`, and it keeps

running after the run for the next one. When it cannot be started, the launch

logs a warning and continues: reads return nothing and writes wait in

`$USER_DATA_PATH/experience-kb/spool` until the service serves again, so they

survive a removed container. A non-loopback `HYPERLOOM_KB_URL` names a

service this workspace does not run; Hyperloom connects to it without starting

anything. Without `HYPERLOOM_KB_URL`, Hyperloom runs without Experience KB reads

or writes.

No enable flag, declaration path, service identity, Run scope, worker identity,

job identity, or spool path is required.

The service keeps every schema written to it, so a workspace whose declaration

changed keeps its older Experiences; a run's reads search only the schema that

run writes.

A global Experience KB, named by `HYPERLOOM_GLOBAL_KB_URL` and

`HYPERLOOM_GLOBAL_KB_TOKEN` in `.env`, is shared through the local service;

runs never read or write it directly. When the user asks to share or fetch

Experiences, run in the optimizer's environment, with `.env` loaded:

```bash

python -m hyperloom.inference_optimizer.experience_kb_service push # this workspace's Experiences not pushed yet

python -m hyperloom.inference_optimizer.experience_kb_service pull # the global KB's Experiences of this workspace's schemas

```

Report the one summary line each prints (global URL; `created`, `unchanged`,

`skipped`, `rejected`) and never the token. A push resumes where an earlier one

stopped and never sends back what was pulled. Push and pull use the service as

it runs and never restart it under a running session; when they warn that its

settings differ, `experience_kb_service ensure` or the next launch applies them. With `HYPERLOOM_KB_AUTO_PUSH=1`,

every run pushes after its Experiences are written locally; a failed automatic

push is only a warning, and the next push sends what it missed. An unusable

switch value or a missing global KB is a launch warning, and that run does not

push.

During FRAMEWORK_AGENT the service is read at two points and the returned block

is injected into the prompt:

- every orchestration tick, with the untested proposals as context;

- every specialist dispatch, with the specialist's domain, investigation, and

task as context. The block renders as the specialist prompt's

`EXPERIENCE KB` section.

Every written Experience is readable by the next read. Retrieved evidence is

advisory and never replaces the measured benchmark baseline. A read failure

soft-degrades to the original prompt. An AgentX run neither reads nor writes

Experiences: the Experience schema cannot yet tell its workload from a

synthetic one. Neither does a run graded on anything but output throughput

(for example `HYPERLOOM_PERF_METRIC=intvty_v1`), since every Experience records

the throughput objective.

Every complete measured attempt is written idempotently when the session

breakdown is written. Rendered Experience references from an orchestration

grid, proposed or delegated, are carried through to the measured Experience. A

network write failure is spooled for retry.

Each injection appends one entry to `state.json` `experience_kb_injections`

(latest last, capped at 20); an orchestration entry is added only when its

injected Experience set changes:

`{tick, phase, ts, consumer, domain, gap_canonical_id, read_id, experience_ids, experiences, prompt_block}`.

`consumer` is `orchestration` or `specialist`; `domain` and `gap_canonical_id`

identify the specialist dispatch and are empty for orchestration.

`prompt_block` is the injected text; each Experience appears in it under an

`Experience <id>` heading with its complete record. A free-text field over

2 KiB, typically a source patch in `change.content`, appears as

`<external content sha256:...>` and is written whole under

`<session>/experience_kb/contents/`, each patch also as its own file; the block

ends with those paths. Records are injected whole while they fit 40,000

characters; the rest of a read is left out, never cut. The injected agents

cite the Experiences that shaped a proposal in its `experience_citations`,

which reach the measured Experience's `provenance.extra`. `experiences` holds one

summary per injected Experience, in `experience_ids` order: `experience_id`,

`source_run_id`, `change_summary`, `decision`, `baseline_value`,

`outcome_value`, `score`, and `why_matched`. `read_optimizer_state.py` prints

the latest orchestration and specialist entries with one line per injected

Experience. When a poll shows a new entry, report each Experience's summary

together with the matching section of `prompt_block` to the user.

- When Experience KB is enabled, after the workload Skill resolves `MODEL_PATH`

- and `FRAMEWORK`, validate the same production path:

- ```bash

- mkdir -p "$USER_DATA_PATH/optimizer_runs"

- python3 -m hyperloom.inference_optimizer.tools.cold_start_check \

- --model "$MODEL_PATH" \

- --framework "$FRAMEWORK" \

- --require-experience-kb \

- --output "$USER_DATA_PATH/optimizer_runs/experience-kb-cold-start.json"

- ```

- Require `cold_start_ready=true`.

### Tool source fields (prompt → env, sandbox-only)

Prompt fields naming read-only source trees consumed by sandbox-side

`install.sh` / launcher. `export <K>="<v>"` in the launcher shell before

`install.sh`. These are **sandbox-only** — never ask the platform to bake them

into multi-node pod env; those pods have their own paths and do not consume

these.

| Prompt field | Env name | Consumer |

|---|---|---|

| `INFERENCEX_PATH: <path>` | `$INFERENCEX_PATH` | `src/hyperloom/inference_optimizer/assets/install.sh:ensure_inferencex` |

| `TRACELENS_ROOT: <path>` | `$TRACELENS_ROOT` | `src/hyperloom/agents/kernel/scripts/install.sh:ensure_tracelens` (public) |

| `TRACELENS_INTERNAL_ROOT: <path>` (optional) | `$TRACELENS_INTERNAL_ROOT` | `src/hyperloom/agents/kernel/scripts/install.sh:ensure_tracelens` (internal; only when set) |

**Multi-node escape hatch**: if `$TRACELENS_ROOT` / `$TRACELENS_INTERNAL_ROOT` / `$GEAK_ROOT` /

`$WORKSPACE_ROOT/Magpie` / `$INFERENCEX_PATH` may move or differ across nodes,

`rsync -a` them into `$SESSION_DIR/vendor/<name>/` and override the matching

env vars BEFORE running `install.sh`. Single-node WekaFS-mount setups (the

production default) need none of this — `ensure_tracelens`

already handles the read-only-source case.

### Step 1.5 — Write the advisory `model_arch` profile (best-effort)

After the CLI creates the session directory, produce an **advisory**

architecture profile so the orchestration + specialist prompts carry richer

model context than the coarse `--model-class` tag. This is **best-effort

and non-fatal**: a missing / invalid file simply causes Hyperloom to omit

the section — it never blocks launch, never replaces `--model-class`

(still required), and is always **subordinate to live TraceLens evidence**

at runtime (it drives no deterministic gating — atom seed grid, framework

gap token, recipe key, and prompt label all stay on `model_class`).

Steps for the launching agent:

1. **Gallery lookup** — fetch the LLM Architecture Gallery

(`https://sebastianraschka.com/llm-architecture-gallery/`) and locate

the card for the model being launched. Extract the schema fields below.

2. **Fallback classify** — if the model is not in the gallery, do a

lightweight classify from the model's local `config.json` (decoder

type, attention variant, expert counts, MTP, SWA window) and set

`"source": "config_classify"`.

3. **Write the profile BEFORE launch and point `$HYPERLOOM_MODEL_ARCH_FILE`

at it.** The CLI creates `session_dir` and reads the profile in the *same

process*, so a file written to `<session_dir>/model_arch.json` after the

session dir appears always loses the race — the run seeds

`state.model_arch={}` and the profile never reaches any prompt. Write the

JSON to a launcher-owned path instead and export

`HYPERLOOM_MODEL_ARCH_FILE=<that path>` in the shell that spawns

`optimize`; the CLI copies it into `<session_dir>/model_arch.json` for

provenance. Use a **session-unique** filename — `$USER_DATA_PATH` is shared

by concurrent sessions on WekaFS, so a fixed name races other launches.

Include `model_name` (required for the stale-file guard). Set it to the

**clean model name** (e.g. `Qwen2.5-7B-Instruct`);

the guard normalizes launch forms — flat dirs, HF repo ids, and HF hub

cache `models--org--repo/snapshots/<hash>` paths — so do NOT use the

snapshot commit hash. All other fields are optional; renderers drop

empty fields.

```json

{

"model_name": "DeepSeek-R1-0528",

"source": "gallery",

"decoder_type": "Sparse MoE",

"attention": "MLA",

"layer_mix": "61 MLA",

"kv_cache_per_token": "68.6 KiB",

"active_params": "37B active / 671B total",

"num_experts": 256,

"experts_per_tok": 8,

"mtp": true,

"swa_window": null,

"norm": "RMSNorm",

"notes": "DeepSeek V3-style: dense prefix + shared expert + MTP-1 path"

}

```

If you cannot determine the architecture, skip this step — do not write a

placeholder file. Hyperloom degrades silently (WARNING in its own logs)

when the file is absent, invalid, or stale.

### Step 2 — Launch

**Multi-node (`nodes >= 2`):** [`multi_node/SKILL.md`](multi_node/SKILL.md).

```bash

python3 -m hyperloom.inference_optimizer.cli optimize \

--model "$MODEL_PATH" \

--framework vllm \ # sglang (default) / vllm / atom / xdit / custom

--gpu-type MI300X \ # or omit for rocm-smi auto-detect

--model-class moe_mla \ # dense / moe_mla / moe_swa / moe_mla_nsa; categorical key for atom seed grid + framework gap token + recipe key + prompt label

--isl 512 --osl 512 \ # workload shape — pass whatever the prompt states; omitting them uses defaults ISL=1024/OSL=1024

--conc 64 \ # client concurrency — pass the prompt's value; default 64

--tp 1 --ep 1 \ # parallelism — pass the prompt's TP/EP; defaults 1/1

--precision bf16 \ # match the checkpoint (bf16 default); use fp8 for an FP8 checkpoint

--max-hours 2 \

--compare-against-gpu B200 # optional — when set, fetches real InferenceX reference; when unset, target_analysis still runs and writes a 'no_target_gpu_configured' marker JSON

```

**Caller responsibility (post-classify-removal)**: the in-loop `setup` /

`classify` actions were deleted; the SKILL caller is now expected to

supply session metadata directly via CLI flags. **Any workload value the

operator states in the prompt (ISL, OSL, CONC, TP, EP, precision, budget, and

every `--extra-env`) MUST be forwarded as the matching CLI flag** — these flags

are the only source of truth; an omitted flag silently falls back to its default

and the operator's stated value is lost:

| Surface | CLI flag | Notes |

|---|---|---|

| Model path | `--model` | required |

| Framework | `--framework` | `sglang` (default) / `vllm` / `atom` / `xdit` / `custom` — atom is single-node-only; xdit is scriptable diffusion (`img/s`, no serving server); `custom` is an operator-supplied workload and **additionally requires `--framework-path` and `--benchmark-scripts-dir`** (see below) |

| Custom source tree | `--framework-path` | **Required for `--framework custom`.** The workload's own checkout; patches are authored against it. |

| Custom bench scripts | `--benchmark-scripts-dir` | **Required for `--framework custom`.** Holds the entrypoint, looked up as `custom_<gpu-type>.sh`. Every knob it reads must be forwarded as `--extra-env`; the throughput unit is whatever its report declares. |

| GPU type | `--gpu-type` | rocm-smi auto-detect when unset |

| Model class | `--model-class` | categorical key for the deterministic consumers (atom seed grid, framework-agent gap search token, recipe key, prompt label); when unset, Coordinator boot infers and persists it from model metadata or model-path family keywords. For richer advisory model context see Step 1.5 (`model_arch.json`) |

| Input seq length | `--isl` | Pass the prompt's ISL. Default `1024` when omitted. |

| Output seq length | `--osl` | Pass the prompt's OSL. Default `1024` when omitted. |

| Concurrency | `--conc` | Pass the prompt's CONC (max in-flight requests). Default `64`. SWEEP measures a ladder around it (under AgentX only with `--enable-conc-sweep`); `--conc-sweep-concs` overrides the workload's default ladder. |

| Tensor parallel | `--tp` | Pass the prompt's TP. Default `1`. |

| Expert parallel | `--ep` | Pass the prompt's EP for MoE. Default `1`. |

| Precision | `--precision` | Match the checkpoint (`bf16` default / `fp8` / ...). Keep consistent with `--quantize`. |

| Budget | `--max-hours` | Pass the prompt's time budget. Default `2.0`. |

| Latency SLA | `--max-latency-ms` | **Scriptable frameworks only** (`xdit`, `custom`); refused for serving, where AgentX already grades interactivity. Pass any stated ceiling on per-request latency ("must stay under 250 ms", "interactive workload"). A **constraint, not a target**: it composes with `--target-*` rather than competing, and refuses any KEEP whose mean end-to-end latency exceeds it — including one that reported no latency at all. Off when omitted, which does not lose a preference but does remove the SLA from the search. |

+ | GPU power settings | `--gpu-power-cap-w` / `--gpu-perf-level` | Assertions, not requests. Only when the prompt says the cards were set to a cap or perf level; Hyperloom never changes them. The session refuses to start if a card differs. |

| Max model len | `--max-model-len` | Optional; auto-derived from ISL+OSL+headroom when omitted. |

| External reference GPU | `--compare-against-gpu` | `target_analysis` writes `target_analysis/target_baseline.json` for query/status metadata and `competitor_target.json` for both advisory and final-report comparisons. Without a target GPU it writes `reason="no_target_gpu_configured"` and clears the competitor target. AgentX reads accepted `current_best.total_throughput / state.tp` and `current_best.e2e_norm_intvty_p90` at `state.conc`; it does not reread raw results or recipes. Missing targets or axes remain unavailable. This is a cross-system advisory, not proof of identical measurement estimators or deployment, and never changes Objective or KEEP/REVERT. |

| Target advisory | `--no-target-advisory` | Disable external-target hints in prompts without disabling final-report comparison. `primary_gap` uses the existing latency/throughput categories; the interactivity axis is displayed as interactivity. |

| Quantization prelude | `--quantize` | Optional. Natural-language quantization request. Runs the quantization-agent once before the loop and rewrites `--model` to the quantized model. See Step 2b. Never runs on a resume. |

| Env pins | `--extra-env NAME=VALUE` | Repeatable; forward **every** one verbatim as its own flag (do not drop any or fold into the `Environment:` block). The CLI persists them in `state.json` and serializes them into `$INFERENCE_OPTIMIZER_EXTRA_ENV`; a dropped pin is lost silently — e.g. a missing `SGLANG_USE_AITER=0` leaves the explore aiter-MoE filter blind. A `--resume-from` re-exports the persisted set, so re-pass them only to change the set. |

### Step 2b — Optional quantization prelude (`--quantize`)

When the user asks to **quantize the model before optimizing** (e.g. "quantize

to FP8 then optimize", "run this in MX-FP4"), pass `--quantize "<scheme prompt>"`

to the same `optimize` command. This runs the **quantization-agent once as a

prelude**, before any baseline/session work: it drives AMD Quark PTQ from the

prompt, then rewrites `--model` to the exported quantized model so the entire

optimization loop runs on the quantized model.

```bash

python3 -m hyperloom.inference_optimizer.cli optimize \

--model "$MODEL_PATH" \

--framework vllm \

--quantize "fp8 global scheme, fp8 kv_cache, exclude lm_head; accept up to 5% relative eval gap" \

--max-hours 2

```

- The `--quantize` text is the quantization request only (scheme / kv-cache /

excluded layers / acceptable eval gap). **Do not** repeat the model path or

export dir — the adapter folds `--model` + a per-model export dir under the

workspace root (`<workspace_root>/quantization/<model>/quantized`) into the

prompt automatically.

- **Structured path for UI/backends**: instead of free text, pass

`--quantize-scheme <enum>` (one of `none` / `fp8` / `ptpc_fp8` / `mxfp4` /

`mxfp4_fp8`); `mxfp4` / `mxfp4_fp8` are **MI355X-only**. It resolves to a

curated prompt internally (`src/hyperloom/orchestrator/phases/quantization_schemes.py`). `none` or

omit = no quantization. Free-text `--quantize` takes priority when both given.

- **Keep `--precision` consistent with the quantization.** When a quantization

scheme is requested, also set `--precision`/`PRECISION` to that scheme (e.g.

`--quantize-scheme fp8` → `--precision fp8`). Otherwise the

benchmark configs, display names, and the optimization report carry the stale

operator-supplied precision label (e.g. `fp8`/`bf16`) and **mislabel** an

actually-quantized model. Never leave a conflicting precision when quantizing.

- Behavior: one-shot, **never runs on a resume**. On a failed/unusable

quantization the run **hard-stops (`SystemExit(3)`)** — it never silently

optimizes the un-quantized source after an explicit `--quantize`.

The one exception is a **pre-flight scheme/GPU mismatch** via

`--quantize-scheme` (e.g. `mxfp4` on a non-MI355X target): this is **skipped**

(not a hard stop) and continues on the un-quantized model, emitting a

`QUANTIZATION_SKIPPED:` line on stdout and setting

`$HYPERLOOM_QUANTIZATION_SKIPPED` so the caller can detect it.

- Prerequisites (in addition to the normal Setup): `$QUARK_ROOT` must point at

a Quark checkout containing `.claude/skills/quark-torch-*`, and the installed

`amd-quark` package version must match that checkout (install editable from

`$QUARK_ROOT` to keep them consistent). Claude SDK auth is the same

`ANTHROPIC_*` env the rest of the loop uses.

- After it finishes, the `Quantization prelude: model -> <dir>` line on stdout

shows the quantized model path that the rest of the run will use; include it

in status reports.

A user request to optimize a model is approval to run Step 1 on a fresh

node; do not stop for an extra confirmation. After IR-2, smoke-test the

CLI:

```bash

export HYPERLOOM_KERNEL_AGENT_ROOT="$REPO_ROOT/src/hyperloom/agents/kernel"

export KERNEL_AGENT_ROOT="$HYPERLOOM_KERNEL_AGENT_ROOT"

export WORKSPACE_PATH="${WORKSPACE_PATH:-/workspace}"

# TRACELENS_ROOT: leave unset to let install.sh clone AMD-AGI/TraceLens

# to ${HYPERLOOM_CACHE_DIR:-$REPO_ROOT/.cache}/TraceLens@<sha> and pin it

# to a fixed SHA. Only export it as an operator override to point at a

# pre-existing checkout you maintain; this skips both the clone and the

# SHA pin.

# export TRACELENS_ROOT=/path/to/your/TraceLens

# Optional TraceLens-internal checkout; export only to enable it (open-source-only if unset):

# export TRACELENS_INTERNAL_ROOT=/workspace/TraceLens-internal

export PYTHON="${PYTHON:-$(command -v python3)}"

export PATH="$(dirname "$PYTHON"):/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:${PATH:-}"

bash "$INSTALL_SH"

"$PYTHON" -m hyperloom.inference_optimizer.cli --help

```

Quirks: with `set -u`, assign dependent vars on separate lines (chained

`export A=... B=$A` can fail with `unbound variable`). The installer

leaves a live Ray head; `ray status` must succeed because `trace_analyze`

submits tasks with `num_gpus>=1` — never restart Ray with `--num-gpus=0`.

`_preflight()` runs every launch as the in-loop counterpart of IR-2 and

**owns** the things the launcher must NOT do by hand: re-export auth

aliases (LLM) from `OPENAI_API_KEY`,

auto-`pip install` the SDKs / `ray` / `Magpie` /

`InferenceX`, ROCm hygiene, `--gpu-type` auto-detect, and it emits the

canonical `Preflight diagnostics:` block (paste verbatim into status

reports). Two checks **abort** the run on failure: the model gate

(probed against `<OPENAI_BASE_URL>/models`; the allowlist

{`claude-opus-5` preferred, `claude-opus-4-8`, `claude-opus-4-7`,

`claude-opus-4-6` fallback}

binds only under `INFERENCE_OPTIMIZER_ALLOW_CUSTOM_ORCH_MODEL=0` — otherwise

the catalog probe is the sole gate; see

`## Failure Handling`) and, when `--critic-agent` is active, the

critic-agent runtime probe (`## Critic Backend Selection`).

Don't manually pip-install SDKs, start Ray,

or `curl /v1/models` — `_preflight()` owns these. See `docs/reference/kernel-execution-path.md`

for the kernel dispatch and artifact layout.

### Recovery

SIGHUP follows the ordinary terminal drain path, like other supported stop

signals. There is no automatic supervision or restart. After inspecting the

failure and confirming that the old process is gone, explicitly resume the same

session with `--resume-from "$SESSION_DIR"` when appropriate. Historical stop

reasons remain readable; they are not instructions to restart automatically.

`recover-session` remains an offline artifact-reconstruction tool, not a runtime

recovery action.

If the CLI exits with `Claude SDK exit code 1` or `Primus.00009 token not present`,

the gateway rejected the request. Check that `OPENAI_BASE_URL` / `OPENAI_API_KEY`

are set in `.env` (or the calling shell) and that the gateway is reachable:

```bash

curl -sS -H "Authorization: Bearer $OPENAI_API_KEY" "$OPENAI_BASE_URL/models" | head

```

If `_preflight()` itself fails, run install in `--check-only` mode to see

which piece is missing, then re-run full install:

```bash

bash "$REPO_ROOT/src/hyperloom/inference_optimizer/assets/install.sh" --check-only

bash "$REPO_ROOT/src/hyperloom/inference_optimizer/assets/install.sh"

```

If install repeatedly fails while building GEAK / `mini-swe-agent` with

missing files such as `src/minisweagent/...`, the workspace-shared GEAK

mirror may be half-created (`.git` exists but `src/` is incomplete) or

the filesystem may be showing stale metadata. Do not manually clone GEAK,

delete only `build/`, or edit `source-mirrors/` in place. Stop any other

installer using the same `$USER_DATA_PATH`, remove the entire

`${HYPERLOOM_ROOT:-${USER_DATA_PATH:-/workspace/hyperloom}/runtime/source-mirrors}/geak`

directory, then rerun the full install so `install.sh` owns the fresh

clone. Multiple concurrent installs sharing one `$USER_DATA_PATH` also

share `source-mirrors/`; avoid running them at the same time.

In sandboxes where `/workspace/hyperloom` is unwritable, override the

**workspace root** with `USER_DATA_PATH` (not the per-session subdir):

```bash

export USER_DATA_PATH="/shared/hyperloom-sessions" # workspace root

mkdir -p "$USER_DATA_PATH"

```

The CLI calls `make_session_dir(model_name=…)` once at startup; that

creates `$USER_DATA_PATH/<model_basename>/<UTC_ts>/` and pins

`$INFERENCE_OPTIMIZER_CURRENT_SESSION_DIR`.

## Portable Preflight

Implements **IR-1**. Run order is always **IR-2 → IR-1 → launch**:

IR-2 must complete first so `torch` is available. Run the preflight tool

and abort on any non-zero exit before spawning the optimizer:

```bash

"$PYTHON" "$REPO_ROOT/src/hyperloom/inference_optimizer/tools/preflight_optimizer.py" "$MODEL_PATH"

```

A non-zero exit means the GPU state is unknown or a violation was detected;

do not continue to `python -m hyperloom.inference_optimizer.cli optimize`.

## Benchmark Config

Default configs live here:

```bash

src/hyperloom/inference_optimizer/assets/configs/baseline_sglang.yaml

src/hyperloom/inference_optimizer/assets/configs/baseline_vllm.yaml

src/hyperloom/inference_optimizer/assets/configs/profile_sglang.yaml

src/hyperloom/inference_optimizer/assets/configs/profile_vllm.yaml

```

Two fields in each YAML are **fallback only** — the optimizer overrides

them at runtime:

- `benchmark.model` <- `--model` / `$MODEL_PATH`

- `benchmark.runner_type` <- `--gpu-type` / `$GPU_TYPE` / rocm-smi auto-detect

`benchmark.benchmark_script` is deliberately NOT set in the shipped

YAMLs. At materialize time Hyperloom pins it to

`{framework}_{runner_type}.sh` (e.g. `sglang_mi300x.sh` /

`sglang_mi355x.sh`) so Magpie's resolver hits priority 1 (explicit

user override) and uses the generic script — which respects

`RESULT_DIR` and `EXTRA_*_ARGS`. Each shipped YAML has a commented

`# benchmark_script: ...` template right under `framework:` for manual

debug overrides; Orchestration can also route per-task via

`params.benchmark_script` (sanitized).

Before a new model run, verify these fields match the environment:

- `benchmark.model`: model path.

- `benchmark.envs.TP`: tensor parallel size.

- `benchmark.envs.CONC`, `ISL`, `OSL`: workload.

- `benchmark.envs.ROCR_VISIBLE_DEVICES`: GPU pinning.

- `benchmark.envs.PATH`: must lead with the launcher Python's bin dir

(`$(dirname "$PYTHON")`).

### Magpie leak-path salvage (`INFERENCE_OPTIMIZER_RESCUE_PATHS`)

In-loop, defense-in-depth — the launcher does not touch this. Magpie

shell wrappers hardcode artifacts under `/workspace/`

(`inferencex_result.json`, `server.log`, `gpu_metrics.csv`,

`profile_*.trace.json.gz`). When a task's in-workspace search finds no

usable measurement, the executors run an mtime-gated salvage pass over

`$INFERENCE_OPTIMIZER_RESCUE_PATHS` (unset = no salvage) and copy

fresh matches into the task workspace, tagged in `nonfatal_warnings`

(`rescued_from_leaked_path:` / `harvested_leaked_artifact:`). Extend the

scan roots via `$INFERENCE_OPTIMIZER_LEAK_ROOTS` if a script leaks

elsewhere; the default `{framework}_{runner_type}.sh` already respects

`$RESULT_DIR` so salvage normally never fires.

Operators only interact through two `task.params` knobs:

`params.benchmark_script` (bare

sanitized `*.sh` name; overrides the gpu_type auto-pick) and

`params.result_dir` (forwarded as `$RESULT_DIR`). A baseline retry after a

failure MUST change at least one of `params.benchmark_script` /

`params.result_dir` / `params.extra_server_args` / `params.extra_envs`

(prompt RULE F1 — LLM-side judgement, not a PolicyGate deny); a proposal

repeating a recent failing params fingerprint is dropped as a duplicate.

With enablement admitted (`--enablement all/launch/eval`), the first baseline

failure routes PRELUDE into the ENABLEMENT phase, which authors patches until

the combo boots. With `--enablement off`, three consecutive failures stop the

run with `stop_reason='baseline_failed'`.

Operator server flags have one supported CLI entry point:

`optimize --server-args "<framework serve flags>"`. The CLI exports this as

`INFERENCE_OPTIMIZER_SERVER_ARGS`, and YAML materialization routes it into

`EXTRA_VLLM_ARGS` / `EXTRA_SGLANG_ARGS` / `EXTRA_ATOM_ARGS` for baseline,

profile, explore, and sweep. Explicit `--max-model-len` / `$MAX_MODEL_LEN`

wins over auto `ISL+OSL+headroom`. A comma `$CONC` value such as

`4,16,128` is accepted for compatibility; baseline uses the first value.

Use `--conc-sweep-concs` to override the ladder SWEEP measures (`256,128,64,32,16,8,4,2` synthetic, `1,4,8,10,14,20,28` under AgentX). Under AgentX the sweep is off unless `--enable-conc-sweep` is passed, since every rung is a 3600 s window.

Operator server flags are the workload baseline, but they are not sacred. When

the configuration arm has evidence or an operator hint that a pinned flag may

be harmful, it

may test an ablation variant with `remove_args` (or `unset_envs` for inherited

environment variables). Do not simulate deletion by adding an unrelated

counter-flag: emit an explicit explore grid entry such as

`{"name": "remove_cuda_graph_max_bs", "remove_args": ["--cuda-graph-max-bs"]}`.

The executor removes those inherited args before appending the variant's

`extra_args`, then records the removal fields in `explore_search` for dedup and

audit.

### Workload-contract reuse (baseline → explore/sweep)

`baseline` materializes its YAML once with the operator's process env

(`CONC` / `ISL` / `OSL` / `TP` / `MAX_MODEL_LEN` / `PRECISION` / `RUN_EVAL`

/ `ROCR_VISIBLE_DEVICES` + adaptive `NUM_PROMPTS` / `NUM_WARMUPS`), saves it

as `baseline_config.with_envs.yaml`, and forwards the path as

`task.params["config_path"]` to every `explore` / `sweep` task — so variants

benchmark the **same workload baseline ran** (without it they'd fall back to

the YAML's fallback defaults `CONC=64`/`ISL=1024`/`OSL=1024`). Per-variant

`extra_envs` still win (applied last).

## Critic Backend Selection

The Critic role has two backend modes. Default is `--critic-agent` (no

flag needed).

| Flag | Backend class | Behaviour |

|---|---|---|

| (none) / `--critic-agent` | `CriticAgentBackend` | Drives the `hyperloom.agents.critic` skill runtime via `python -m hyperloom.agents.critic.runtime.cli prepare-review` → Codex chat completion → `python -m hyperloom.agents.critic.runtime.cli commit-review`. Adds KB priors lookup (with circuit-breaker for unreachable services), per-session memory + idempotent `reviewed_msg_ids` (no double-verdict), `judge_bundle.review_constraints` injected into the LLM prompt, and `needs_review` / `critic_unavailable` source when context is missing. |

| `--critic-mock` | `MockCriticBackend` | Always-approve adapter. Use for offline / smoke tests when Codex creds aren't available. |

Default is overridable per pod via

`INFERENCE_OPTIMIZER_DEFAULT_CRITIC_BACKEND` (one of `mock` / `agent`).

### Review transport: `--critic-protocol {auto,openai,anthropic}`

Both transports run the full critic-agent runtime; only the review reasoning

call differs. There is no degraded critic, so a transport whose credentials are

missing fails at startup rather than silently changing the review quality.

`auto` (the default) picks `anthropic` for an Anthropic-only config and `openai`

otherwise. Force `anthropic` when both sides are configured but the Critic

should run on a Claude subscription; `auto` would otherwise choose the OpenAI

transport.

Per-value behaviour and the exact credential each one accepts are in

[references/critic.md](references/critic.md#review-transport---critic-protocol-autoopenaianthropic).

That table lived here in duplicate and had already drifted from the code once,

so it is kept in one place.

### Required env when `--critic-agent` is active

| Var | Purpose | Default |

|---|---|---|

| `CRITIC_AGENT_ROOT` | Path to the directory containing `runtime/cli.py`. | in-tree `$REPO_ROOT/src/hyperloom/agents/critic/` |

| `CRITIC_KB_CLIENT_MODE` | `inmemory` keeps KB writes / reads off the wire. `live` requires `KB_BASE_URL`. | `inmemory` |

| `KB_BASE_URL` | KB service URL when `CRITIC_KB_CLIENT_MODE=live`. | unset (live mode aborts at start if absent) |

| `KB_TIMEOUT_MS` / `KB_RETRY_MAX` / `KB_DEAD_LETTER_DIR` | Forwarded to the runtime; see `src/hyperloom/agents/critic/README.md`. | runtime defaults |

| `CRITIC_SESSION_MEMORY_DIR` | Where the runtime persists per-session decisions / reviewed_msg_ids. | `$SESSION_DIR/critic-session-memory` (auto-set by the optimizer; co-located with the Coordinator session and cleaned up alongside it). |

| `WORKSPACE_PATH` | Skill root the critic-agent runtime resolves prompt assets against. | `$REPO_ROOT` (auto-set). |

CLI startup checks that `CRITIC_AGENT_ROOT` resolves to a real directory

with `runtime/cli.py`, then runs `python -m hyperloom.agents.critic.runtime.cli --help`

(90s timeout, override via `CRITIC_AGENT_PROBE_TIMEOUT_SEC`) before the

Coordinator boots. Missing or broken runtime aborts

the run with a clear error pointing at `--critic-mock` as the offline

bypass.

### Per-turn artefacts (audit trail)

Each Critic turn writes a 6-digit workdir under

`$SESSION_DIR/critic-workdir/<turn_idx>/` (`request.json` /

`judge_bundle.json` / `review.json` / `emit.json`) plus session memory

under `$SESSION_DIR/critic-session-memory/<session_id>/`. The backend

prunes to the latest 50 turn workdirs each tick. Inspect these when

debugging critic verdicts (see `## Failure Handling`).

## Framework Selection

A session is single-framework. Pick `sglang` (default), `vllm`, or

`atom` via `--framework` or `$FRAMEWORK`:

```bash

python3 -m hyperloom.inference_optimizer.cli optimize --framework vllm --model "$MODEL_PATH" --max-hours 2

FRAMEWORK=vllm python3 -m hyperloom.inference_optimizer.cli optimize --model "$MODEL_PATH" --max-hours 2

python3 -m hyperloom.inference_optimizer.cli optimize --framework atom --model "$MODEL_PATH" --max-hours 2 # IR-8 single-node only

```

Resolution order: `--framework` > `$FRAMEWORK` > `sglang` (default).

What this controls:

- Which Magpie YAML the executors default to —

`baseline_{sglang,vllm,atom}.yaml` and

`profile_{sglang,vllm,atom}.yaml`. The per-framework resolver

`_default_profile_config()` in `src/hyperloom/orchestrator/actions/executors/profile.py` picks

the right file from `$FRAMEWORK`.

- Which framework-specific seed grid the `explore` action falls

back to when no `params.grid` is supplied. atom is the only

framework with a programmatic seed today

(`_default_grid_for_framework("atom", ...)` in

`src/hyperloom/orchestrator/actions/executors/explore.py`, populated by

`_atom_default_grid()`); sglang and vllm continue to rely on

the orchestration LLM emitting `provenance='default_grid'`

variants and will fail with `error_class="empty_grid"` on a

cold-start with no LLM input.

- Which extra-args env name `_grid_runner` writes

(`EXTRA_VLLM_ARGS` / `EXTRA_SGLANG_ARGS` / `EXTRA_ATOM_ARGS`)

- Which KB partition orchestration reads for hints

Mixing frameworks in a single session is not supported; the CLI

locks `$FRAMEWORK` for the run. Resume re-reads `$FRAMEWORK` from the

shell — set it when you resume a non-default session.

**`--framework atom` specifics (IR-8):** single-node only

(`--nodes>=2` fails fast). Shipped configs `baseline_atom.yaml` /

`profile_atom.yaml`; the Magpie atom wrapper bridges `PROFILE=1` to

atom's `--torch-profiler-dir`, and TraceLens consumes the resulting

`*.pt.trace.json.gz` unchanged. atom source roots (`/app/ATOM/atom/`)

are in the kernel search roots + `_REUSABLE_SOURCE_ROOTS`, and the repo

URL `https://github.com/ROCm/ATOM.git` is in `hyperloom.agents.framework.repo_map`.

Unlike sglang/vllm, atom is the only framework with a programmatic

cold-start seed grid (`_atom_default_grid`: `atom_level_{2,3}`,

`atom_prefix_cache`, `atom_kv_fp8` on FP8, model-class-gated `atom_ep` /

`atom_dp_attn` / `atom_mtp_{1,3}`, `atom_cudagraph_bracket`) — sglang/vllm

fail `error_class="empty_grid"` on a cold start with no LLM variants.

## Enablement Targeted Builds

When a run needs a compiled component that only exists in source (AITER

FP4/MLA/NSA kernels, `sgl-kernel`, or vLLM-from-source), the enablement

subsystem acquires it off-loop. The launcher does not drive this; the notes

below describe the runtime behavior operators observe.

- **Off-loop build lane.** Compiled-component builds run on a dedicated

single-slot build lane. Each build is spawned as a detached process group

and polled (reaped) across coordinator ticks against a wall-clock budget, so

a multi-hour compile never blocks the tick loop. An in-flight build is tracked

by a durable sentinel (`pending_targeted_build`) so a crash/resume can recover

or terminate it.

- **Runnable gate.** A verified build does not earn KEEP by

artifact-verification alone — it must actually launch the model through a

launch probe (the enablement runnable-decision gate) before it is kept.

- **Interpreter switch.** A from-source build records the venv interpreter it

was compiled against (`runtime_python_exe`) and emits it as the

`HYPERLOOM_FRAMEWORK_PYTHON` env into the per-variant YAML benchmark envs

(`benchmark.envs`). Both backends export that mapping to the server env, and

bypass additionally launches the server via `python -m` with that

interpreter. This guarantees the server loads the exact build.

- **`build_budget_sec`.** Per-build-action wall-clock timeout knob; `0` selects

the per-component default.

See `docs/reference/environment-variables.md` "Targeted builds (Rung 5)" for the

full env-var set, and `docs/reference/session-breakdown.md` for the emitted

`enablement` / `build_attempts[]` fields.

## GPU Runner Type

Pick the GPU explicitly with `--gpu-type` or `$GPU_TYPE`; without

either, the optimizer auto-detects via `rocm-smi --showproductname`

(falling back to `torch.cuda.get_device_properties(0).gcnArchName`).

```bash

python3 -m hyperloom.inference_optimizer.cli optimize --gpu-type mi355x --model "$MODEL_PATH" --max-hours 2

GPU_TYPE=mi300x python3 -m hyperloom.inference_optimizer.cli optimize --model "$MODEL_PATH" --max-hours 2

```

Accepted values: `mi300x`, `mi308x`, `mi325x`, `mi355x`. **`mi308x` and

`mi325x` map to `runner_type=mi300x`** with a warning, since the GPUs share the

same runner family and Magpie has not shipped `sglang_mi308x.sh` /

`sglang_mi325x.sh` / `vllm_mi308x.sh` / `vllm_mi325x.sh` yet. If you

need a true MI308X/MI325X-specific script, uncomment the `benchmark_script:`

template in the relevant YAML and point it at your script under

`InferenceX/benchmarks/...`.

Do not set `HIP_VISIBLE_DEVICES` on the known ROCm stack unless the user asks;

it can make `torch.cuda.is_available()` return false. Use

`ROCR_VISIBLE_DEVICES` for GPU pinning.

## SGLang Parameter Search

Serving-parameter search runs through the `explore` action (the legacy

`params` / `backends` actions were merged into it); candidates are

written via `EXTRA_SGLANG_ARGS` / `benchmark.envs`. This is internal to

the optimizer — the launcher does not drive it. Useful InferenceX-derived

candidate families a specialist may surface: `--disable-radix-cache`,

`--max-running-requests`, `--tokenizer-worker-num`, `--stream-interval`,

and ROCm/TileLang envs (`SGLANG_OPT_USE_MULTI_STREAM_OVERLAP`,

`SGLANG_HACK_FLASHMLA_BACKEND=tilelang`). Speculative decoding

(`SGLANG_ENABLE_SPEC_V2` / `--speculative-*`) is model-specific — only

where a draft/MTP path exists, benchmarked with chat-formatted prompts.

### Per-Run Asset Override (advanced)

To override shipped configs without editing them, materialize a per-run asset

root and `export INFERENCE_OPTIMIZER_ASSET_ROOT="$ASSET_ROOT"` (env var, not a

CLI flag; `asset_root()` raises `AssetRootNotFound` if the dir is missing).

`mkdir -p "$ASSET_ROOT/assets/configs"`, `ln -sfn` `actions/` from

`$REPO_ROOT/src/hyperloom/inference_optimizer/` and `orchestrator/` from

`$REPO_ROOT/src/hyperloom/`, then

copy + edit the relevant `baseline_*.yaml` / `profile_*.yaml`. Reach for this

only when `_workload_envs.materialize_config_with_envs` defaults don't fit

(e.g. per-yaml `profiler.torch_profiler.enabled`); otherwise `--model` /

`--gpu-type` overrides are enough.

## Launch a New Optimization

Assumes Step 1 (install) already ran. Set `$USER_DATA_PATH` to the **workspace

root** (parent of per-session dirs). The CLI creates

`$USER_DATA_PATH/<model_basename>/<UTC_ts>/` via `make_session_dir`.

Launcher stdout / PID files go under that session's `optimizer_runs/`.

For sandboxes that don't persist `export`s across shell calls (Cursor agents),

copy `src/hyperloom/inference_optimizer/assets/setup_env.sh.example` to a

**session-scoped** path:

`$USER_DATA_PATH/optimizer_runs/setup_env_${CLAW_SESSION_ID:-$(date +%s)}.sh`,

fill in the workload block, and `.` it each call.

**IMPORTANT**: never use a shared filename like `setup_env.sh` — concurrent

sessions on different pods share `$USER_DATA_PATH` via WekaFS; a single file

causes MODEL_PATH race conditions where sessions launch the wrong model.

After launching, locate the optimizer via

`pgrep -af 'hyperloom.inference_optimizer.*optimize'` — `$!` may be a wrapper PID.

**How you launch depends on the harness.** Two things have to hold before the

bash tool's `run_in_background=true` is the right form: `$CLAW_SESSION_ID` must

be set, and your bash tool must take a `run_in_background` parameter. Both, or

`setsid nohup ... &` stays correct.

Each answers half the question. The session id says a platform is reclaiming

this sandbox on what it can see, so a run detached by hand is one it cannot see.

The parameter says this deployment serves background shells at all — switched

off, the tool does not offer one, and there is nothing to hand the block to.

Off-platform neither holds, and `setsid` is what you want there anyway: nothing

is reclaiming anything, and a harness's own background shell may not outlive the

connection that started it. Both forms are detached — the difference is whether

anything outside the process knows it exists.

```bash

set -e

cd "$REPO_ROOT"

INSTALL_SH="${REPO_ROOT}/hyperloom/inference_optimizer/assets/install.sh"

if [ ! -f "$INSTALL_SH" ]; then

INSTALL_SH="${REPO_ROOT}/src/hyperloom/inference_optimizer/assets/install.sh"

. "${INSTALL_SH%/*}/runtime_env.sh"

load_dotenv_no_clobber

export USER_DATA_PATH

# Resolve the launch interpreter in this shell; preflight loads generated runtime state.

export PYTHON="${PYTHON:-$(command -v python3)}"

export PATH="$(dirname "$PYTHON"):/usr/local/bin:$PATH"

export RUN_TAG="$(basename "$MODEL_PATH")-$(date +%Y%m%d_%H%M%S)"

# RUN_LOG/PID/launch-info live under the workspace until the session_dir

# is known; move or re-tail from $session_dir/optimizer_runs/ after reading

# session_dir from the launch-info JSON below.

# /workspace/hyperloom is only the fallback when $USER_DATA_PATH is unset.

export RUN_DIR="${USER_DATA_PATH:-/workspace/hyperloom}/optimizer_runs"

export RUN_LOG="$RUN_DIR/run_${RUN_TAG}.log"

export PID_FILE="$RUN_DIR/run_${RUN_TAG}.pid"

export LAUNCH_INFO_FILE="$RUN_DIR/launch_${RUN_TAG}.json"

mkdir -p "$RUN_DIR"

# $RUN_TAG carries a timestamp, so it cannot be recomputed later. Persist the

# run-scoped vars where the health check can source them: under Claw the launch

# is its own background tool call, so the health check necessarily runs in a

# DIFFERENT shell and inherits none of these exports. Session-scoped name, for

# the same WekaFS reason setup_env.sh must never be a shared filename. Outside

# Claw, set $RUN_ENV yourself before launching if two runs share a host.

export RUN_ENV="$RUN_DIR/run_env_${CLAW_SESSION_ID:-$(hostname)}.sh"

printf 'export RUN_TAG=%q RUN_DIR=%q RUN_LOG=%q PID_FILE=%q LAUNCH_INFO_FILE=%q\n' \

"$RUN_TAG" "$RUN_DIR" "$RUN_LOG" "$PID_FILE" "$LAUNCH_INFO_FILE" > "$RUN_ENV"

python3 -m hyperloom.inference_optimizer.cli --verbose optimize \

--model "$MODEL_PATH" \

--framework "${FRAMEWORK:-sglang}" \

--target-gain "${TARGET_GAIN:-10}" \

--max-hours "${MAX_HOURS:-5}" \

--tick-interval-sec 30 \

--launch-info-file "$LAUNCH_INFO_FILE" \

> "$RUN_LOG" 2>&1 < /dev/null

```

**When both conditions hold**, pass that block to the bash tool with

`run_in_background=true` and no `setsid nohup` and no trailing `&` — the tool is

what detaches it. The

tool returns a `shell_id`, not a pid, so `$PID_FILE` is written in the health

check below from the launch-info JSON, which carries the real one. Do not skip

that reconciliation when checking the optimizer process.

If that reconciliation cannot find a real pid, the health check **removes**

`$PID_FILE` rather than leaving whatever the launch put there. On the `setsid`

path that leftover may be a dead wrapper pid. An absent pidfile means the

optimizer PID is unknown, not that the optimizer stopped. Inspect the launch

log and persisted state before making an explicit recovery decision; a dead

wrapper alone never justifies restarting.

Because that launch is its own background tool call, the health check has to be

a **separate foreground** call — it cannot be appended to the same block, or the

`sleep 30` and every line it prints would run in the background too, invisible

until you poll `bash_output`. A separate call means a separate shell with none of

the launch block's `export`s, which is what `$RUN_ENV` above is for: the health

check and later status queries source it instead of assuming carried-over state.

**Everywhere else**, prefix the `python3 ... optimize` command with `setsid nohup`,

append ` &`, and `echo $! > "$PID_FILE"`. That form is required for runs > 5 min

under Cursor, whose background shell can die on SSH disconnect. Reconcile the

file afterwards all the same: `$!` may be a setsid wrapper that exits immediately,

not the optimizer PID.

**Docker.** Do **not** wrap this launch in `docker exec -d`. Detached exec

discards stdout and stderr, so an optimizer that dies on startup looks like

"backgrounding failed." Run one attached

`docker exec -w "$REPO_ROOT" "${HYPERLOOM_CONTAINER_NAME:-hyperloom-local}" bash -lc` that runs

this same launch block. Preflight loads `kernel-agent.env.sh`; do not source

it. Under Claw, hand that attached exec to the bash tool with

`run_in_background=true` and no `setsid`, `nohup`, or trailing `&`. Everywhere

else, the command inside the exec is the `setsid nohup … > "$RUN_LOG" 2>&1 <

/dev/null &` recipe (plus `--launch-info-file`). Confirm with

`pgrep -af 'hyperloom.inference_optimizer.*optimize'`. If launch-info has no

`.session_dir`, read `$RUN_LOG` and fix that error; do not retry with a

different backgrounding trick.

**Why the difference is load-bearing under Claw.** `setsid nohup ... &` detaches

the run from everything, including the platform. The sandbox is deleted once

`lastActivity + 15m` passes, `lastActivity` only moves for traffic through the

Router, and Claw stops pinging the moment the agent turn reaches a terminal

state — so a hand-detached optimizer is indistinguishable from an abandoned

sandbox, and the pod is reclaimed out from under it about fifteen minutes after

the turn ends. That is not hypothetical: on 2026-09-02 four dispatches lost a

live sandbox exactly this way, the cleanest of them reporting `task.completed`

at 10:43:14 and losing its pod at 10:57:46, with no error in between and the

optimizer still working.

`run_in_background=true` detaches the same way but through the door the platform

can see: Hands registers the shell against the session, which is what lets the

sandbox be kept alive for as long as the run needs and reclaimed normally once

it does not. It also gives `bash_output` and `kill_shell`, which are better than

a pidfile — a `/proc` check answers "alive" for a zombie, and sandbox PID 1 does

not reap.

One thing has to be true on the Claw side for that to hold, and it is the same

thing the second condition above tests: `BG_SHELL_ENABLED` must be on in the

deployment, or the tool refuses `run_in_background` outright and the foreground

ceiling is the only route.

The other half is already there. Claw's keepalive consults the shell registry

before treating a sandbox as idle, so a registered shell holds the sandbox for

as long as it runs and stops holding it once it exits — both directions

confirmed against a live deployment, with a thirty-minute run held across the

end of its turn and the sandbox reclaimed a few minutes after the run finished.

One thing it costs: a background shell is registered in Hands' memory and Hands

takes its shells down with it on SIGTERM, so a Hands restart ends the run where

`setsid nohup` would have outlived it. A sandbox restart ends it either way, and

being visible is worth more than surviving a restart nothing would have noticed.

Critic defaults to `--critic-agent`. See

[Critic Backend Selection](#critic-backend-selection) for `--critic-mock` and

`INFERENCE_OPTIMIZER_DEFAULT_CRITIC_BACKEND` for a pod-level override.

After launching, do a short health check:

```bash

sleep 30

# Re-enter the launch shell's environment. Required under Claw, where this is a

# separate foreground tool call and the `export`s from the launch block are gone;

# harmless when it is the same shell.

RUN_ENV="${RUN_ENV:-${USER_DATA_PATH:-/workspace/hyperloom}/optimizer_runs/run_env_${CLAW_SESSION_ID:-$(hostname)}.sh}"

. "$RUN_ENV"

# `jq` is not on every node, so read the launch-info JSON with python3.

read_json() { python3 -c "import json,sys;print(json.load(open(sys.argv[1])).get(sys.argv[2],''))" "$1" "$2" 2>/dev/null; }

# Authoritative pid, from the CLI rather than from whatever launched it: under

# Claw the tool returned a shell_id, and under setsid `$!` is the wrapper.

pid="$(read_json "$LAUNCH_INFO_FILE" pid)"

if [ -n "$pid" ]; then

echo "$pid" > "$PID_FILE"

else

# A stale wrapper pid is not the optimizer's identity. Missing means unknown;

# inspect the launch log and persisted state without restarting anything.

rm -f "$PID_FILE"

echo "ERROR: no .pid in $LAUNCH_INFO_FILE; removed $PID_FILE rather than" \

"leave a stale wrapper pid. Inspect $RUN_LOG; do not auto-resume." >&2

# Not `test -d /proc/$pid`: a zombie keeps its /proc entry, and sandbox PID 1

# does not reap, so that check reports a dead optimizer as alive indefinitely.

# Ask for the state and reject Z.

ps -o stat= -p "$pid" 2>/dev/null | grep -qv '^Z' \

&& echo "optimizer_alive=true pid=$pid"

# Authoritative session dir from the launch-info JSON (--launch-info-file).

# Never guess by timestamp: overlapping sessions break any "latest dir" pick.

session_dir="$(read_json "$LAUNCH_INFO_FILE" session_dir)"

if [ -z "$session_dir" ]; then

echo "ERROR: no .session_dir in $LAUNCH_INFO_FILE (launch-info JSON missing or" \

"malformed). The optimizer likely died before emitting launch info;" \

"inspect the HYPERLOOM_LAUNCH line and errors in $RUN_LOG." \

"Refusing to guess the session dir from timestamps." >&2

return 1 2>/dev/null || exit 1

test -f "$session_dir/manifest.json" && echo "manifest_present=true session_dir=$session_dir"

test -f "$session_dir/state.json" && echo "state_exists=true" \

&& python3 -c "import json; print(json.load(open('$session_dir/state.json')).get('stop_reason'))"

```

Healthy = optimizer process alive + `manifest.json` + `state.json`

exist + no early `stop_reason`.

Under Claw the launch also returns a `shell_id`; `bash_output` on it is the

cheaper liveness answer and the one that survives the pid moving, so prefer it

and keep the `ps` check for the optimizer PID recorded in the pidfile.

Health checks only observe and reconcile launch metadata; they must not kill,

restart, or automatically resume the optimizer.

## Resume Existing Session

`--resume-from "$SESSION_DIR"` is the only way to resume; there is no

flag that lets the CLI choose a session for you. Get `$SESSION_DIR` from the

authoritative sources listed under the **Session rule** above — never by

walking `$USER_DATA_PATH/<model>/<UTC_ts>/` for the newest dir.

`$USER_DATA_PATH` must stay at the **workspace root** so

`runtime/kernel-agent.env.sh` resolves. The CLI refuses to start if

`manifest.json` or `state.json` is missing in that session dir.

Reuse the Launch template above with these diffs: drop `--model`, add

`--resume-from "$SESSION_DIR"`, set `RUN_TAG="resume-$(date +%Y%m%d_%H%M%S)"`.

Resume preserves baseline, current best, params-search state, event history,

and kernel-agent artifacts; the CLI clears stale `stop_reason` and

`crash_count` before retrying.

**Most of the launch shape does not need re-passing.** `state.json` is the

authority for it, so `--resume-from "$SESSION_DIR"` keeps `--server-args`, every

`--extra-env` pin, and the warm-replay gates from the original launch — the CLI

re-exports the derived env from the persisted state and prints each one it

restored. Re-pass a flag only to *change* it: an explicit flag on the resume

wins and is persisted as the new value for later resumes. Resuming is an

explicit operator decision; no background process relaunches the optimizer.

Three exceptions:

- **`--nodes` must be re-passed for a multi-node resume**, together with

`--mn-backend` / `--gpus-per-node` if they were set and the

`HYPERLOOM_MN_EXT_*` hand-off. The cluster hand-off is resolved from argv before

`state.json` is read, so omitting these flags on a `--nodes >= 2` resume

benchmarks against the wrong endpoint.

- `--reference-script` is fixed at launch — the recipe is parsed into

`state.json` once, so re-passing it on a resume does nothing. Start a fresh

session to change the reference.

- `--server-args` and `--extra-env` are restored as a **set**: re-passing either

replaces the whole thing, so changing one pin means re-passing them all.

## Monitoring

Each poll is one fast read of persisted state, and you poll only when you are

next invoked. **Never block to reach the next poll**: no `sleep`, no wait loop,

no `tail -f`. The only sanctioned wait is the one-shot `sleep 30` health check

right after launch. Do not start a background watchdog or automatic resume

loop. If the user requests recurring status checks, use the hosting platform's

scheduled invocations to read persisted state without holding a connection open.

**Why this is load-bearing, not style.** A blocking call holds the sandbox

connection open for its whole duration. Overlap it with a `roofline` trace flush

— 8 ranks serialising torch traces to NFS, which takes minutes on a large TP=8

MoE — and the connection can drop; the agent harness reads that as an

unreachable sandbox and rebuilds it, which kills the optimizer and the in-flight

profile with it. Session `Kimi-K3_20260818T062936Z_a62853e5` lost 3/3 rooflines,

its `analysis.md`, and its whole KERNEL phase to a `sleep 110; sleep 110;

sleep 80` progress loop. Launch, check once, and report.

Resolve `$SESSION` from the launch-info JSON — never from `$USER_DATA_PATH`,

which is the workspace root, not the session dir.

```bash

# Another separate tool call under Claw: source the run-scoped env for

# $LAUNCH_INFO_FILE before resolving the session dir from it.

RUN_ENV="${RUN_ENV:-${USER_DATA_PATH:-/workspace/hyperloom}/optimizer_runs/run_env_${CLAW_SESSION_ID:-$(hostname)}.sh}"

. "$RUN_ENV"

export SESSION="$(python3 -c 'import json,sys; print(json.load(open(sys.argv[1]))["session_dir"])' "$LAUNCH_INFO_FILE")"

python3 "$REPO_ROOT/src/hyperloom/inference_optimizer/tools/read_optimizer_state.py" "$SESSION"

```

It prints `stop_reason`, `baseline_tput`, `cumulative_gain_validated`, `current_best`,

`last_kernel_opt`, `last_trace_analyze`, `last_conc_sweep`, `explore_last_round`,

`phase`, the latest orchestration and specialist Experience KB injections when

they exist, plus the recent lifecycle events.

Recent action counts from SQLite (last 500 events grouped by category):

```bash

# Self-sufficient: $SESSION from the block above is gone if this is its own

# tool call, so re-source and re-resolve rather than inheriting it.

RUN_ENV="${RUN_ENV:-${USER_DATA_PATH:-/workspace/hyperloom}/optimizer_runs/run_env_${CLAW_SESSION_ID:-$(hostname)}.sh}"

. "$RUN_ENV"

export SESSION="$(python3 -c 'import json,sys; print(json.load(open(sys.argv[1]))["session_dir"])' "$LAUNCH_INFO_FILE")"

python3 "$REPO_ROOT/src/hyperloom/inference_optimizer/tools/event_counts.py" "$SESSION"

```

## Expected Flow

The optimizer should:

1. Establish or reuse `baseline_tput`.

2. **Coordinator** auto-enqueues an analysis task at the end of

PRELUDE (after baseline) and at each validated-tput watermark

(`current_tput / last_roofline_tput >= 1.10`; compound). Default is

`roofline` (profile + trace_analyze + analysis.md); `--no-enable-roofline`

switches to plain `profile`. The LLM cannot propose either —

both names are Coordinator-managed and a proposal for either is denied with

`rule="coordinator_managed_action"`. Concurrent GPU work is

serialised by the lane / GPU lease rather than a policy deny, so

explore / kernel dispatches keep flowing while analysis refreshes.

Each analysis also stamps a decode roofline ceiling

(`src/hyperloom/inference_optimizer/roofline_ceiling.py`) for the report's

`## Roofline Comparison` section.

3. Run `trace_analyze` once per trace/config and cache the result in

`last_trace_analyze`.

4. Pick only `reusable_native_kernel_ids` for `run_optimization`.

5. Require compile + correctness + microbench/E2E evidence before KEEP.

6. Use `explore_search` to test parameters incrementally and remember

rejected candidates across resume. The ledger keys entries by

**content fingerprint** (a sha1 hash of sorted `extra_server_args` +

sorted `extra_envs`), so renaming an already-tested variant does not

bypass dedup — LLM-supplied `params.grid` is filtered through the same

ledger as the default seed grid.

7. Use `optimization_stack` so backend + params + kernel changes do not

overwrite each other.

8. Use `sweep` to understand workload-specific results beyond the smoke

workload.

## Cache Topology & Cold-start Discipline

SGLang/vLLM on ROCm route hot fused kernels (RMSNorm / attention / MoE / GEMM /

RoPE) through `aiter`, which JIT-compiles per-shape variants on first sight

and caches `.so` on disk. First launch of a fresh (model, dtype, TP,

`max_model_len`, `max_num_seqs`, `gpu_memory_utilization`) signature can spend

30+ min in `hipcc` for 671B FP8 MoE; later launches reuse the cache in seconds.

### Cache locations

| Cache | Path | Clear |

|---|---|---|

| aiter JIT (primary cold-start cost) | The runtime-selected JIT directory (`AITER_JIT_DIR`, otherwise the package `jit/` or initialized user cache), containing serving `.so` modules and `build/` staging | Manual staging cleanup is limited to `build/`. Serving modules are invalidated through the shared JIT transaction, with the scope chosen by the caller. |

| Triton | `~/.triton/cache/` (resolves via `$HOME`) | `rm -rf ~/.triton/cache` |

| torch.compile / Inductor | `/tmp/torchinductor_<user>/` (override `$TORCHINDUCTOR_CACHE_DIR`) | `rm -rf /tmp/torchinductor_root` |

An AITER runtime-JIT source patch invalidates the full serving-module set together

with `build/`; revert restores its baseline and removes candidate artifacts in that scope.

CSV/GEMM registry preparation invalidates only its selected modules, preserving

unrelated serving modules. Full invalidation can require substantial first-use

compilation; its cost depends on the installed cache and workload. Do not delete

serving modules manually or infer patch success from a warm, stale module.

`sgl_kernel` (`site-packages/sgl_kernel/common_ops.*.so`) is build-time only;

only `kernel_opt` / `integrate` may rebuild it.

### Cold-start triggers

First launch on this pod; change to `--max-model-len` / `--max-num-seqs` /

`--gpu-memory-utilization` / `--cuda-graph-max-bs` / `--quantization` /

`--enable-torch-compile`; pod rebuild; manual cache `rm`; aiter source patch.

### Benchmark deadlines and output visibility

Benchmark subprocesses use two finite, positive limits:

- `INFERENCE_OPTIMIZER_BENCHMARK_TIMEOUT_SEC` (default **7800** seconds) is the

hard wall-clock limit for each actual spawn, including server boot and accuracy

evaluation. Output never extends this deadline.

- `INFERENCE_OPTIMIZER_BENCHMARK_SILENCE_TIMEOUT_SEC` (default **600** seconds)

measures output silence only after real server readiness. It is not armed

during pre-ready startup or for scriptable workloads without a serving server.

Session `--max-hours` and cancellation remain effective independently of those

limits. Session exhaustion is cooperative: it cannot guarantee termination of a

frozen Coordinator. Busy logs are not proof of useful progress. Setting

`PYTHONUNBUFFERED=1` only affects Python output; it does not force shell or native

programs, or upstream subprocess capture, to stream output. The Magpie streaming

fix is local-only and unpublished; do not assume the installed pin includes it

or that a run has been validated with it.

Cold-start detection is diagnostic, not a separate benchmark timeout policy.

Profile, KernelForge, GEAK, LLM-call, and framework-owned watchdog budgets retain

their own contracts. In particular, SGLang's server watchdog is independent of

these optimizer benchmark limits.

## Kernel Apply Safety

Kernel optimization may modify `/sgl-workspace/aiter`, `/sgl-workspace/sglang`,

or compiled artifacts. Before applying a patch:

- Back up source files.

- Back up compiled `.so` / `.co` artifacts when available.

- On REVERT, restore compiled artifacts first, then source files, then restart

the server. Avoid a rebuild on revert when the original compiled artifact was

backed up.

- Only KEEP when correctness and E2E are acceptable.

If the user has not explicitly approved environment mutation, stop before real

apply/rebuild and ask. Dry-run and analysis are safe.

## Kernel E2E Retry Discipline

Microbench speedups are not enough. After `run_optimization` returns a candidate

kernel patch, `integrate` must validate the patch with E2E Magpie throughput and

record every attempt in `state.json`.

For the same `kernel_id + patch_path + EXTRA_SGLANG_ARGS`:

- `KEEP`: accept only when E2E gain clears the configured threshold.

- `REVERT`: reject that patch immediately and do not run it again.

- `NEEDS_REVIEW`: allow at most 3 E2E attempts. If none clears the KEEP

threshold, reject that patch and move on to params search or a different

reusable native kernel.

Do not repeatedly integrate the same patch because its microbench was strong.

If E2E results are unstable around zero gain, the correct action is to mark the

patch rejected, preserve the artifacts for human review, and spend the remaining

budget on untested params/backend candidates or the next kernel.

## Failure Handling

Auth / SDK drift (`Claude SDK exit code 1`, `Primus.00009 token not present`,

`ANTHROPIC_AUTH_TOKEN not set`, `BackendError: claude-agent-sdk not installed`,

`Fatal error in message reader`) is owned by `_preflight()`; see

`## Setup → Recovery` for explicit diagnosis and an install rerun. Manual SDK

fallback if frozen pip blocks `_ensure_python_sdks()`:

`python -m pip install 'claude-agent-sdk>=0.2.110' 'openai>=1.50' 'httpx>=0.27'`.

Transient SDK errors retry/resume up to the Coordinator emergency threshold.

### Model-gate errors (preflight #10)

Custom orchestration models are enabled by default and are validated against the

configured gateway catalog. Set `INFERENCE_OPTIMIZER_ALLOW_CUSTOM_ORCH_MODEL=0`

only when you intentionally want the strict AMD Claude allowlist

(`claude-opus-5` / `claude-opus-4-8` / `claude-opus-4-7` / `claude-opus-4-6`).

| Symptom | Fix |

|---|---|

| `--claude-model=... is not allowed` | You likely set `INFERENCE_OPTIMIZER_ALLOW_CUSTOM_ORCH_MODEL=0`; unset it or set it to `1`, then ensure the model appears in the gateway `/models` catalog. |

| `gateway catalog unreachable after retries` (4 probes at 0/1/3/5s) | Reproduce: `curl -k -H "Authorization: Bearer $OPENAI_API_KEY" "$OPENAI_BASE_URL/models" \| jq '.data[].id'`. Gateway answers → proxy/SSL is wrong; gateway down → fix gateway. Fail-fast is intentional vs. 401 mid-baseline. |

### Critic-agent runtime errors

Inspect `$SESSION_DIR/critic-workdir/<latest>/{request,judge_bundle,review,emit}.json`.

Bypass with `--critic-mock` for offline / smoke runs. See

`## Critic Backend Selection`.

| Symptom | Fix |

|---|---|

| `--critic-agent selected but critic-agent runtime not found` | `export CRITIC_AGENT_ROOT=/path/to/src/hyperloom/agents/critic`, or check the `src/hyperloom/agents/critic/` install. |

| `hyperloom.agents.critic.runtime.cli prepare-review/commit-review exited rc=2` | Schema/validation bug (per `src/hyperloom/agents/critic/README.md` §Exit codes). Inspect workdir payload; retry with `--critic-mock` while fixing. |

| `hyperloom.agents.critic.runtime.cli ... timed out after 30s` | KB stuck. If `CRITIC_KB_CLIENT_MODE=live`, drop to `inmemory`. Reproducing in `inmemory` is a bug — that path must not block on I/O. |

| All verdicts `('needs_review','critic_unavailable')` + `kb_skipped=missing_critical_context` | Static context load failed. Check `manifest.json` has non-empty `model_name`/`framework`; grep `logs/cli.log` for `critic_agent_backend static_context`. |

### Run-time signals

- `No accelerator` (Magpie): subprocess `PATH` must lead with `$(dirname "$PYTHON")` (or set `MAGPIE_PYTHON`); use `ROCR_VISIBLE_DEVICES`, not `HIP_VISIBLE_DEVICES`.

- Repeated `trace_analyze` with unchanged trace/config: bug — reuse `last_trace_analyze`.

- `correctness_passed=false`: do not integrate; the kernel-agent report must contain explicit correctness evidence.

- `stop_reason=no_more_leverage`: stop and report; only resume if the user changes workload / search space / model / strategy.

- `policy_denied` streaks (not a stop reason — nothing in the runtime stops a run over them): repeated denial of the same (action, rule) pair is advisory only — there is no auto-prune at streak ≥5 and no stop at streak ≥10. Inspect `SharedState.policy_denial_history` via the `why_denied` tool or the `=== Recent policy denials ===` block, then change something substantive (a new `params.grid` variant, a different `benchmark_script`, or a sibling action family). Do not hand-edit `state.json`.

- `stop_reason=time_exhausted`: resume same session (`--resume-from`); do not start fresh.

## Report Back To User

Report concise status:

- session id (from `manifest.json`) and log path

- `cumulative_gain_validated` and `current_best`

- explore accepted/rejected summary

- last kernel optimized, correctness, micro speedup, E2E gain, decision

- any new `experience_kb_injections` entry: each injected Experience's summary and its section of `prompt_block`

- whether the process is still running or stopped and why