git:20260913.1425eeb to git:20260913.bdd1f28

---

name: srtctl

description: Author, validate, submit, and read back srt-slurm (srtctl) benchmark jobs on a Slurm cluster

---

# srtctl

`srtctl` runs LLM inference benchmarks on Slurm: it reads a recipe (YAML), asks Slurm for the nodes, launches the engine workers (SGLang, vLLM, TRT-LLM, or the Dynamo mocker), the frontend (Dynamo, or the engine's own router), the services they need (etcd, NATS, the Mooncake master, the metrics exporters, anything declared under `services:`), the tachometer metrics scraper, and the benchmark client, then collects logs, a benchmark rollup, the tachometer parquet, and a per-run HTML dashboard under `outputs/<job_id>/`.

## Before anything else

- Work from the srt-slurm checkout that has the cluster's `srtslurm.yaml` (account, partition, `srtctl_root`, model and container aliases). Never invent aliases; read that file.

- Run `srtctl dry-run -f <recipe>` before `srtctl apply`. It renders the sbatch script, every mount and environment variable, and every service (implied ones are marked `implied by:`). A recipe that fails dry-run will fail on the cluster.

- Recipes are schema 2 (`schema: 2`). Write block-style YAML, never `{}` or `[]` flow style. Keep every recipe loadable by `srtctl dry-run`.

## The 2.0 recipe shape

```yaml

schema: 2

name: "qwen3-0.6b-sglang-dynamo-agg"

model:

path: "qwen3-0.6b" # alias from srtslurm.yaml, an hf:<repo> spec, or a path

container: "sglang" # alias or image

precision: "bf16"

resources:

gpu_type: "h100"

gpus_per_node: 8

frontend:

args:

router-mode: "kv"

engine: sglang # sglang | vllm | trtllm | mocker, or a mapping with engine-wide knobs

roles: # one block per worker role: prefill, decode, agg

agg:

nodes: 1

workers: 2

gpus: 1

env:

PYTHONUNBUFFERED: "1"

args: # the engine's own CLI flags, as a mapping

tensor-parallel-size: 1

mem-fraction-static: 0.5

benchmark:

type: "sa-bench" # sa-bench | sglang-bench | gsm8k | custom | manual | ...

isl: 128

osl: 128

concurrencies: "4x8"

```

- `dynamo.source` chooses how Dynamo is installed: `pypi: "1.4.2"`, `wheel: <path>`, or `git: <url>` with `rev: <sha, tag, or refs/pull/N/head>`; `srtctl apply` pins the rev to a commit.

- `placement.node: dedicated` on `frontend` or `benchmark` reserves a node for it.

- `services:` declares sidecars. etcd and NATS (Dynamo), the Mooncake master (when a `mooncake-master` service is declared), and the DCGM and node exporters (tachometer) are implied; declare one by name only to change it (`placement.node: dedicated`, `container`, `options`, `external: <address>`, `enabled: false`).

- `--set KEY=VALUE` and `--unset KEY` on `apply` and `dry-run` override any recipe key without editing the file: `--set resources.gpu_type=b200 --set roles.agg.gpus=2`.

- `srtctl migrate -f <recipe> --in-place` rewrites a v1 recipe (`backend:`, `*_environment`, `infra:`) to this shape; `--verify` proves the two resolve identically.

## Commands

```bash

srtctl dry-run -f recipe.yaml [--set K=V ...]

srtctl apply -f recipe.yaml -y --json # one JSON line per submission: slurm_job_id, output_dir

srtctl apply -f recipe.yaml --serve-only # keep the endpoint up, no benchmark

srtctl monitor # live view of your jobs

srtctl migrate -f recipes/ --verify

squeue --me ; sacct -j <id> -X ; scancel <id>

```

## Reading a run

Everything is under `outputs/<job_id>/`:

- `logs/sweep_<job_id>.log`: the orchestrator. Stages in order: services (infra), workers, frontend, health, benchmark, cleanup. `[ERROR]` lines and `Critical process ... exited` tell you what died.

- `logs/<node>_<mode>_w<i>.out`: one per worker. `logs/<node>_frontend_0.out`, `logs/<node>_router_0.out`: the frontend.

- `logs/service_<name>.out`: etcd, nats, dcgm-exporter, node-exporter, mooncake-master, and declared services.

- `logs/benchmark.out` and `logs/benchmark-rollup.json`: the client and its normalized result.

- `logs/tachometer/raw/scrape/final.parquet`: every scraped metric sample; `logs/perf_dashboard.html`: the rendered dashboard.

- `recipe.lock.yaml`: the exact resolved recipe, pinned sources, and container identity.

Cleanup is graceful: workers, frontends, and services are SIGTERMed through their Slurm steps, then etcd and NATS. A `scancel` of the job triggers the same path.

## MCP

`srtctl-mcp` exposes the schema tools (`schema_summary`, `explain_field`, `validate_config`, `resolve_config`, `get_config_reference`) anywhere, and the job tools (`submit_job`, `dry_run`, `job_status`, `job_logs`, `list_jobs`, `cancel_job`) when it runs on a Slurm login node of the cluster.