AGENTS.md@src/srtctl/core git:20261008.9c44395A

Immutable. This exact content never changes and is served at /api/v1/blob/5cbb73b23d41e62c.

# Core

Rules for `src/srtctl/core/` (schema, runtime, topology, processes, health). Status reporting (`core/status.py`) is covered in `src/srtctl/status_server/AGENTS.md`.

## Ports

Fixed ports are constants in `src/srtctl/ports.py`. Every port a worker process binds is a `PortKind` in the same module (`name`, `base`, `stride`, `per_node`) and is handed out by `NodePortAllocator.next(kind, node, size)` in `src/srtctl/core/topology.py`: per-node counters for listeners only bound on that node (`HTTP_PORTS`, `BOOTSTRAP_PORTS`, `DP_RPC_PORTS`, `DIST_INIT_PORTS`, `TOKENSPEED_PORTS`) and global counters for side channels peers address (`SYS_PORTS`, `KV_EVENTS_PORTS`, `NIXL_PORTS`, `KVBM_ZMQ_PORTS`, `NCCL_PORTS`, `VLLM_SCAN_PORTS`, `MORIIO_HANDSHAKE_PORTS`, `MORIIO_NOTIFY_PORTS`, `TRTLLM_DIST_INIT_PORTS`, `SIDECAR_GRPC_PORTS`). `size > 1` reserves a block when the engine adds a rank offset to the port it is given (MoRI-IO adds the local DP and TP rank to its notify base). `endpoints_to_processes` allocates the generic kinds and stores them on `Process`; a backend adds its engine-specific kinds with `dataclasses.replace` on the way out (SGLang `nccl_port` and `dist_init_port`, vLLM `vllm_scan_port` or, for a discovery connector, `moriio_handshake_port` and `moriio_notify_port`, TRT-LLM `trtllm_dist_init_port`, TokenSpeed `tokenspeed_port` and a `dist_init_port` block). `do_sweep` builds the one allocator per job, seeding the sidecar gRPC base from `dynamo.sidecar_port`.

Rules that follow: a new listener is a new `PortKind` plus a `Process` field, allocated in the topology builder, never `some_base + (sys_port - DYN_SYSTEM_PORT_BASE)` or any other arithmetic on another port at command-build time. Consumers read the field; a `None` means the topology was built without that kind, and a consumer that needs it raises rather than guessing. A worker's own address is `get_hostname_ip(node, runtime.network_interface)`, not upstream's interface guess. Fixed scan bases such as `VLLM_PORT` exist only to keep co-located `get_open_port()` scans apart; if a connector allocates inside forked children, leave the base unset so the kernel assigns ports. Frontends bind `FRONTEND_PUBLIC_PORT` (8000), or `FRONTEND_INTERNAL_PORT` (8180) behind nginx when `enable_multiple_frontends` is set. Service ports are `ServiceKind` defaults overridden by `options`. `tests/test_port_allocator.py` asserts over every example recipe that no two processes on a node share a port and global kinds never repeat.

## Liveness

Readiness belongs to the frontend (`src/srtctl/frontends/AGENTS.md`); `core/health.py` owns only timing, abort, and progress logging, plus the probe shapes frontends build on.

Liveness is the process registry's (`core/processes.py`). `check_failures()` fails a critical process when its srun exits non-zero, or when the step is still running but a new line of its log matches one of the process's `fatal_log_patterns`; the monitor thread then sets `stop_event`, which makes `wait_for_model` return `False` and the run exit 1 with the matched line in the failure details. The patterns come from the backend (`Backend.fatal_log_patterns(mode)`, a tuple of regular expressions; `()` for an engine whose step exits with it, the launcher's `Rank<N> Task exit code: <non-zero>` line for TRT-LLM, whose `trtllm-llmapi-launch` keeps the step alive after the engine dies) plus the recipe's `health_check.extra_fatal_log_patterns`; `health_check.fatal_log_markers: false` hands the registry an empty tuple. A marker must be a line that is only ever printed after the engine is gone: `Traceback` and `MPI_Abort` appear in healthy runs and are not markers. `SrunConfig.kill_on_bad_exit` (TRT-LLM: true) adds `--kill-on-bad-exit=1` to every endpoint step so a task that does exit non-zero ends the step.

## Process cleanup and graceful shutdown

Every long-running `srun` (workers, frontends, nginx, services, tachometer) is launched with a `step_name` and tracked as a `ManagedProcess` carrying the same name. `ProcessRegistry.cleanup()` stops processes by `shutdown_tier`: tier 0 (workers, frontends, sidecars) is SIGTERMed all at once through `scancel --signal=TERM --full <job>.<step>`, waited for up to each process's `terminate_timeout`, and killed if still up; then tier 1 (Mooncake master, stores), then tier 2 (etcd, NATS). SIGTERM aimed at the `srun` client itself aborts the step and SIGKILLs the task, which is why the step name matters: it is the only way the engine, router, or scraper sees the signal and gets to deregister, drain, or flush. New launch sites must pass `step_name` to both `start_srun_process` and `ManagedProcess`.

## ResourceConfig

Recipes set worker shape per role (`roles.<role>.nodes`, `workers`, `gpus`, with `nodes: colocate` for decode on the prefill nodes); `docs/topology.md` shows the forms. They load into `RoleConfig` entries (inheriting the backend-facing `RoleSettings` data contract) on `SrtConfig.roles` (`core/schema.py`); `SrtConfig.topology` derives the per-role counts (`gpus_per_prefill`, `num_decode`, `total_nodes`, ...) that the rest of the code reads. Add a derived value as a property there instead of recomputing it in a consumer.