# DESIGN — real parallel execution and real runtime control flow for `npa.workflow/v0.0.1` This document covers the two tiers added to the `npa.workflow` engine: 1. **Parallel execution** — a spec can declare a fan-out group that launches as genuinely concurrent SkyPilot jobs, with a barrier and bounded concurrency. 2. **Runtime control flow** — an orchestrator above `build_scheduler_task` that submits a wave, polls it to a terminal state, reads the *actual* decision artifact from S3, and replans: real early-exit, real data-dependent branching, retry/idempotency/resume, and a trigger/watch pattern. Both are **additive and opt-in**. Every pre-existing spec plans, renders and submits exactly as before, and the plan-time `--assume-decision` path is retained as the offline/plan-only mode. --- ## 1. Where the engine was | Layer | Behaviour before this change | | --- | --- | | `interpreter.build_plan` | Statically unrolls loops (`for iteration in range(1, max+1)`) into a flat serial plan; the loop-exit predicate is fed a **plan-time assumption** (`--assume-decision`) | | `interpreter.run_workflow(execute=True)` | A dynamic walker that *does* read `config.decision_uri` from S3 (`decision_reader`) — but it executes every step locally with `subprocess` | | `scheduler.build_scheduler_task` | Portable per-step task doc (name, resources, command, image, outputs) — the seam | | `skypilot_render.render_skypilot_yaml` | Emits `execution: serial` pipelines and **rejects** anything else | | `submit.prepare_npa_workflow_for_submit` | One-shot: load → plan → render → `sky jobs launch`. No monitor, no replan | | `decisions.py` + `write_*_decision` toolRefs / `data_factory_stages.grade_gate` | Already *produce* and normalize decision artifacts; only a runtime *consumer* was missing | So the missing pieces were: a concurrent submission shape, and a driver that consumes decisions between stage submissions. --- ## 2. Spec surface (all optional, all `npa.workflow/v0.0.1`) ```yaml states: sweep: # (1) fan-out group parallel: [v-lr-1e-3, v-lr-3e-4, v-entropy-0, v-entropy-0-01] maxConcurrency: "{{config.max_concurrency}}" # int or config ref needs: [prepare] next: select-best # barrier edge, owned by the group v-lr-1e-3: # (2) per-state config overlay resources: trainer-gpu params: variant: lr-1e-3 overrides: "agent.algorithm.learning_rate=1.0e-3" variant_uri: "s3://{{config.bucket}}/{{config.prefix}}/variants/lr-1e-3/" run: { shell: "... {{config.variant}} {{config.variant_uri}} ..." } ingest: # (3) trigger / watch trigger: uri: "s3://{{config.bucket}}/{{config.prefix}}/inbox/" pollSeconds: 30 maxPolls: 20 minObjects: 1 toolRef: workbench.dataset.ingest ``` ### Why these shapes * **`parallel:` is a sibling of `sequence:`.** Fan-out already existed in the catalog (`av-night-scene-hardening.yaml` lists two per-view branches under `sequence:`), and `docs/workbench/npa-workflow-guide.md` already named `parallel` as the field advanced scheduling should use. Rejected: Jinja loops, `foreach` templating. * **`params:` instead of `{{item.*}}` matrix templating.** A `toolRef`'s argv template is fixed, so sweep members must differ *somehow*. A per-state config overlay needs **no new token scope** — `{{config.learning_rate}}` simply resolves against `config | params` for that state — and the resulting spec is the direct analogue of the SkyPilot template's four explicit task documents. `params` is also useful outside sweeps (the gate-loop's `escalate` state reuses one dashboard toolRef with a different output prefix). * **`trigger:` gates a state that does work.** A wait-only state would render as an empty scheduler task; validation rejects it. Watermarks live in the runtime ledger so a resumed run does not wait again for data it already saw. ### Validation rules (all fail at `validate-spec`) * `parallel` and `sequence` are mutually exclusive; a `parallel` state may not also carry `run`/`toolRef`, and may not carry `loop` (wrap it in a `sequence`). * Members must exist, be unique, be leaf states (no `sequence`/`parallel`/`loop`), must not be `terminal`, and must not declare `next`/`transitions` — the group owns the barrier edge, so the graph stays deterministic. * `maxConcurrency` requires a `parallel` group and resolves to `>= 1`. * `params` values and `trigger.uri` are token-resolved at validate time (they see the same per-state overlay the command does). * `trigger` requires `run`/`toolRef` on the same state. ### Why not bump the apiVersion `SUPPORTED_API_VERSIONS`, `blueprints.py`, the submit matrix, ~36 shipped specs, the skills and the guide all key on `npa.workflow/v0.0.1`. The new fields are optional and ignored by every existing spec, so a `v0.0.2` fork would duplicate the catalog for zero semantic gain. The guide's "out of scope (v0.0.1)" list said these features "belong in spec v0.0.2+ as explicit fields (`parallel`, `gang`, `foreach`)"; that section is updated in this change — we adopt the *explicit field* direction it asked for, inside v0.0.1, and note it in the doc. `gang` and `foreach` remain out of scope. --- ## 3. Parallel model ### Planning: one plan, two lenses `build_plan` **flattens** a `parallel:` group in declared order, exactly like `sequence:`. Consequences: * `--plan-only` output, `plan-spec`, and every existing plan-only guardrail (including "every rendered header says `execution: serial`") are unchanged for serial **and** parallel specs. * Offline preview of a parallel spec is still meaningful (it is the serialized execution of the same work). `waves.build_wave_plan` is the second lens: it folds the flat step list into **waves** using `PlanStep.group`: | Wave kind | Contents | Submitted as | | --- | --- | --- | | `serial` | exactly one step | SkyPilot pipeline document (`execution: serial`) | | `parallel` | the members of one group | SkyPilot **JobGroup** (`execution: parallel`), chunked by `maxConcurrency` | `plan-spec --waves` prints this shape offline. ### Rendering: a separate entry point, serial output unchanged `render_skypilot_yaml` still raises on anything but `execution: serial` — the historic guardrail test passes verbatim — and its **output is byte-identical** to before (verified by planning three dynamic specs on the base commit and diffing). Its *body*, however, was refactored: it now delegates to a shared `_render_docs(..., execution="serial")` that the parallel renderer also uses. So the accurate claim is "**serial output unchanged**", not "serial renderer untouched": there is one shared code path, and the guard plus the byte-identical-output check are what protect it. Parallel rendering is a separate public entry point, `render_skypilot_job_group_yaml` (with the dispatching `render_skypilot_steps_yaml`). Every task doc is still produced by `build_skypilot_task_doc` → `scheduler.build_scheduler_task`, so the portable-task seam is intact and the runtime tier never reaches into rendering internals. ### Why a SkyPilot JobGroup SkyPilot 0.12 treats a multi-document YAML whose header sets `execution: parallel` as a JobGroup: * all tasks share **one managed `job_id`** but each launches **its own cluster concurrently** (`sky/jobs/scheduler.py`: *"For JobGroups, multiple tasks share the same job_id but each launches a different cluster in parallel"*); * `primary_tasks` is intentionally **omitted**, which marks every task primary, so the group only reaches a terminal state once **all** members do. That last property is the barrier, enforced by SkyPilot itself rather than by bookkeeping in the driver. Polling stays a single `workflow_status(job_id)` call, and the existing aggregation ("SUCCEEDED only if all tasks SUCCEEDED") already handled multi-task jobs. ### Barrier and bounded concurrency * **Barrier**: the wave boundary. The runtime tier submits the downstream state only after the group's wave reached a terminal state. Live evidence is the per-task timeline: the barrier task's `submitted_at` is later than every member's `end_at`. * **Bounded concurrency**: JobGroups have no concurrency cap, so a group larger than `maxConcurrency` is **chunked into batches**, each batch one JobGroup, batches submitted in order. The declared bound is therefore authoritative and observable (`plan-spec --waves` prints the batch count). `--max-concurrency` overrides every group at submit time (cost control for GPU sweeps). --- ## 4. Runtime tier ### The state machine ``` for each wave produced by the traversal: key = f"{sequence}|{group}|{loop_label}:{state}:{iteration},..." if resuming and ledger says key succeeded -> replay record, continue render wave (1 step -> serial doc, N steps -> JobGroup batches) submit_workflow(...) -> job_id [ledger] poll workflow_status(job_id) every poll_seconds until terminal while polling a parallel wave, sample per-task statuses [ledger] SUCCEEDED -> record per-task timeline [ledger] FAILED -> retry (<= --retries, backoff) else fail the run timeout -> cancel job + cluster, fail the run if the state writes a decision or has transitions: load the decision artifact from S3 (existing contract) [ledger] traversal decides: iterate / break early / goto / next / terminal ``` ### It is the existing traversal, not a second engine `interpreter._execute_state_machine` already implements loops, `loop.until`, `transitions`, `needs`, depth guards and *S3 decision refresh*. The runtime tier injects two things into it: * `step_executor` — `SkyPilotWaveExecutor`, whose `execute(step)` / `execute_parallel(steps, group, max_concurrency)` replace the local `subprocess` execution; * `trigger_waiter` — the driver-side S3 watcher. The default (`step_executor=None`) still calls the module-level `_execute_step`, so local `--execute` behaviour and the tests that monkeypatch it are untouched. The alternative — a new iterative interpreter with a serializable cursor — was rejected: it would duplicate loop/branch semantics and the two implementations would drift. A unit test pins the equivalence directly: for a fixed decision sequence, the runtime submission order equals `build_plan(assume_decision=)`. ### Decision contract (consumed, not reinvented) Decisions are read through the existing helpers: * `decisions.refresh_context_decision(context, reader)` → `config.decision_uri` * `decisions.load_decision` → `decision_from_payload` (`decision` / `last_decision` / `action`) → `normalize_decision` * producers unchanged: `workbench.*.write_*_decision` toolRefs and `data_factory_stages.grade_gate` (which derives the decision from a **real** VLM eval score). `RecordingDecisionReader` wraps the reader so every runtime gate read (URI, raw body, normalized decision, timestamp) lands in the ledger — that is what makes a live claim auditable after the fact. Failure modes are deliberately asymmetric: a gate artifact that **does not exist yet** falls back to the plan-time assumption (recorded in the ledger with `source: assume_decision_fallback`), because that is the documented offline behaviour; a gate artifact that exists but is **unreadable or malformed** fails the run, because silently looping on corrupt JSON would be worse than stopping. Two control-flow shapes fall out of the existing semantics: * **Bounded loop with early exit** — `loop.until: promote_checkpoint` on a `sequence` whose member sets `writesDecision: true`. The decision is re-read from S3 after each iteration, so a promoting gate breaks the loop *before* submitting the remaining iterations. * **Data-dependent branching** — `transitions: [{when: promote_checkpoint, goto: publish}, {when: loop_back, goto: escalate}]` on a state outside a loop body. The runtime reads the artifact and follows the matching edge. (Inside a loop body, `follow_transitions=False` — the loop drives control flow; that is pre-existing engine behaviour, documented in the catalog's blueprints.) ### Durability, idempotency, resume `RuntimeLedger` writes `npa.workflow.runtime.v1` to `/npa-workflow/runtime.json` through the existing `RunStateStore` (reader/writer injection = unit-test seam), containing every wave attempt (key, states, kind, job id/name, sky status, timings, per-task timeline, concurrency observations, outputs, error), every decision read, and every trigger watermark. `RunManifest` (`npa.workflow.run.v1`) is untouched. Resume is **memoized replay**: `--resume` re-runs the traversal, and any wave whose key is already recorded as `succeeded` is replayed from the ledger instead of resubmitted. This works because the traversal is deterministic given the same decision artifacts, which are re-read from the same S3 objects. Re-running a completed run is therefore a no-op, and a run that failed at wave *k* resumes at wave *k*. Wave keys embed the loop label, state name, iteration and a monotonic sequence number, so a loop body that runs three times produces three distinct keys. **Determinism constraint (important).** Because the key carries that in-process sequence number, replay is only sound while the traversal is deterministic. It is deterministic when the inputs are: decisions are re-read from the same S3 objects and trigger watermarks are already satisfied. If a gate artifact *changes* between runs (a later run reads `promote_checkpoint` where the first read `loop_back`), the graph legitimately diverges, keys shift from the divergence point onwards, and the waves after it re-run. That is the correct outcome — the plan really is different — but it means `--resume` is "continue this run", not "reproduce this run". **Never resubmit work that may still be running.** A wave is recorded `running` the moment it is submitted, so a driver that dies mid-poll leaves a job that may still be billing. On `--resume`, such a record is *reconciled* rather than replaced: the recorded job is queried and then adopted if it already succeeded, attached to (kept polling) if it is still alive, or replaced only once it is observably terminal-failed. If its state cannot be determined at all, the run fails instead of launching a second copy. `--resume` without `config.bucket` (i.e. without a ledger) fails fast for the same reason. **Never leave a job running after an abort.** Every wave failure path — workflow error, unexpected tooling error, `KeyboardInterrupt` — goes through `_abort_wave`, which cancels the managed job (by id, or by cluster name when the submit reported no id) before recording the failure. Status queries are treated as unreliable rather than fatal: up to `MAX_CONSECUTIVE_STATUS_ERRORS` transient `sky jobs queue` failures are tolerated (and recorded in the ledger) because a failed *query* says nothing about the job, while the wave deadline still applies. A submit that reports no job id is rejected outright, since polling would otherwise sit on `UNKNOWN` for the whole deadline while the job ran. ### Failure, retry, cancellation * A wave whose managed job reaches a terminal failure is retried up to `--retries` times with a backoff; each attempt is a separate ledger entry (`attempt: 1, 2, ...`) so a flaky node is visible rather than hidden. * A wave that does not reach a terminal state within `--max-wait-seconds` is cancelled (job + cluster) unless `--no-cancel-on-timeout`, then fails the run — no leaked clusters. * A failed batch of a parallel group stops the group; the remaining members are recorded as skipped, and the run fails with the **root cause** (the first failure), not the cascade. ### Local `--execute` vs runtime: one intentional difference With the local executor a failed member of a `parallel:` group does **not** stop the other members (they run in declared order, each recorded independently); with the runtime executor a failed *batch* stops the group and the remaining members are recorded as skipped. Both report the same root cause, and both fail the run. The runtime behaviour is deliberate: continuing to launch cloud jobs whose barrier can no longer be satisfied only spends money. ### Trigger / watch pattern `trigger:` is polled **driver-side** (`s3_trigger_waiter`): list the prefix until `minObjects` keys exist, record the watermark in the ledger, then submit the state's wave. The wait is bounded twice — by `maxPolls` when the spec sets one, and *always* by the run's `max_wait_seconds`, so the default `maxPolls: 0` cannot mean "wait forever". No cloud job is burned to wait, and a resumed run skips a watch it already satisfied. `sim_to_real_trigger`'s SkyPilot-specific watcher is unchanged; this is the npa.workflow-native analogue. ### CLI surface ``` npa workbench workflow submit --runtime \ [--resume] [--poll-seconds N] [--max-wait-seconds N] \ [--retries N] [--max-concurrency N] [--no-cancel-on-timeout] \ [--var k=v] [--secret-env NAME] [--registry ...] [--output-format json] npa workbench workflow plan-spec --waves [--json] ``` `--runtime` runs the driver in the foreground and prints a JSON summary (status, waves with job ids and timelines, decisions, run prefix, runtime-state URI); without it, `submit` behaves exactly as before. A detached driver (submit and poll from a supervisor process) is deliberately out of scope here — it needs a process supervisor story of its own. --- ## 5. Backwards compatibility | Guarantee | How it is held | | --- | --- | | Existing specs render identically | `build_plan` unchanged for specs without `parallel:`; `render_skypilot_yaml` untouched | | Serial-only renderer guard preserved | Parallel uses a *different* function; `test_render_rejects_parallel_execution` passes verbatim | | Plan-only guardrail ("`execution: serial`" in every rendered twin) | Parallel groups flatten for `--plan-only` | | `--assume-decision` remains the offline path | Untouched in `build_plan` / `submit`; the runtime tier only uses it as a fallback when a gate artifact cannot be read | | Local `--execute` behaviour | `step_executor=None` → module-level `_execute_step` (monkeypatch-compatible) | | `RunManifest` schema | Unchanged; runtime state is a new, separate document | | Scheduler seam | Both renderers build docs through `build_skypilot_task_doc` → `build_scheduler_task` | ### Deviations that were necessary (and why) 1. **`SubmitLiveCase` gained fields** (`runtime`, `config_vars`, `expected_parallel_tasks`, `image_tool`, `max_wait_seconds`) and the one-shot live test now skips `runtime=True` cases. An `expected_execution` field was considered and deliberately **not** added: because `--plan-only` always renders the flattened serial plan (D6), the plan-only guardrail's `"execution: serial"` assertion holds for parallel specs too, so the field would never carry a value other than `"serial"`. Reason: submitting a parallel sweep through the one-shot path would render the flattened serial plan — valid, but it proves nothing about concurrency and would burn four GPU-hours running a sweep serially. The plan-only matrix still covers every spec including the new ones. 2. **`workbench.insights.ingest_run` is not a universal barrier.** The first live fan-out run proved it: the insights ingester only recognises dataset, scenario and decision artifacts, so a caption fan-out made it fail with "no known manifest/report schemas found". The fan-out and sweep specs now use purpose-built join stages (`fanout_join.join_shards`, `rl_sweep.select_best`) that additionally *verify* every predecessor's artifact exists — a better demonstration of a barrier anyway. The gate-loop spec still uses `insights.ingest_run`, because `decision.json` *is* an ingestible artifact. 3. **The Isaac sweep uses `run.shell` + a real module, not `workbench.rl.policy_train`.** That toolRef's argv (`--learning-rate`, `--batch-size`, `--input-path`) does not match the actual `npa workbench isaac-lab train` CLI (`--override`, `--num-envs`, `--steps`, `--output-path`), and that CLI is a *launcher* (it provisions a VM/serverless job), so calling it inside a SkyPilot task would nest infrastructure. The in-pod contract is the upstream RSL-RL training script — exactly what the SkyPilot template this spec ports does. Logic lives in `npa/src/npa/workflows/rl_sweep.py` with unit tests, per the repo's "put testable logic in a real module" rule. Fixing the pre-existing `workbench.rl.policy_train` mismatch is out of scope for this change. 4. **Root-cause error reporting for parallel groups** — `_execute_state_machine` now raises the *first* failure of a group rather than the last, so the cascade of "skipped after batch N failed" records cannot mask the real error. 5. **The new fields are validated in Python, not by the JSON Schema.** The shipped `npa.workflow.v0.0.1.schema.json` gained entries for `parallel`, `maxConcurrency`, `params` and `trigger`, but the hand-rolled walker in `schema_validation.py` does **not** resolve `$ref` / `$defs` / `additionalProperties`, so `states..*` bodies have never actually been schema-enforced (pre-existing, unrelated to this change). Writing the tests for it exposed the consequence for the new fields — `parallel: shard-a` was iterated character by character and reported "duplicate parallel member" — so `_parse_state` now rejects a non-list `parallel` and non-string members with actionable messages, alongside the existing `params`/`trigger` mapping checks. Teaching the walker to resolve `$ref` would retroactively tighten validation for every shipped spec and is deliberately left out of this change. --- ## 6. Testing model * **Unit (mocked, default suite).** `test_parallel_waves.py` (spec validation, params overlay, wave folding/batching, JobGroup rendering, serial guard), `test_runtime_orchestrator.py` (early exit, full budget, `goto` branch, plan/runtime equivalence, JobGroup batching, barrier stop on failure, retry, retry exhaustion, timeout→cancel, resume/replay, trigger poll and give-up, ledger contents), `test_rl_sweep.py`, `test_fanout_join.py`. Every dependency (submitter, status, timeline, canceller, sleeper, clock, S3) is injected — no infrastructure, no sleeping. * **Live (separate tier, env-gated).** `NPA_E2E_NPA_WORKFLOW_RUNTIME=1` plus the existing `NPA_INTEGRATION_E2E` / `NPA_E2E_NPA_WORKFLOW_SUBMIT` gates. Two live tests: terminal success + concurrency + barrier for `runtime=True` matrix cases, and the gate-loop early-exit vs full-budget pair. Runner: `scripts/npa-workflow-runtime-live-e2e.sh`. Live results, run ids, job timelines and decision artifacts are in [`EVIDENCE.md`](EVIDENCE.md). --- ## 7. Deliberately out of scope * Multi-step branches inside a `parallel:` group (each member is a leaf today; a branch would need one managed job per branch instead of one JobGroup). * `gang` scheduling and `foreach` templating. * A detached/daemonized runtime driver and a unified `workflow status` view for runtime runs (the ledger JSON is the source of truth today). --- # DESIGN — retiring the raw SkyPilot task catalog `npa/src/npa/workflows/skypilot/` shipped 36 raw SkyPilot task templates alongside the `npa.workflow/v0.0.1` catalog. They are being retired so that a spec is the **only** workflow authoring surface. This section records the decisions that work depends on; the live runs backing each one are in [`EVIDENCE.md`](EVIDENCE.md). SkyPilot itself is **not** being retired: it remains the execution engine, and `detect_submit_format()` still accepts a customer's own SkyPilot YAML. What goes away is the shipped *catalog* of templates. ## R1. What "equivalent twin" actually means (the finding that shaped everything) 13 specs declared `skypilotTwin:`, which reads as "this spec replaces that template". Two of those twins turned out **not** to be equivalent, and only a live run could show it: | Twin | Looked fine | Failed live because | | --- | --- | --- | | `sonic-export` | validates, plans, renders | the tool only accepted **local** paths; the template did the S3 download/upload in ~60 lines of inline bash. `toolRef` argv has no such escape hatch, so the spec handed `s3://.../checkpoint.pt` to `Path.exists()` | | `sonic-eval` / `sonic-export-eval` | same | `sonic eval` could *write* its result to S3 but only ever *read* local files | The general rule this exposes: **a template that does I/O staging in bash cannot be replaced by a `toolRef` unless the tool itself speaks object storage.** The fix is therefore in the tool, not the spec: `export_onnx` and `evaluate_onnx_policy` now stage `s3://` inputs and publish `s3://` outputs through the same `StorageClient` the eval result path already used. Local behaviour is unchanged — the historic export body moved to `_export_onnx_local` and is called directly when nothing is an object URI, so a local run never constructs a storage client. Two smaller traps fell out of the same run, both now covered offline: * An ONNX is a **pair** of files when `torch.onnx.export` spills tensors to `.onnx.data`, and onnxruntime resolves that path *relative to the model*. Staging must move the sibling too. * The exporter resolves shapes from the policy when no `--obs-spec` is given, so a fixture policy must expose `obs_dim` / `action_dim`. ## R2. Three-tier contract: the third tier moves onto npa.workflow `test_three_tier_contract.py` bound 11 CLI/SDK capabilities to a **SkyPilot YAML `envs` block**. Deleting those YAMLs would delete the guardrail, which is not allowed, so the tier was migrated rather than dropped: | Tier | Before | After | | --- | --- | --- | | 1 | Typer callback parameter + flag | unchanged | | 2 | SDK signature parameter | unchanged | | 3 | `envs:` key declared **and referenced** in the YAML | the shipped spec loads, declares the `toolRef`, and the toolRef's `argv_template` passes the parameter's **real CLI flag** | *Sharper*, because the old check could not tell whether the flag existed: a YAML could declare `TRAIN_LEARNING_RATE` while the tool was invoked with a nonexistent `--learning-rate`. *Narrower*, because a catalog argv exposes fewer knobs than a YAML `envs` block did. Rather than hide that, every contract pins `spec_gap` — the CLI parameters a spec author cannot set today — and `SPEC_GAP_REASONS` classifies each one: * `boolean` — the option is a paired/flag-only boolean (`--headless/--no-headless`). A v0.0.1 argv template is a fixed list with no conditional rendering, so it cannot express one. Closing these needs a spec-level conditional-argv feature. * `infra` — the option selects infrastructure (`--image`, `--gpu-type`). The engine already owns image/accelerator selection via `resources.`; passing it again inside the pod would nest infrastructure choices. * `knob` — a plain value the argv simply does not pass yet. These are the ones worth closing, tool by tool, with a live run each. The test asserts the computed gap equals the declared gap **exactly**, so the gap is visible in review, cannot grow silently, and shrinks only by a deliberate edit. `workflow/trigger/{run,watch}` keep the legacy YAML tier until the sim-to-real port gives them a spec twin; `LEGACY_YAML_TIER` pins that set so it can only shrink. ## R3. A catalog-wide argv guardrail replaces what the YAML tier used to imply Nothing checked that a `toolRef`'s argv flags exist on the command they are handed to, so a template could validate, plan and render and still die with `No such option` in the pod. `guardrails/tool_catalog_argv.py` walks the `npa` Typer tree along an argv template and reports flags the resolved command rejects; `test_no_tool_ref_argv_passes_a_flag_its_cli_rejects` asserts **zero** drift across the whole catalog. It immediately found two: `workbench.rl.policy_train` (`--learning-rate`, `--batch-size`, `--input-path` — the half `DESIGN §7` already knew about) and `workbench.rl.evaluate_policy` (`--episodes`, previously unknown). Both are fixed rather than allowlisted, so three specs that could never have run now can. Trainer hyper-parameters go through Isaac Lab's repeatable Hydra `--override KEY=VALUE`, which is what its CLI documents for exactly this. ## R4. Per-toolRef npa extras, instead of requiring a vendor image A stage on SkyPilot's default image gets only the base `npa` install, so `npa workbench sonic export` failed with its own advice: *"requires torch ... or use the npa[sonic] extra"*. `TOOL_REF_PIP_EXTRAS` maps a toolRef prefix to an npa extra and `render_setup_for_tool` installs it **from the same source tree it installed npa from** (recorded in `/tmp/npa-src-root`). This is the pattern the renderer already used to install vLLM for self-hosted `vlm_eval`. Consequence: the SONIC specs need no vendor image at all. The alternative — requiring every workbench image to be SkyPilot-hostable before its spec can run — is still worth having (see R5) but is no longer a *blocker* for a spec to be live. Rendering constraint worth knowing: the setup script must not contain `${var}`. `assert_no_unresolved_placeholders` rejects any braced expansion, because SkyPilot would leave it literal — the first live submit failed in 1.15 s on exactly that. Compose paths with `printf` instead. ## R5. Image hostability is a repo deliverable, not operator lore `EVIDENCE §8.5` listed "make the shipped Isaac image SkyPilot-hostable" as a follow-up, and the SONIC image had the same defect. `SKYPILOT_HOSTED_IMAGES` in the image guardrail now covers `sonic` as well as `isaac-lab`, the SONIC Dockerfile carries the same four ingredients, and each has a `Dockerfile.k8s-prereqs` so an operator can repair an already-published tag in-cluster. The list grows as the catalog shrinks: **once a tool's only workflow surface is a spec, its image must be able to host a SkyPilot task.** ## R6. Live fixtures that do not depend on gated weights `sonic export` needs a loadable torch policy checkpoint, and the repo deliberately does not vendor NVIDIA's gated `nvidia/GEAR-SONIC` weights, so those twins could only ever be covered plan-only. `npa.workflows.sonic_fixture` builds a small deterministic MLP policy the shipped exporter can trace, and `scripts/stage-sonic-export-fixture.sh` runs it **in-cluster** through a ConfigMap — no torch download touches the dev VM, which routinely sits above 90 % disk. The harness seeds from `NPA_E2E_SONIC_CHECKPOINT_SRC` / `NPA_E2E_SONIC_ONNX_SRC`, mirroring the existing `NPA_E2E_SONIC_MOTION_SRC` hook. ## R7. A machine-checked retirement tally `test_skypilot_catalog_retirement.py` pins the exact remaining templates with a reason each. A new raw template cannot appear without a deliberate edit, and each retirement shows the count dropping in a one-line diff instead of a prose claim in a PR body. Deleting an entry is the **last** step: the twin must already have a live run id in `EVIDENCE.md`. ## R8. Harness parsing must not hide a live result `json.loads(result.output)` in the live harness assumed the CLI writes nothing but JSON. `CliRunner` merges stderr into `output` on this click version, and the submit path legitimately writes `Hint: consider --secret-env NGC_API_KEY` first — so a **successful** submit was reported as `JSONDecodeError: Expecting value: line 1 column 1`, and the real failure downstream stayed invisible for two runs. Both parsers now slice the JSON document out of the stream, and a matrix guardrail asserts every case declares the secrets its plan hints at so the advisory line is not emitted at all. ## R9. `num_nodes`: multi-node stages on the resource profile A spec could not ask for a multi-node block at all — that capability lived only in `npa burst submit --nodes` (`burst.core.BurstSpec.num_nodes`), i.e. outside the workflow surface. Closing it needed one decision: **where does the field go?** SkyPilot places `num_nodes` at the **task** level, a sibling of `resources` (`sky/utils/schemas.py`; `burst.core.build_task_spec` does the same). A spec has no task-level surface — per-stage shape lives on the resource profile a state selects — so the field is declared there and the renderer lifts it back out: ```yaml resources: gang: cloud: kubernetes cpus: 2 memory: 4Gi num_nodes: 2 # -> task-level `num_nodes: 2`, NOT inside `resources` ``` `normalize_resources()` deliberately does not pass it through; inside a `resources` block it would be invalid. `build_scheduler_task` carries it so the portable seam stays complete for a non-SkyPilot backend. **Additive by construction.** A profile without `num_nodes` (or with `1`) emits no key, so every previously rendered document is byte-identical — asserted, not assumed. `validate_spec` rejects a non-integer, a bool, `< 1`, and `> MAX_PROFILE_NODES` (32): a gang block that large would sit `PENDING` on a shared cluster rather than fail fast, which is the worst failure mode to debug. Rejected: a `gang:` state field (DESIGN §7 lists `gang` as out of scope, and a state that selects a 2-node profile already *is* the gang); and a submit-time `--num-nodes` flag (node count is a property of the stage, not of one invocation — unlike `--max-concurrency`, which is a cost cap). The live proof is `npa-workflows/multi-node-probe.yaml`: the gang stage writes one `rank-.json` per node and a single-node stage fails unless it finds one report per expected rank **from distinct hostnames** — so a gang that collapsed onto one node cannot pass. See `EVIDENCE.md` §R14. ## R10. BYOF resource profiles are not workflow templates `isaac-lab-rl-train*.yaml` and `byof-*-smoke.yaml` describe a pod shape (accelerator, cpu/memory floors, image placeholder, smoke command) that the BYOF runner substitutes into — one task, no orchestration. The workflow surface for them is already the spec `byof.yaml`, whose `workbench.byof.repo` toolRef passes one through `--yaml {{config.resource_profile_yaml}}`. So they were **relocated** to `npa/src/npa/workflows/byof/profiles/` rather than ported or deleted, joining `byof-solution-smoke-rtxpro-gpu.yaml` and `skypilot-kubernetes-rtxpro.yaml` which were already there — a move `byof/live.py`'s own comment had anticipated ("so the SkyPilot reference catalog can be deprecated/removed without breaking BYOF live runs"). Rejected: rewriting `run_isaac_lab_rl.py` / `run_byof_datagen.py` / `run_byof_container_verify.py` onto `prepare_npa_workflow_for_submit`. They carry render-only modes, output-root rewriting and BYOF image plumbing the engine does not model, so the port would risk the BYOF onboarding live path for no gain in the *workflow* surface. `test_byof_profiles.py` keeps the boundary honest: the file set is pinned, each profile must be a **single** task (chaining stages means it belongs in a spec), `live.py`'s constants must resolve inside `profiles/`, and no runner may resolve a path under the retiring catalog. ## R11. A per-`toolRef` **run** preamble, not just a setup hook R4 added a per-`toolRef` `setup:` hook so a stage could install an npa extra. The self-hosted VLM backend needed something the setup hook structurally cannot do: **start a process that must still be alive while the stage's command runs.** SkyPilot executes `setup:` and `run:` as separate shells, so anything backgrounded in setup is gone by the time the command starts. So `render_run_preamble_for_tool(tool_ref, config)` mirrors `render_setup_for_tool` and is prepended to the `run:` script — after the interpreter shim, so the server uses the same `python3` the command will. Why this belongs in the renderer rather than in the tool: the tool's job is to *call* an OpenAI-compatible endpoint, and it should not care whether that endpoint is hosted, remote or in-pod. Where the endpoint comes from is a property of the **execution environment**, which is exactly what the renderer owns. Rejected alternatives: * *Bake vLLM serving into `vlm-eval run`.* Couples a scoring tool to a serving stack, and would start a server per invocation — the loop would pay it three times. * *Require a prebuilt serving image.* That is what the template did (`NPA_VLM_IMAGE`), and it is precisely the operator lore R5 argues against: the live proof runs on SkyPilot's default image. * *A generic `sidecar:` field in the spec schema.* Real, but much larger, and it would put serving mechanics into the authoring surface. Revisit if a second tool needs a companion process. The preamble also fixes the environment's shape rather than demanding a shape: `ninja` and a CUDA compiler both come from pip wheels vLLM already depends on, and the JIT-dependent sampler falls back to pure PyTorch. A workflow engine must not require a build toolchain in a task image. ## R12. Retiring a template may require *adding* a capability The plan (D3) assumed `sim-to-real-loop.yaml` could retire because the staged sim2real engine covers sim-to-real. Reading the template disproved that. It owns two things nothing else has: per-rollout scoring of a rollout **set** (`vlm-eval run` scores one rollout — it discovers frames recursively, so a prefix of many rollouts blends into a single score), and the `task_success_report.json` aggregate that the cookbook documents as the gate. The capability existed twice and was reachable from neither the CLI nor a spec: as `jq` inside the template, and as Python inside a gated GPU test. That is the signature of a missing tool feature, and it generalises: **a template's bash is either accidental glue or an unimplemented capability, and only reading it tells you which.** The same reading is what produced the sonic S3 staging work (R1) and the two `detection-training` gaps. Consequence for the remaining work: a retirement is not a delete plus a reference sweep. The honest order is (1) read the template, (2) name every behaviour it owns, (3) put the ones the tool lacks into the tool, (4) verify live, (5) delete. Where step 3 is not finished, the template stays and the tally records what is missing — which is why `bdd100k-pipeline.yaml` survives with three named gaps rather than a half-ported runner. ## R13. Two failure classes worth encoding as guardrails, not fixes Both were found by reading twins rather than by a test, and both are *classes*: **A spec may not name a path inside the repo checkout.** `vlm-eval-benchmark.yaml` passed `--dataset npa/src/npa/workbench/vlm_eval/fixtures/.../benchmark.json`. It resolves on a laptop and never in a pod. The guardrail fires only when a resolved argv value is a *relative* path that also *exists in this repo* — deliberately narrow, because such a value is always a mistake and the check then cannot produce false positives. It immediately found five more specs, which is the argument for a guardrail over five fixes. **A same-named spec is not automatically a twin.** `tokenfactory-rollout-judge` has a spec and a template that share a name and do different things (the template's GPU stage *produces* what its judge scores; the spec's first stage is an unrelated reasoner). Equivalence cannot be inferred from the file name, and the retirement tally now carries the *reason* a template survives rather than a status word — so "twin live-verified" cannot stand in for "I compared them". ## R14. Executing a *plan step* is a better validation than executing a template's bash `bdd100k-pipeline.yaml` shipped with something none of the other templates had: a `--mock-endpoints` mode in its runner that stood up in-process LanceDB and detection-training servers, ran every task's `run:` bash against them, and asserted the exact request sequence. It was the closest thing in the repo to an end-to-end test of a pipeline without infrastructure. Porting the runner onto the spec forced a choice about that mode, because a spec has no bash to run. Executing **each plan step's resolved argv** turns out to be strictly better: * it is *exactly* what the engine will run in a pod, so a passing mock run is evidence about the real execution path rather than about a parallel bash implementation; * it exercises the toolRef → CLI boundary, which is where this change kept finding defects; * it removes the drift risk of a second implementation of the pipeline. It proved itself immediately: the first spec-driven mock run failed on `No such option '--table'` from `lancedb create-mv` — a toolRef defect that had been invisible because the flag audit skipped `bash -c` entries — and then on a declared artifact URI missing a path separator. Neither is visible in a diff, and neither needed a GPU, a cluster or a credential to find. The generalisation for the remaining ports: **where a template's runner has a no-infrastructure validation mode, port that mode to plan steps before porting the submit path.** It is the cheapest place to learn that a spec is wrong, and for a pipeline whose live run is blocked on a missing service it is the only honest offline proof available. The check should assert *order*, not just counts. `--wait` and `--discover-checkpoint` are both invisible to a POST-count assertion: what distinguishes them is that a `GET /status` follows every `POST /train`, and a `GET /runs` precedes every `POST /eval`.