sim2real-operate git:20261029.fe9128eA

Immutable. This exact content never changes and is served at /api/v1/blob/8f1047883426b33f.

---
name: sim2real-operate
description: Operate the compositional 14-stage Sim2Real npa.workflow on Kubernetes through the standard SkyPilot runtime, durable S3 resume ledger, real component images, and artifact audit.
---

# Operate compositional Sim2Real

Use the one canonical spec:

`workflows/main/sim2real.yaml`

It is `npa.workflow/v0.0.1` and declares `executionMode: runtime`. Use
`npa workbench workflow ... --runtime`;
there is no direct-Kubernetes Sim2Real submit path. The old materializer and
`k8s_submit` implementation have been removed; the retained CLI command exits
with an actionable migration to this canonical spec.

## Preflight

1. Validate tenant/project/region, bucket, registry, Kubernetes context, Ready
   RT-core nodes, GPU concurrency that fits actual capacity, and the read-only Isaac cache PVC.
2. Require registry-qualified immutable digests for controller, Transfer,
   EnvGen, Isaac, and viewer images. Confirm each image attests the exact
   source SHA; never use source overlays or best-effort bootstrap.
   Supply the five canonical image config variables and their common
   `source_sha`. The historical September 4 bundle predates MiniMax/v5 and
   concurrent receipt support; select a newly qualified coherent set. Legacy
   Genesis, LeRobot VLM-RL, and reference-policy images are not stages in this
   graph. Stage 8 calls Token Factory from the controller image; it needs no
   separate Reason image or GPU service.
3. Validate the task-aligned seed manifest, HF/NGC access, S3 read/write, image
   pulls, and primary/side/overhead capture before a full run.
   Inspect the primary frames selected for hosted evaluation for object and
   end-effector visibility, including intermediate action times. Stage 8 sees
   only primary images; clear secondary views and valid temporal bindings cannot
   make an occluded primary image useful for task-specific visual credit.
   Rebuild and regenerate rollouts when this visibility check fails; never
   relabel an existing view or rewrite its pose metadata in place.
   Provision the selected RWX cache claim explicitly with `npa workbench
   sim2real cache provision --name <claim> --context <context> --namespace
   <namespace>`. The canonical workflow's CPU-only
   `stage-00-isaac-cache-warm` then runs the selected immutable Isaac image's
   real bootstrap, requires `ready=yes`, and publishes a source-bound
   `cache/isaac-warm.json` receipt before any GPU state consumes the claim
   read-only/offline. The cache stamp includes the bootstrap script, so
   unchanged wheel pins alone do not prove that a rebuilt image can reuse it.
   Do not apply an out-of-band warm Job; preserve older versioned cache trees.
   Inspect first and last frames from all three cameras for the robot and task;
   successful PNG decoding alone does not prove useful rendered content. Serialized
   camera poses remain WXYZ, while Isaac Lab 3 sensor offsets require XYZW. Use
   `camera_rotation_for_isaac_lab` at each sensor boundary. Reject blank or
   misdirected capture before hosted evaluation or PPO; rebuild the affected image
   and regenerate the invalid rollouts rather than replaying them as complete.
   Hosted Stage 8 must bind each action to selected primary-frame metadata by
   exact `sim_step`. Sample order and a final context frame are not action
   timestamps. The v5 evaluator contract requires null camera references, zero
   confidence, neutral tags, and explicit insufficient evidence for unsampled
   actions. Stage 9 must reject older or inconsistent bindings before PPO; do not
   rewrite archived critiques to attach a different frame. Unsupported visual
   events contribute no auxiliary reward, corrective action, or PPO tag counts.
   Generation must enforce those same bindings with one ordered JSON Schema
   `prefixItems` entry per action, fixed step/camera fields, and neutral values
   for unsupported events. Verify actual endpoint schema support with real
   rollout input; retain strict parser rejection and never repair model output.
   Default `evaluation_max_frames=0` sends every declared primary frame;
   positive values retain deterministic sampling and neutral unsupported
   actions. `evaluation_concurrency` defaults to 8 and must be positive.
   Verified per-rollout S3 receipts bind source, endpoint, model, threshold,
   actions, frame metadata, and frame bytes. Resume reuses only exact matches;
   malformed receipts or storage denial fail before another paid request.
   Transfer seed frames must use one strict numbered family: canonical
   `camera-<N>.png`, or the seeder-compatible fallback `frame-<N>.png` when no
   camera family exists. Unrelated PNG objects are never admitted as frames.
   Isaac vector-environment resets consume curated training scenarios through
   one monotonic round-robin cursor. Partial reset batches must cover the split
   tail before wrapping; per-environment reset counters can silently strand
   scenarios when the split is larger than the vector-environment count.
   Large cold images may need `--image-bootstrap-timeout-seconds 0`; this removes
   only the observation deadline and still requires immutable digest, registry
   authorization, source attestation, runtime capabilities, and verified probe
   cleanup. Repeat-safe runs reuse only digest-bound compatible evidence.
4. Run `validate-spec`, `plan-spec --waves`, scheduler-plan, and submit plan-only
   on the same canonical file.

## Submit and resume

Compute-only Cosmos3 and EnvGen stages may select `B200:1` through
`cosmos3_accelerator` and `envgen_accelerator`. Isaac stages stay on
`RTXPRO6000:1`. Use available reserved capacity first; an operator-authorized
preemptible pool is a placement fallback. Set `gpu_concurrency` to the actual
schedulable GPU count before planning.

Before provisioning or submitting an Isaac state, load
`skills/atomic/third-party-eula-preflight/SKILL.md`. Isaac acceptance defaults on
for non-interactive submissions; pass `--no-accept-eula` to opt out. An opted-out
run fails before work is created. Optional privacy and telemetry remain disabled.

Submit with `--runtime --resume`. Pass tenant-specific data only through
`--var`, isolated config, and secret envs. For a no-deadline run pass
`--max-wait-seconds 0`; the runtime still records wave/job status in
`<run-root>/npa-workflow/runtime.json`.

The runtime driver stays on the submitting host. Run it on an always-on
operator VM for long jobs; a laptop entering sleep interrupts network access
even when GPU jobs continue on the cluster. After a driver interruption, verify
recorded jobs before using `--resume-run` for the same run. Never start two
runtime drivers for one run concurrently. Status includes stages discovered in
the runtime ledger when the initial manifest has no steps. Successful recorded
jobs alone do not prove that the remaining workflow graph completed.

The graph owns every stage Job. Isaac rollout/PPO/eval execute their proven
payload inside their already admitted SkyPilot GPU task and must report
`npa_workflow_skypilot_task`; a hidden sibling Job is a contract failure.

Use a deliberate controller restart after Stage 8 and `--resume` to prove the
Stage 8→9 barrier. Also restart during Stage 14 in the integration ladder. The
runtime must adopt/replay complete waves from declared S3 outputs and resubmit
only incomplete work.

## Diagnostic held-out rerun

`npa workbench sim2real rerun heldout-only --run-id <run-id>` is diagnostic.
With publication enabled it writes the new report and render tree only under
`eval/gold-heldout/outer-XX/attempts/<attempt>/`; it never replaces the
canonical Stage 10 report or render aliases. The command reports those immutable
attempt URIs. Use `--no-publish` for local-only diagnosis, and use the canonical
workflow/resume path when Stage 10 authority must advance.

## Audit

Canonical archive regeneration with upload must bind two distinct authorities:
the immutable input report/history and the current writer. Preserve the retained
report source, upstream records and input byte hashes. Bind the replay task's
actual digest-attested source, image and job to its new output generation/hashes;
these may differ from the input producer. Never inherit the original job/device
claim or assign an old source SHA to a newer image. Without qualified current
task provenance, use local preview only: it may emit recordings but must leave
canonical report and ComponentRecord authority untouched.

Require exactly 14 canonical ComponentRecords. Stages 1–11, 13, and 14 are
`WORKS`; Stage 12 alone is `SEAM`. For GPU stages verify workflow Job identity,
immutable digest, source SHA, GPU product, and explicit S3 inputs/outputs.
Verify train/validation/gold digest disjointness, validation-only checkpoint
selection, exact checkpoint SHA/size loaded by gold, strict 5 cm stable
placement, bounded non-degenerate temporal signals, and explicit gold render
lineage.

Download and independently decode non-empty `reports/sim2real.rrd` and
`reports/sim2real.mcap`. Confirm 10 FPS (or configured FPS) timestamps,
primary/side/overhead footage, progress/policy/evaluation evidence, and
checkpoint accessibility. Pipeline completion does not imply policy efficacy;
report measured strict success without weakening the threshold.