git:20260906.525ee24 to git:20260912.a817706

37 added, 0 removed. Audit A to A.

---

name: sim2real-operate

description: Operate the compositional 14-stage Sim2Real npa.workflow on Kubernetes through the standard SkyPilot runtime, durable S3 resume ledger, real component images, and artifact audit.

---

# Operate compositional Sim2Real

Use the one canonical spec:

`workflows/main/sim2real.yaml`

It is `npa.workflow/v0.0.1`. Always use `npa workbench workflow ... --runtime`;

there is no direct-Kubernetes Sim2Real submit path. The old materializer and

`k8s_submit` implementation have been removed; the retained CLI command exits

with an actionable migration to this canonical spec.

## Preflight

1. Validate tenant/project/region, bucket, registry, Kubernetes context, Ready

RT-core nodes, bounded GPU concurrency, and the read-only Isaac cache PVC.

2. Require registry-qualified immutable digests for controller, Transfer,

EnvGen, Reason, Isaac, and viewer images. Confirm each image attests the exact

source SHA; never use source overlays or best-effort bootstrap.

3. Validate the task-aligned seed manifest, HF/NGC access, S3 read/write, image

pulls, and primary/side/overhead capture before a full run.

+ Inspect the primary frames selected for hosted evaluation for object and

+ end-effector visibility, including intermediate action times. Stage 8 sees

+ only primary images; clear secondary views and valid temporal bindings cannot

+ make an occluded primary image useful for task-specific visual credit.

+ Rebuild and regenerate rollouts when this visibility check fails; never

+ relabel an existing view or rewrite its pose metadata in place.

+ Run the selected Isaac image's `isaac-bootstrap status` against the cache

+ PVC and require `ready=yes` for its reported `expected_tree` before using

+ read-only/offline mode. The cache stamp includes the bootstrap script, so

+ unchanged wheel pins alone do not prove that a rebuilt image can reuse it.

+ If absent, run `npa/docker/workbench/common/warm-isaac-cache.yaml` with that

+ exact image digest to create its versioned tree; preserve older cache trees.

+ Inspect first and last frames from all three cameras for the robot and task;

+ successful PNG decoding alone does not prove useful rendered content. Serialized

+ camera poses remain WXYZ, while Isaac Lab 3 sensor offsets require XYZW. Use

+ `camera_rotation_for_isaac_lab` at each sensor boundary. Reject blank or

+ misdirected capture before hosted evaluation or PPO; rebuild the affected image

+ and regenerate the invalid rollouts rather than replaying them as complete.

+ Hosted Stage 8 must bind each action to selected primary-frame metadata by

+ exact `sim_step`. Sample order and a final context frame are not action

+ timestamps. The v4 evaluator contract requires null camera references, zero

+ confidence, neutral tags, and explicit insufficient evidence for unsampled

+ actions. Stage 9 must reject older or inconsistent bindings before PPO; do not

+ rewrite archived critiques to attach a different frame. Unsupported visual

+ events contribute no auxiliary reward, corrective action, or PPO tag counts.

+ Generation must enforce those same bindings with one ordered JSON Schema

+ `prefixItems` entry per action, fixed step/camera fields, and neutral values

+ for unsupported events. Verify actual endpoint schema support with real

+ rollout input; retain strict parser rejection and never repair model output.

Transfer seed frames must use one strict numbered family: canonical

`camera-<N>.png`, or the seeder-compatible fallback `frame-<N>.png` when no

camera family exists. Unrelated PNG objects are never admitted as frames.

Isaac vector-environment resets consume curated training scenarios through

one monotonic round-robin cursor. Partial reset batches must cover the split

tail before wrapping; per-environment reset counters can silently strand

scenarios when the split is larger than the vector-environment count.

Large cold images may need `--image-bootstrap-timeout-seconds 0`; this removes

only the observation deadline and still requires immutable digest, registry

authorization, source attestation, runtime capabilities, and verified probe

cleanup. Repeat-safe runs reuse only digest-bound compatible evidence.

4. Run `validate-spec`, `plan-spec --waves`, scheduler-plan, and submit plan-only

on the same canonical file.

## Submit and resume

Before provisioning or submitting an Isaac state, load

`skills/atomic/third-party-eula-preflight/SKILL.md`. Isaac acceptance defaults on

for non-interactive submissions; pass `--no-accept-eula` to opt out. An opted-out

run fails before work is created. Optional privacy and telemetry remain disabled.

Submit with `--runtime --resume`. Pass tenant-specific data only through

`--var`, isolated config, and secret envs. For a no-deadline run pass

`--max-wait-seconds 0`; the runtime still records wave/job status in

`<run-root>/npa-workflow/runtime.json`.

+ The runtime driver stays on the submitting host. Run it on an always-on

+ operator VM for long jobs; a laptop entering sleep interrupts network access

+ even when GPU jobs continue on the cluster. After a driver interruption, verify

+ recorded jobs before using `--resume-run` for the same run. Never start two

+ runtime drivers for one run concurrently. Status includes stages discovered in

+ the runtime ledger when the initial manifest has no steps. Successful recorded

+ jobs alone do not prove that the remaining workflow graph completed.

The graph owns every stage Job. Isaac rollout/PPO/eval execute their proven

payload inside their already admitted SkyPilot GPU task and must report

`npa_workflow_skypilot_task`; a hidden sibling Job is a contract failure.

Use a deliberate controller restart after Stage 8 and `--resume` to prove the

Stage 8→9 barrier. Also restart during Stage 14 in the integration ladder. The

runtime must adopt/replay complete waves from declared S3 outputs and resubmit

only incomplete work.

## Audit

Require exactly 14 canonical ComponentRecords. Stages 1–11, 13, and 14 are

`WORKS`; Stage 12 alone is `SEAM`. For GPU stages verify workflow Job identity,

immutable digest, source SHA, GPU product, and explicit S3 inputs/outputs.

Verify train/validation/gold digest disjointness, validation-only checkpoint

selection, exact checkpoint SHA/size loaded by gold, strict 5 cm stable

placement, bounded non-degenerate temporal signals, and explicit gold render

lineage.

Download and independently decode non-empty `reports/sim2real.rrd` and

`reports/sim2real.mcap`. Confirm 10 FPS (or configured FPS) timestamps,

primary/side/overhead footage, progress/policy/evaluation evidence, and

checkpoint accessibility. Pipeline completion does not imply policy efficacy;

report measured strict success without weakening the threshold.