---
name: workbench-reference-workflows
description: Use when working on NPA reference workflow specs, runner scripts, cookbooks, customer-adaptable pipeline implementations, or the guarded examples that are not workflow authoring surfaces.
---

# Workbench Reference Workflows

> The supported, customer-facing catalog and source of truth is the `npa.workflow` spec set under
> `workflows/`. The old raw SkyPilot task catalog has
> no remaining templates. Raw SkyPilot YAMLs may still exist only as guarded
> tool-specific examples or resource profiles, not as workflow authoring
> surfaces. SkyPilot remains the engine that executes rendered specs.

The catalog keeps promoted principal pipelines in `workflows/main/`; see
[the main catalog](../../../workflows/README.md#main-workflows) for the current
selection and its measured validation scope.
Partner integrations live in `workflows/partners/<partner>/`; general reference
workflows, component tests, and fixtures live in `workflows/testing/`.
IAA and EVG live in `workflows/cosmos-data-factory/` with their own workflow/guide
table and adapted NVIDIA skills under `skills/cosmos-data-factory/`.
Keep the catalog overview in `workflows/README.md`. Maintain a
`Workflow | Guide | Purpose` table in `workflows/main/README.md` and
`workflows/testing/README.md`, beside the YAMLs. Link to the existing guide
location instead of copying or moving it; label CLI or general-guide fallbacks
explicitly. Keep partner indexes and runbooks beside their specs.

## When To Use

Use this skill for repository workflow YAMLs, runner scripts, cookbooks,
artifact contracts, and customer-adaptable pipeline implementations.

## Procedure

1. Start from the closest checked-in `npa.workflow` spec under
   `workflows/`.
2. Reuse a toolRef from
   `npa/src/npa/orchestration/npa_workflow/catalog.py`; add missing behavior to
   the workbench tool rather than implementing it again in a runner.
3. Keep the runner thin. Python runners materialize config, call the workflow
   submission helper, and report artifacts; the spec owns the stage graph.
4. Keep all input and output paths configurable and run-scoped through S3.
   Stages run in separate pods and cannot depend on a repository-relative path.
5. Declare the output the tool actually writes. Extend
   `test_spec_declared_outputs.py` when a tool exposes a result-URI helper.
6. Run `validate-spec`, then `plan-spec --run-id preview`, before live submit.
   Register every shipped spec in `SUBMIT_LIVE_MATRIX`.

For `field-failure-policy-improvement.yaml`, the native reference adapters join
metric RGB-D reconstruction and shared-scene Isaac navigation. Follow the
[native adapter contract](../../../docs/workbench/cookbooks/field-failure-policy-improvement.md#native-metric-capture-and-isaac-reference).
Include both components in the reviewed source distribution, pin the navigation
module inventory separately from the immutable image, and preserve source-overlay
provenance. TSDF reconstruction is CPU work; native renderer/learner execution on
RTX supplies the GPU evidence. Retain before/after replays of training-exposed
failures separately from the untouched paired cohort. Only the latter may drive
promotion; valid losing baseline episodes must not be discarded by a standalone
success threshold. Public reference results do not establish compatibility with
an undisclosed proprietary policy or calibration.

For the public scan-to-policy sample, derive the parked isolation-control pose
from dense rotated stance support and exact source topology before learning.
Preserve its measured selection evidence in `support.json`; five corner/center
rays or a large center clearance alone cannot exclude a small hole or a source
boundary beneath a foot. Keep the free/obstacle controls, action sequence,
native contact classification, cohort sizes and success gates unchanged when
repairing initial parking placement. Initial CPU support evidence does not
replace native control and full training/evaluation qualification.

For `field-failure-reference-demo.yaml`, freeze training, development and final
cohorts and the simulation-failure admission rule before learning. Evaluate the
exact baseline on every frozen office training route before admitting its public
capture; recompute complete native outcomes, bind admission into the public bundle
reference, and reverify it before reconstruction and training. A crashed or
control-failed observation cannot admit a capture; zero failures publish an honest
terminal report without continuation or final evaluation. Preserve the distinction
between these observed simulation failures and operator-supplied physical logs.
Before admitting final evaluation, development selection
must pair the exact case IDs and seeds and apply every per-case regression bound
from the frozen `plan.metrics`, using the final comparator's strict boundary.
Compute success gain as the mean paired success delta so an improvement of exactly
one percentage point passes without an epsilon or a weakened threshold. Failed
development selection must publish the measured violations in the HTML report
and leave final evaluation untouched.
Interrupted selection/report publication may accept identical retained bytes
to finish missing report files. Reject different bytes without overwriting
evidence; report recovery must not change eligibility or consume final cases.

## Current Reference YAMLs

For Lyra, `workflows/testing/lyra-reconstruction.yaml` is the standalone
captured-video reconstruction workflow. The dedicated `npa-lyra2` recipe is at
`npa/docker/workbench/lyra2/`; its model, source and inference environment are
runtime fetches. The workflow pins the RTX PRO 6000-qualified public release digest and
baked NPA revision without a source overlay. Preserve the completed
Gaussian/video/geometry/HTML outputs and
the pinned model provenance. The earlier image retains separate B200 evidence;
do not transfer that claim to the repaired image. A reconstruction result does not qualify the
separate `lyra-scene-actions.yaml` physics or wrist-camera path. Follow the
[Lyra guide](../../../docs/workbench/guides/lyra-physical-augmentation.md).

For stitched mono 2:1 panoramic video or JPG/PNG sequences, use
`npa.workflows.lyra_panorama_capture` from the updated local checkout before
submitting the unchanged reconstruction spec. Follow the
[360° capture guide](../../../docs/workbench/guides/lyra-360-capture.md) for fixed
perspective reprojection, output dimensions, source ordering and exact excerpts.
Raw fisheye requires a calibrated stitched export; do not guess its lens model
or treat a resized panorama as pinhole input. Keep each view direction in a
separate sequence. Source filenames, paths and EXIF are omitted from the bundle,
but pixels and reconstruction artifacts remain private. Synthetic CPU projection
checks do not establish reduced haze or transfer the existing RTX qualification
to panoramic captures. Preserve an independently validated splatting baseline
and require a matched quality comparison before claiming improvement. Panorama
preparation supplies no metric calibration for the collision/action path.

When formalizing a splatting baseline without a supplied implementation, use
the [COLMAP → gsplat reference runbook](../../../docs/workbench/guides/gaussian-splatting-reference.md).
It specifies manual upstream camera registration, image/camera export, pinned
gsplat training and private recipe notes. Label it as an unqualified reference;
do not claim it reproduces an operator's original pipeline or invent results.
Use calibrated perspective input and retain actual registration and appearance
evaluation evidence. The upstream appearance split may share images with pose
estimation, and Lyra's native replay does not supply a matched held-out benchmark.
Keep commands with private paths and exact capture/scene details outside Git.

`workflows/testing/gaussian-splatting-reference.yaml` encodes that reference as
native COLMAP and gsplat states with private S3 handoffs. Its runtime image is
an explicit placeholder; retain its readiness record and require a compatible
operator digest before execution. The shareable
`docs/demos/lyra-splatting-overview.html` contains a synthetic projection preview
and exact downloadable YAML for both this reference and the existing Lyra path.
It is a workflow explainer, not a report of real capture reconstruction quality.

When a user needs a real validated example, start from the
[native Lyra reference review](../../../docs/workbench/guides/lyra-validated-example.md),
not the synthetic workflow explainer. `npa.workflows.lyra_validated_example`
requires the retained HTML's qualified digest, native identities and embedded
video hashes; it fully decodes paired media and measures RGB replay PSNR before
packaging real playback, depth samples, evidence and workflow YAML. Keep the
generated file private under the recorded internal R&D scope. The
`lyra2-panorama-rtx-20261016.json` qualification records a separate real 5.7K
Insta360 One X run: full 827-frame preparation, normal 128-view GPU inference,
8,658,944 native Gaussians and complete artifact verification. The older TUM
perspective qualification and its CPU repackaging remain separate evidence.
Packaging does not launch GPU work. Neither replay establishes held-out
quality, haze improvement, raw-fisheye support or X5 performance; 8K preparation
tests remain synthetic.

For a first BEHAVIOR DEV evaluation, use
[the one-file onboarding guide](../../../docs/workbench/challenge-onboarding.md)
and `npa workbench workflow challenge init`, `check`, and `prepare`.
The helper materializes the canonical evaluation workflow and preserves the ten
prescribed cases. A local `ready-to-prepare` result does not prove GPU readiness.
The simulator runtime, licensed assets and task-configured policy remain
operator prerequisites. Keep generated kits private; use the campaign path from
the outset for durable case recovery. Other challenges are not supported by
this helper yet.

The retired catalog path is machine-checked by
`npa/tests/guardrails/test_skypilot_catalog_retirement.py`, so a raw template
cannot quietly reappear there.

No raw SkyPilot templates remain in the retired catalog. Author workflow examples
as `npa.workflow/v0.0.1` specs under `workflows/testing/` or, for partner
integrations, `workflows/partners/<partner>/`.

## Retired Templates

These raw templates were retired once their `npa.workflow` spec had a live run
(run ids in `EVIDENCE.md`). Use the spec under
`workflows/`:

- `isaac-lab-rl-sweep.yaml` — parallel GPU sweep (`--runtime`).
- `cosmos3-reason.yaml` — Cosmos3 reason-stage manifest.
- `sonic-export.yaml`, `sonic-eval.yaml`, `sonic-export-eval.yaml` — SONIC
  export/eval. The tools now accept `s3://` inputs and outputs directly, which is
  what the templates' inline download/upload bash used to do.
- `token-factory-caption.yaml`, `token-factory-generate.yaml`,
  `token-factory-cosmos-reason.yaml` — hosted Token Factory stages.
- `mjlab-eval.yaml` — MJLab locomotion evaluation.
- `retargeting.yaml` — motion retargeting. The harness synthesizes a SOMA-CSV clip
  (`npa.workflows.motion_fixture`) when no real motion set is staged.
- `vlm-eval.yaml`, `vlm-eval-benchmark.yaml` — self-hosted VLM scoring and the labeled
  sweep. The renderer now starts and health-checks the vLLM server the spec asks for, so
  no prebuilt serving image is needed.
- `cosmos3-text-to-image-inference.yaml` — retired to
  `workflows/testing/cosmos3-text-to-image.yaml`. The procedure it carried as bash inside an `envs:`
  block is now `npa workbench cosmos3 text-to-image`.
- `bdd100k-pipeline.yaml` — retired to `workflows/testing/bdd100k-pipeline.yaml`. A live run needs
  both in-cluster services (`lancedb` and `detection-training`) deployed first.
- `dataset-ingest-curate.yaml` — retired to `workflows/testing/dataset-ingest-curate.yaml`, whose
  `register` stage reads back what `ingest` wrote to the in-cluster LanceDB service
  (`npa workbench lancedb deploy --runtime kubernetes --namespace workbench`).
- `sim-to-real-pipeline.yaml` / `sim-to-real-trigger.yaml` — retired. The pipeline ran the
  deprecated `npa.workflows.sim_to_real real-loop`; the maintained path and the
  watcher's submit target are the single staged-engine YAML,
  `workflows/main/sim2real.yaml`.
- `cosmos2-transfer.yaml` — retired to `workflows/testing/cosmos2-transfer.yaml`, which runs the
  REAL Cosmos-Transfer2.5 model (`--execute`) instead of printing a `contract_ready` payload.
- `isaac-franka-capture-reason.yaml` — retired to
  `workflows/testing/isaac-franka-capture-reason.yaml`. The capture code moved into the package
  (`npa.workflows.isaac_capture`), so the stage no longer needs a repo mounted into the pod.
- `sim2real-actions.yaml` — retired into `workflows/testing/sim2real-envgen-shards.yaml` as its
  fourth stage, which conditions the train slice the `split` stage just wrote.
- `tokenfactory-scene-to-rollout-judge.yaml` — hosted reasoner, GPU rollout, hosted judge. Its
  twin keeps the chain: `vlm-eval run --task-from` reads the reasoner's artifact, so the judge
  scores the rollout against the plan rather than a literal string.
- `tokenfactory-rollout-judge.yaml` — GPU rollout then a hosted VLM judge. Its twin is
  `workflows/testing/tokenfactory-rollout-judge-combo.yaml`; note the older same-named spec is a
  *different* workflow (a Cosmos reasoner feeding a judge over externally-seeded rollouts).
- `tokenfactory-train-triage.yaml` — GPU LeRobot training then a hosted triage report. Its twin
  `workflows/testing/tokenfactory-train-triage.yaml` trains in the stage's own pod (the renderer
  switches to the vendor image's interpreter) and triages with
  `npa.workflows.token_factory_triage`. Needs a SkyPilot-hostable LeRobot image; 0.5.1 ships a
  torch/torchcodec ABI mismatch, 0.6.0 does not.
- `cosmos3-ea-fetch.yaml` — Cosmos source/checkpoint fetch. Its twin
  `workflows/testing/cosmos-fetch.yaml` is the two CLI commands the template wrapped in ~60 lines
  of setup bash; the renderer installs `huggingface_hub[cli]`, which was the only load-bearing
  line of that preamble.
- `cosmos3-generate.yaml` — Cosmos 3 omni-model generation in the `npa-cosmos3`
  image. Its twin `workflows/testing/cosmos3-generate.yaml` ran through the live
  submit matrix and produced `generated/generate.json` plus a non-flat 960x960
  `generated/vision.jpg`.
- `nurec-reconstruct.yaml` — **relocated**, not retired: #234 deliberately
  shipped and live-verified a single-pod NuRec/NRE SkyPilot task in addition to
  the multi-pod `npa.workflow` spec. It now lives at
  `npa/src/npa/workbench/nurec/examples/` with its own README and guardrail.
- `sim2real-envgen-split.yaml` — raw env generation + 80/20 split. Its twin
  `workflows/testing/sim2real-envgen-shards.yaml` declares the shard fan-out as a `parallel:`
  group instead of relying on a Kubernetes Job completion index, and runs on CPU.
- `scenario-gen-adversarial.yaml` — adversarial scenario mining. Its twin
  `workflows/testing/scenario-gen-smoke.yaml` runs the same two CLI commands; the template's GPU
  image advertised an RL adversary the CLI cannot select.
- `sim-to-real-loop.yaml` — the rollout-SET loop. Retired via a new tool capability
  (`npa workbench vlm-eval loop`), because nothing else produced
  `task_success_report.json`; the spec is `workflows/testing/vlm-eval-loop.yaml`.
- `isaac-lab-cosmos-sdg-burst-smoke.yaml` — **relocated**, not retired: a single-task input to
  `npa burst submit-yaml`, now at `npa/src/npa/burst/examples/`. Burst is scoped to one
  executable task, so there is no plan or stage graph for a spec to describe.
- `isaac-lab-rl-train-rtxpro.yaml`, `isaac-lab-rl-train-rtxpro-smoke.yaml`,
  `isaac-lab-rl-train.yaml`, `byof-datagen-rtxpro-smoke.yaml`,
  `byof-container-smoke-rtxpro.yaml` — **relocated**, not retired: they are BYOF
  *resource profiles* (a pod shape), not workflows, and now live beside their
  runner at `npa/src/npa/workflows/byof/profiles/`.

The retired catalog is pinned empty in
`npa/tests/guardrails/test_skypilot_catalog_retirement.py`; do not add new raw
workflow templates.

## Three-Tier Contract

- CLI: use `npa workbench workflow ...` and tool-specific workflow commands
  such as `npa workbench mjlab workflow` or `npa workbench sonic retargeting workflow`.
- SDK: route through shared workflow submission helpers rather than shelling out
  from business logic.
- Workflow: the `npa.workflow` spec is the executable source of truth for stage
  order, resources, configuration, and artifact paths. ToolRef argv templates
  are the source of truth for commands; SkyPilot is the rendered execution
  layer.

## Gotchas

- The native Ray CLIP recipe has an optional completed-result archive companion
  at `npa/workflows/workbench/ray-clip-development/archive.py`; follow
  `docs/testing/ray-clip-archive.md` for quiescent source inventories, conditional
  S3 publication and hash-bound private restore. It does not submit Jobs or
  resume incomplete checkpoints, and its local rsync path needs no S3 credentials.

- Customer-provided raw SkyPilot `envs` does not support self-referencing
  interpolation; repository specs use resolved `config` tokens.
- `sky jobs launch` has no dry-run flag. Use `workflow submit --plan-only` for a
  rendered spec preflight.
- Keep SONIC locomotion orchestration in its spec; do not add a Python runner
  that re-implements the graph.

## Verify

```bash
npa/.venv/bin/python -m pytest npa/tests/guardrails/test_skills_index.py -q
```

The smoke test parses the listed workflow YAMLs and invokes workflow CLI help.