sim-to-real · diff
git:20260918.416f964 to git:20261002.bccd515
19 added, 0 removed. Audit A to A.
---
name: sim-to-real
description: Use when designing, reviewing, or operating sim-to-real workflows that move data through simulation, policy training, synthetic generation, evaluation, and loop control.
---
# Sim-To-Real
## When To Use
Use this skill for robotics teams that need configurable data and artifact flow
through simulation, policy training, synthetic data generation, evaluation, and
iteration without customer-specific names or infrastructure baked into source.
## Procedure
1. Define a run ID and run-scoped S3 prefixes before launching.
2. Import source robot, scene, or task data into the run prefix.
3. Generate or augment simulation data with configured workbench tools.
4. Train or fine-tune the policy against the run-scoped dataset and write
checkpoints to S3.
5. Evaluate the policy with deterministic metrics or a configured VLM backend.
6. Decide whether to stop, continue, or route artifacts for review based on
configurable thresholds.
## Three-Tier Contract
- CLI: use `npa workbench workflow` and tool commands
such as Genesis, LeRobot, SONIC, MJLab, Retargeting, LanceDB, Cosmos, and
VLM-eval.
- SDK: keep workflow submission and config materialization in shared helpers so
notebooks and services use the same artifact paths.
- Workflows: `workflows/testing/vlm-eval-loop.yaml` is the executable reference for the
VLM-eval gating loop (scores a whole rollout set and writes
`task_success_report.json`). The staged engine's single maintained end-to-end
YAML is `workflows/main/sim2real.yaml`; similarly named
npa.workflow files are explicitly demo-only DSL fixtures.
`sim-to-real-pipeline.yaml` and `sim-to-real-trigger.yaml` are retired — the first ran
`npa.workflows.sim_to_real real-loop`, which raises a DeprecationWarning pointing here.
+ ## Field failure navigation improvement
+
+ Use `workflows/testing/field-failure-policy-improvement.yaml` for the standard
+ six-stage capture validation, reconstruction, policy fine-tuning, paired
+ held-out evaluation, and recommendation flow. Follow
+ `docs/workbench/cookbooks/field-failure-policy-improvement.md` for the sealed
+ bundle and adapter contracts. Operators supply real navigation adapters in
+ immutable BYOF images, captures, baseline checkpoint, and a disjoint held-out
+ cohort; missing inputs fail closed. GPU qualification remains unverified.
+
+ Preserve the same evaluator callable, source digest, image, scenarios, seeds,
+ and protocol for both policies. Conditional S3 claims isolate attempts;
+ completed stages revalidate retained evidence before reuse. A failed attempt
+ requires a fresh run ID. Raw trajectory hashes must be unique within and across
+ policy arms, including byte-identical deterministic rollouts. Promotion requires
+ positive primary improvement and no regression beyond the sealed per-episode
+ thresholds. The result is a recommendation and never authorizes deployment.
+ Hashes verify identity and consistency, not physical measurement truth.
+
## Gotchas
- Do not hardcode customer names, event names, personal names, tenant IDs,
registry IDs, bucket names, VM IPs, or private endpoints.
- Prefer dry-run plans for data movement, autoscaling, and external service
calls when validating workflow shape.
- Keep artifacts partitioned by run ID so repeated experiments do not overwrite
each other.
- Use existing workbench tools for S3 sync, model inference, training, and VLM
evaluation instead of one-off scripts.
## Verify
```bash
npa/.venv/bin/python -m pytest npa/tests/guardrails/test_skills_index.py -q
```
The smoke test parses the sim-to-real YAMLs and confirms the workflow CLI help
loads through the installed package.