git:20261021.7e918d7 to git:20261021.a2a7409

---

name: cosmos3-nano-sim-auto-inference

description: Prepare inputs or run Cosmos3-Nano-Sim-Auto camera-only or joint camera/LiDAR generation with a released or exported student in Cosmos Framework. Check causal conditioning, per-camera captions, rolling metadata and saved outputs.

---

# Run inference with Cosmos3-Nano-Sim-Auto

Use the four-step causal student to generate synchronized camera views and

optional LiDAR from initial RGB, per-camera captions and WSM/HD-map controls.

The released model is optimized for the reference camera rig. You can try a

custom rig, but camera geometry differences can degrade quality. Follow the

[reference-rig retargeting or custom-rig workflow](../../cosmos3-nano-transfer-auto/inference/references/input-preparation.md#choose-reference-rig-or-custom-rig-generation).

Retargeting WSM does not reproject initial RGB images. For adaptation, fine-tune

Transfer-Auto on your rig's paired data, then distill it into Sim-Auto.

## Start here

- **New rig:** compare it with the reference cameras in

[Camera Rig Explorer](../../cookbooks/cosmos3/simulation/auto/wsm/docs/camera-rig/guide.md), then follow the

shared [camera-to-embedding mapping](../../cosmos3-nano-transfer-auto/inference/references/input-preparation.md#2-map-physical-cameras-to-learned-identities).

Map each camera to the closest supported identity and keep RGB, WSM and caption

assignments consistent. For domain/rig adaptation, post-train Transfer-Auto

first and then [distill it](../post-training/SKILL.md).

- **Captions:** use the same

[camera-local, training-aligned style](../../cosmos3-nano-transfer-auto/shared/caption-policy.md#keep-training-and-inference-captions-aligned)

as Transfer-Auto, checking appearance and motion across views and time.

- **Sampling:** use guidance **1**, control guidance **1** and the checkpoint's

**four-step** schedule. Read [why Sim-Auto settings differ](../../cosmos3-nano-transfer-auto/inference/references/input-and-execution.md#sim-auto-uses-a-distilled-schedule);

guidance 1 still uses captions and controls, but a negative prompt adds no CFG effect.

- **Timing:** use aligned 30-FPS cameras for the reference recipe; joint Sim-Auto

requires that camera rate. Read the [FPS guide](../../cosmos3-nano-transfer-auto/inference/references/input-and-execution.md#source-timing-model-fps-and-playback-fps)

before using lower-rate recordings. Changing playback speed does not correct

model timing or resample controls.

- **Size and duration:** both 480p and 720p are supported; use the

[training-matched landscape preset](../../cosmos3-nano-transfer-auto/inference/references/input-and-execution.md#resolution-and-aspect-ratio).

Sim-Auto supports long rolling generation, with qualitatively observed

multi-minute rollouts. Read the [duration guide](../../cosmos3-nano-transfer-auto/inference/references/input-and-execution.md#generation-length-and-long-rollouts)

for quality expectations and full-horizon input requirements.

## Choose the task

- For input preparation only, follow the

[input-preparation guide](references/input-preparation.md) and

[caption guidance](references/input-preparation.md#camera-local-captions). Return the checked media inventory

and per-view caption sidecars without launching generation.

- To generate videos and optional LiDAR, continue below.

## Prepare

1. Follow [Framework setup](../../cosmos3-nano-transfer-auto/shared/framework-runtime.md).

Set `SIM_AUTO_HF` to the downloaded student directory, or a complete student

export from [distillation](../post-training/SKILL.md).

2. Follow [input preparation](references/input-preparation.md). Both Auto models share

the [reference scene](../../cosmos3-nano-transfer-auto/shared/reference-scene.md#get-the-scene);

3. Set `AUTO_REQUEST="$AUTO_SAMPLE_ROOT/input-sim-auto.jsonl"` for the

reference scene, or prepare your own JSON/JSONL request using

[these fields](references/input-preparation.md#request-fields). Set a fresh `AUTO_OUTPUT` directory.

The sample request selects paragraph captions; use

[caption selection](../../cosmos3-nano-transfer-auto/shared/reference-scene.md#choose-a-caption-style)

for the recommended timestamped alternative.

Both models use `daytime_scene`: seven cameras, WSM v4.0.1 controls and paired

RGB/LiDAR targets. `AUTO_SAMPLE_ROOT` points to the downloaded

`assets/sample_data` directory; Sim-Auto uses its `input-sim-auto.jsonl` request.

Supply scene-caption bodies only. Framework automatically adds the camera-rig

and current-view direction/lens/FOV description, plus the WSM-control task

instruction as a system prompt. Do not paste these into your caption files;

see [automatic prefixes](../../cosmos3-nano-transfer-auto/shared/caption-policy.md#caption-text-and-automatic-prefixes).

Use an exported model snapshot, not a training experiment or raw DCP directory.

The artifact must include per-view-caption metadata, matching tokenizers and

`rolling_kv_cache_chunks`, `rolling_kv_sink_chunks`, `rolling_prompt_policy`.

Do not guess missing history settings from the model name.

## Run

```bash

export COSMOS_INTERNAL=0 COSMOS_TRAINING=0

torchrun --standalone --nproc-per-node=4 \

-m cosmos_framework.scripts.inference \

-i "${AUTO_REQUEST:?set a prepared request}" \

-o "${AUTO_OUTPUT:?set a fresh output directory}" \

--checkpoint-path "${SIM_AUTO_HF:?set the local student snapshot}" \

--cp-size 4 --cfgp-size 1 \

--no-use-torch-compile --no-diffusion-cache

```

The request selects camera-only versus joint generation by omitting or including

`lidar.control_path`; there is no separate `--joint-generation` flag here.

The fixed student schedule comes from its model configuration. Do not add

Transfer-Auto's 35-step/shift/negative-CFG recipe.

The causal frontend runs eagerly with serial CFG. It disables unsupported

compilation/CUDA-graph/cache options and reports adjustments. Rolling history

comes from checkpoint metadata; older generic KV/sink-size flags are rejected.

Memory fit and speed depend on the request and hardware. Read

[GPU memory and parallelism](../../cosmos3-nano-transfer-auto/inference/references/input-and-execution.md#gpu-memory-and-parallelism)

before changing the allocation; a four-GPU example is not a guarantee of fit on

every device.

## Check the result

Inspect the sample output directory:

- `vision_view<NN>_<camera_key>.mp4` and corresponding control videos.

- `caption_chunks.json` and `rolling_prompt.json`: text and transitions actually used.

- `sample_args.json` and `sample_outputs.json`: resolved request and completion.

- Joint only: `lidar.safetensors` with `range_m`, `intensity` and `frame_ids`,

plus decoded `lidar.pt`.

At 297 camera frames and 30 FPS, joint inference generates 99 sweeps at 10 Hz.

Check source IDs and numeric ranges, not only a rendered point cloud. Inspect all

camera views and 50/50 WSM overlays; compare caption transitions over the full

requested horizon.

For a preview, the Framework provides

`python -m cosmos_framework.scripts.multiview_collage "$AUTO_OUTPUT" --tile-width 832`.

Preserve original numeric outputs and videos. Use

[troubleshooting](../troubleshooting/SKILL.md) for failures.

## References

- [Input preparation, captions and request fields](references/input-preparation.md)

- [Framework environment and model assets](../../cosmos3-nano-transfer-auto/shared/framework-runtime.md)

- [Shared Auto reference scene](../../cosmos3-nano-transfer-auto/shared/reference-scene.md)