v1.0.0 to v1.0.0

25 added, 25 removed. Audit A to A.

---

name: evaluating-cosmos-policy

- description: Evaluates and benchmarks NVIDIA Cosmos Policy on LIBERO and RoboCasa simulation environments. Use when setting up cosmos-policy for robot manipulation evaluation, running headless GPU evaluations with EGL rendering, comparing KV cache settings, or profiling inference latency on cluster or local GPU machines.

+ description: Evaluates NVIDIA Cosmos Policy on LIBERO and RoboCasa simulation environments. Use when setting up cosmos-policy for robot manipulation evaluation, running headless GPU evaluations with EGL rendering, or profiling inference latency on cluster or local GPU machines.

version: 1.0.0

author: Orchestra Research

license: MIT

- tags: [Cosmos Policy, VLA, Robotics, LIBERO, RoboCasa, Simulation, Evaluation, Benchmarking, KV Cache, EGL Rendering]

+ tags: [Cosmos Policy, VLA, Robotics, LIBERO, RoboCasa, Simulation, Evaluation, Profiling, EGL Rendering]

dependencies: [torch>=2.1.0, mujoco>=3.0.0, robosuite>=1.4.0, robocasa>=0.2.0, transformers>=4.40.0, "cosmos-policy @ git+https://github.com/NVlabs/cosmos-policy.git"]

---

# Cosmos Policy Evaluation

- Evaluation and benchmarking workflows for NVIDIA Cosmos Policy on LIBERO and RoboCasa simulation environments. Covers setup, headless GPU evaluation, inference profiling, and cross-attention KV cache A/B comparison.

+ Evaluation workflows for NVIDIA Cosmos Policy on LIBERO and RoboCasa simulation environments. Covers setup, headless GPU evaluation, inference profiling, and how to map old repo-local wrappers back to the public upstream commands.

## Quick start

Run a minimal LIBERO evaluation using the official public eval module:

```bash

uv run --extra cu128 --group libero --python 3.10 \

python -m cosmos_policy.experiments.robot.libero.run_libero_eval \

--config cosmos_predict2_2b_480p_libero__inference_only \

--ckpt_path nvidia/Cosmos-Policy-LIBERO-Predict2-2B \

--config_file cosmos_policy/config/config.py \

--use_wrist_image True \

--use_proprio True \

--normalize_proprio True \

--unnormalize_actions True \

--dataset_stats_path nvidia/Cosmos-Policy-LIBERO-Predict2-2B/libero_dataset_statistics.json \

--t5_text_embeddings_path nvidia/Cosmos-Policy-LIBERO-Predict2-2B/libero_t5_embeddings.pkl \

--trained_with_image_aug True \

--chunk_size 16 \

--num_open_loop_steps 16 \

--task_suite_name libero_10 \

--num_trials_per_task 1 \

--local_log_dir cosmos_policy/experiments/robot/libero/logs/ \

--seed 195 \

--randomize_seed False \

--deterministic True \

--run_id_note smoke \

--ar_future_prediction False \

--ar_value_prediction False \

--use_jpeg_compression True \

--flip_images True \

--num_denoising_steps_action 5 \

--num_denoising_steps_future_state 1 \

--num_denoising_steps_value 1 \

--data_collection False

```

## Core concepts

- **What Cosmos Policy is**: NVIDIA Cosmos Policy is a vision-language-action (VLA) model that uses Cosmos Tokenizer to encode visual observations into discrete tokens, then predicts robot actions conditioned on language instructions and visual context. It supports cross-attention KV caching for faster autoregressive inference.

+ **What Cosmos Policy is**: NVIDIA Cosmos Policy is a vision-language-action (VLA) model that uses Cosmos Tokenizer to encode visual observations into discrete tokens, then predicts robot actions conditioned on language instructions and visual context.

**Key architecture choices**:

| Component | Design |

|-----------|--------|

| Visual encoder | Cosmos Tokenizer (discrete tokens) |

| Language conditioning | Cross-attention to language embeddings |

| Action prediction | Autoregressive action token generation |

- | KV cache | Optional cross-attention cache for inference speedup |

- **Cross-attention KV cache**: The underlying model architecture uses cross-attention and related caching optimizations, but the public eval CLI does not currently expose a stable one-flag KV cache toggle. If you need cache A/B experiments, treat them as repo-specific automation rather than assuming a portable upstream command.

+ **Wrapper boundary**: The public repo exposes evaluation entrypoints such as `cosmos_policy.experiments.robot.libero.run_libero_eval` and `cosmos_policy.experiments.robot.robocasa.run_robocasa_eval`. If your original research repo used helper wrappers, those wrappers were repo-local orchestration around these public commands rather than a separate upstream API surface.

## Compute requirements

|------|-----|------|-------------------|

| LIBERO smoke eval (1 trial) | 1x A40/A100 | ~16 GB | 5-10 min |

| LIBERO full eval (50 trials) | 1x A40/A100 | ~16 GB | 2-4 hours |

| RoboCasa single-task (2 trials) | 1x A40/A100 | ~18 GB | 10-15 min |

| RoboCasa all-tasks | 1x A40/A100 | ~18 GB | 4-8 hours |

- | KV cache A/B benchmark | 1x A40/A100 | ~20 GB | 30-60 min |

## When to use vs alternatives

**Use this skill when:**

- Evaluating NVIDIA Cosmos Policy on LIBERO or RoboCasa benchmarks

- Profiling inference latency and throughput for Cosmos Policy

- - Running A/B comparisons of cross-attention KV cache configurations

- Setting up headless EGL rendering for robot simulation on GPU clusters

**Use alternatives when:**

- Training or fine-tuning Cosmos Policy from scratch (use official Cosmos training docs)

- Working with OpenVLA-based policies (use `fine-tuning-openvla-oft`)

- Working with Physical Intelligence pi0 models (use `fine-tuning-serving-openpi`)

- Running real-robot evaluation rather than simulation

---

## Workflow 1: LIBERO evaluation

Copy this checklist and track progress:

```text

LIBERO Eval Progress:

- [ ] Step 1: Install environment and dependencies

- [ ] Step 2: Configure headless EGL rendering

- [ ] Step 3: Run smoke evaluation

- [ ] Step 4: Validate outputs and parse results

- [ ] Step 5: Run full benchmark if smoke passes

```

**Step 1: Install environment**

```bash

conda create -n cosmos-policy python=3.10 -y

conda activate cosmos-policy

pip install -e .

pip install robosuite==1.4.0

```

**Step 2: Configure headless rendering**

```bash

export CUDA_VISIBLE_DEVICES=0

export MUJOCO_EGL_DEVICE_ID=0

export MUJOCO_GL=egl

export PYOPENGL_PLATFORM=egl

```

**Step 3: Run smoke evaluation**

```bash

uv run --extra cu128 --group libero --python 3.10 \

python -m cosmos_policy.experiments.robot.libero.run_libero_eval \

--config cosmos_predict2_2b_480p_libero__inference_only \

--ckpt_path nvidia/Cosmos-Policy-LIBERO-Predict2-2B \

--config_file cosmos_policy/config/config.py \

--use_wrist_image True \

--use_proprio True \

--normalize_proprio True \

--unnormalize_actions True \

--dataset_stats_path nvidia/Cosmos-Policy-LIBERO-Predict2-2B/libero_dataset_statistics.json \

--t5_text_embeddings_path nvidia/Cosmos-Policy-LIBERO-Predict2-2B/libero_t5_embeddings.pkl \

--trained_with_image_aug True \

--chunk_size 16 \

--num_open_loop_steps 16 \

--task_suite_name libero_10 \

--num_trials_per_task 1 \

--local_log_dir cosmos_policy/experiments/robot/libero/logs/ \

--seed 195 \

--randomize_seed False \

--deterministic True \

--run_id_note smoke \

--ar_future_prediction False \

--ar_value_prediction False \

--use_jpeg_compression True \

--flip_images True \

--num_denoising_steps_action 5 \

--num_denoising_steps_future_state 1 \

--num_denoising_steps_value 1 \

--data_collection False

```

**Step 4: Validate and parse results**

```python

import json

import glob

# Find latest evaluation result from the official log directory

log_files = sorted(glob.glob("cosmos_policy/experiments/robot/libero/logs/**/*.json", recursive=True))

with open(log_files[-1]) as f:

results = json.load(f)

print(results)

```

**Step 5: Scale up**

Run across all four LIBERO task suites with 50 trials:

```bash

for suite in libero_spatial libero_object libero_goal libero_10; do

uv run --extra cu128 --group libero --python 3.10 \

python -m cosmos_policy.experiments.robot.libero.run_libero_eval \

--config cosmos_predict2_2b_480p_libero__inference_only \

--ckpt_path nvidia/Cosmos-Policy-LIBERO-Predict2-2B \

--config_file cosmos_policy/config/config.py \

--use_wrist_image True \

--use_proprio True \

--normalize_proprio True \

--unnormalize_actions True \

--dataset_stats_path nvidia/Cosmos-Policy-LIBERO-Predict2-2B/libero_dataset_statistics.json \

--t5_text_embeddings_path nvidia/Cosmos-Policy-LIBERO-Predict2-2B/libero_t5_embeddings.pkl \

--trained_with_image_aug True \

--chunk_size 16 \

--num_open_loop_steps 16 \

--task_suite_name "$suite" \

--num_trials_per_task 50 \

--local_log_dir cosmos_policy/experiments/robot/libero/logs/ \

--seed 195 \

--randomize_seed False \

--deterministic True \

--run_id_note "suite_${suite}" \

--ar_future_prediction False \

--ar_value_prediction False \

--use_jpeg_compression True \

--flip_images True \

--num_denoising_steps_action 5 \

--num_denoising_steps_future_state 1 \

--num_denoising_steps_value 1 \

--data_collection False

done

```

---

## Workflow 2: RoboCasa evaluation

Copy this checklist and track progress:

```text

RoboCasa Eval Progress:

- [ ] Step 1: Install RoboCasa assets and verify macros

- [ ] Step 2: Run single-task smoke evaluation

- [ ] Step 3: Validate outputs

- [ ] Step 4: Expand to multi-task runs

```

**Step 1: Install RoboCasa**

```bash

pip install robocasa

python -m robocasa.scripts.setup_macros

python -m robocasa.scripts.download_kitchen_assets

```

Verify `macros_private.py` exists and paths are correct.

**Step 2: Single-task smoke evaluation**

```bash

uv run --extra cu128 --group robocasa --python 3.10 \

python -m cosmos_policy.experiments.robot.robocasa.run_robocasa_eval \

--config cosmos_predict2_2b_480p_robocasa_50_demos_per_task__inference \

--ckpt_path nvidia/Cosmos-Policy-RoboCasa-Predict2-2B \

--config_file cosmos_policy/config/config.py \

--use_wrist_image True \

--num_wrist_images 1 \

--use_proprio True \

--normalize_proprio True \

--unnormalize_actions True \

--dataset_stats_path nvidia/Cosmos-Policy-RoboCasa-Predict2-2B/robocasa_dataset_statistics.json \

--t5_text_embeddings_path nvidia/Cosmos-Policy-RoboCasa-Predict2-2B/robocasa_t5_embeddings.pkl \

--trained_with_image_aug True \

--chunk_size 32 \

--num_open_loop_steps 16 \

--task_name TurnOffMicrowave \

--obj_instance_split A \

--num_trials_per_task 2 \

--local_log_dir cosmos_policy/experiments/robot/robocasa/logs/ \

--seed 195 \

--randomize_seed False \

--deterministic True \

--run_id_note smoke \

--use_variance_scale False \

--use_jpeg_compression True \

--flip_images True \

--num_denoising_steps_action 5 \

--num_denoising_steps_future_state 1 \

--num_denoising_steps_value 1 \

--data_collection False

```

**Step 3: Validate outputs**

- Confirm the eval log prints the expected task name, object split, and checkpoint/config values.

- Inspect the final `Success rate:` line in the log.

**Step 4: Expand scope**

Increase `--num_trials_per_task` or add more tasks. Keep `--obj_instance_split` fixed across repeated runs for comparability.

---

- ## Workflow 3: Benchmarking note

+ ## Workflow 3: Reconstructing local wrappers

- Use this workflow to avoid inventing a public benchmark CLI that does not exist.

+ Use this workflow when an older research repo referenced helper scripts that are not present in the public Cosmos Policy repo.

```text

- Benchmarking Progress:

- - [ ] Step 1: Confirm whether you are using upstream public commands or repo-local automation

- - [ ] Step 2: Run one deterministic smoke eval and save logs

- - [ ] Step 3: Only then layer on any custom A/B benchmarking wrapper

+ Wrapper Reconstruction Progress:

+ - [ ] Step 1: Lock down the upstream public command first

+ - [ ] Step 2: Record the non-command responsibilities your old wrapper handled

+ - [ ] Step 3: Recreate that orchestration in your local repo without inventing new upstream flags

```

- **Step 1: Pick the right command surface**

+ **Step 1: Lock down the public command surface**

- - If you are using the public upstream repo, stick to `cosmos_policy.experiments.robot.*.run_*_eval` and do not invent `--enable-cross-attn-kv-cache` or `--output-dir` flags.

- - If you are using a local research repo that adds wrappers or patched configs, document that repo-specific command beside the upstream command.

+ - Start from `cosmos_policy.experiments.robot.libero.run_libero_eval` or `cosmos_policy.experiments.robot.robocasa.run_robocasa_eval`.

+ - Keep the exact config, checkpoint, dataset stats, and embedding arguments beside the command so the wrapper does not silently hide required state.

- **Step 2: Save one deterministic smoke run first**

+ **Step 2: Capture what the old wrapper actually did**

- - For LIBERO, keep `--num_trials_per_task 1` and a fixed seed such as `195`.

- - For RoboCasa, keep `--num_trials_per_task 2`, fixed `--obj_instance_split`, and a fixed seed.

+ - Export EGL and device-selection variables such as `CUDA_VISIBLE_DEVICES`, `MUJOCO_EGL_DEVICE_ID`, `MUJOCO_GL`, and `PYOPENGL_PLATFORM`.

+ - Standardize cache roots, Hugging Face cache paths, and log/output directories.

+ - Add cluster-specific concerns such as `srun` or `sbatch` options, container enablement, or module loads.

+ - Validate simulator prerequisites such as LIBERO dataset config or RoboCasa kitchen assets before launching eval.

- **Step 3: Treat A/B benchmarking as repo-specific**

+ **Step 3: Recreate orchestration locally**

- - The public Cosmos Policy repo does not document a stable single-flag KV cache benchmark path.

- - If you need cache A/B numbers, use checked-in local automation from the target repo or patch the config in a reproducible way and record that diff next to the command.

+ - Implement wrappers in your own repo as thin launchers around the public `python -m` entrypoints.

+ - Keep wrapper-added behavior explicit in the script body or README: partition selection, cache root, smoke-test defaults, result-copying, and first-outcome snapshots.

+ - Document any repo-specific config patch next to the wrapper instead of implying that it is part of upstream Cosmos Policy.

---

## Expected performance benchmarks

Reference values from official evaluation (tied to specific setup and seeds):

| Task Suite | Success Rate | Notes |

|-----------|-------------|-------|

| LIBERO-Spatial | ~90-95% | Easiest suite |

| LIBERO-Object | ~85-92% | Object manipulation focus |

| LIBERO-Goal | ~80-88% | Goal-conditioned, harder |

| LIBERO-10 | ~75-85% | 10 diverse long-horizon tasks |

- **Benchmark note**: Any KV cache latency claim depends on repo-local implementation details. Record the exact script or config diff you used before comparing numbers across runs.

+ **Reproduction note**: Published success rates still depend on checkpoint choice, task suite, seeds, and simulator setup. Record the exact command and environment alongside any reported number.

---

## Non-negotiable rules

- **EGL alignment**: Always set `CUDA_VISIBLE_DEVICES`, `MUJOCO_EGL_DEVICE_ID`, `MUJOCO_GL=egl`, and `PYOPENGL_PLATFORM=egl` together on headless GPU nodes.

- **Singularity on clusters**: Keep singularity/apptainer enabled on A40 or similar cluster nodes to avoid `GLIBC_2.29` and `transformer_engine` loader failures.

- **Cache consistency**: Use the same cache directory across setup and eval so Hugging Face and dependency caches are reused.

- - **A/B comparability**: Keep task name, object split, seed, and trial count fixed across cache-on vs cache-off comparisons.

+ - **Run comparability**: Keep task name, object split, seed, and trial count fixed across repeated runs.

---

## Common issues

**Issue: `GLIBC_2.29` or `transformer_engine` loader failures on cluster nodes**

Fix: rerun with the default singularity/container path. Do not force host execution.

**Issue: LIBERO prompts for config path in a non-interactive shell**

Fix: pre-create `LIBERO_CONFIG_PATH/config.yaml`:

```python

import os, yaml

config_dir = os.path.expanduser("~/.libero")

os.makedirs(config_dir, exist_ok=True)

with open(os.path.join(config_dir, "config.yaml"), "w") as f:

yaml.dump({"benchmark_root": "/path/to/libero/datasets"}, f)

```

**Issue: EGL initialization or shutdown noise**

Fix: align EGL environment variables first. Treat teardown-only `EGL_NOT_INITIALIZED` warnings as low-signal unless the job exits non-zero.

**Issue: Kitchen object sampling NaNs or asset lookup failures in RoboCasa**

Fix: rerun asset setup and confirm the patched robocasa install is intact:

```bash

python -m robocasa.scripts.download_kitchen_assets

python -c "import robocasa; print(robocasa.__file__)"

```

**Issue: MuJoCo rendering mismatch**

Fix: verify GPU device alignment:

```python

import os

cuda_dev = os.environ.get("CUDA_VISIBLE_DEVICES", "not set")

egl_dev = os.environ.get("MUJOCO_EGL_DEVICE_ID", "not set")

assert cuda_dev == egl_dev, f"GPU mismatch: CUDA={cuda_dev}, EGL={egl_dev}"

print(f"Rendering on GPU {cuda_dev}")

```

---

## Advanced topics

**LIBERO command matrix**: See [references/libero-commands.md](references/libero-commands.md)

**RoboCasa command matrix**: See [references/robocasa-commands.md](references/robocasa-commands.md)

## Resources

- Cosmos Policy repository: https://github.com/NVlabs/cosmos-policy

- LIBERO benchmark: https://github.com/Lifelong-Robot-Learning/LIBERO

- RoboCasa environment: https://github.com/robocasa/robocasa

- MuJoCo documentation: https://mujoco.readthedocs.io/