cosmos3-env-troubleshoot git:20260906.a7b9ddbB

Immutable. This exact content is served forever at /api/v1/blob/a9b46926a46b1104.

---
name: cosmos3-env-troubleshoot
description: Use when Cosmos3 setup, fetch, inference, CUDA, uv, Docker, Hugging Face, GitHub, NGC, or checkpoint staging fails in NPA or in an upstream Cosmos framework checkout.
---

# Cosmos3 Environment Troubleshooting

## Source And Attribution

Adapted from NVIDIA cosmos-framework
`skills/atomic/cosmos3-env-troubleshoot/SKILL.md`.

Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. Used under OpenMDW-1.1.
See `skills/LICENSE-NVIDIA-COSMOS3-OPENMDW-1.1` and
`skills/NOTICE-NVIDIA-COSMOS3`.

## When To Use

Use this skill for import errors, missing Python packages, CUDA or torch
failures, Docker GPU runtime problems, failed source clones, Hugging Face 401s,
NGC credential errors, and inference runtime tracebacks. For "where is this
file or config" questions, use
`skills/atomic/cosmos3-codebase-nav/SKILL.md`.

## First Rule

Never print or commit secrets. Token-bearing values include `GITHUB_TOKEN`,
`HF_TOKEN`, `HUGGING_FACE_HUB_TOKEN`, `NGC_API_KEY`, AWS keys, and any env var
selected through `NPA_COSMOS3_*_TOKEN_ENV`.

## NPA Diagnostic Ladder

1. Confirm the NPA command surface:

   ```bash
   npa/.venv/bin/npa workbench cosmos --help
   ```

   `check` and `fetch` should be present. `skill` and `skills` should not be
   present.

2. Run a redacted access check:

   ```bash
   npa/.venv/bin/npa workbench cosmos check --output json
   ```

   Interpret statuses, not secrets. Expected status labels are `configured`,
   `missing`, `reachable`, `failed`, or `skipped`.

3. For source-only failure isolation:

   ```bash
   npa/.venv/bin/npa workbench cosmos fetch --skip-checkpoint --output json
   ```

4. For inference workflow issues, inspect:

   ```bash
   npa/.venv/bin/python - <<'PY'
   import yaml
   from pathlib import Path
   p = Path("workflows/testing/cosmos3-text-to-image.yaml")
   doc = yaml.safe_load(p.read_text())
   print(doc["metadata"]["name"])
   print(doc["states"][doc["initial"]]["toolRef"])
   PY
   ```

5. If a SkyPilot run fails on the GPU node, collect logs and env status without
   dumping token values:

   ```bash
   printenv | rg '^(NPA_COSMOS3|COSMOS3|HF_HOME|LD_LIBRARY_PATH)='
   python --version
   which python
   nvidia-smi
   python -c "import torch; print(torch.__version__, torch.version.cuda)"
   ```

## Common Error Signatures

For `nano-video-batch`, `nano-video-augment` or their vLLM-Omni serving image,
start with `npa/deploy/cosmos3-nano-video/README.md` and the installed image's
video parser/pipeline. Framework inference arguments may not apply. The pinned
video handler reads sequence length through `extra_params`; size alone does not
select transfer resolution. Inspect retained effective requests instead of
assuming an extra field was accepted.

If an augmentation POST, artifact GET or publication fails, retain its original
request/destination and use `nano-video-augment-recover`. Completed local media
can finish publication without serving credentials; incomplete downloads use
authenticated GETs. Recovery never submits generation. Distinguish transport or
artifact failures from actual model failures, and retain both the initial error
and successful readback evidence. For weak visual changes, verify full original
source-interval controls before tuning the prompt or supported parameters;
tail-only continuation and a changed hash are insufficient augmentation proof.

| Error | Likely cause | Fix |
| --- | --- | --- |
| `Hugging Face auth missing` | `HF_TOKEN` or configured HF env var is unset | Set token and accept the model license upstream before fetch |
| Authenticated HF 401 | The supplied token is missing, invalid, or revoked | Check the selected token env var and replace an invalid or revoked token; an anonymous 401 is not a token discriminator |
| Authenticated HF 403 on a gated repo | The token authenticated but lacks authorization | Accept/request repo access; also check fine-grained repo scope and organization token policy |
| Xet `Unable to parse string as hex hash value` | `huggingface_hub==1.23.0` plus `hf-xet==1.5.1` | The r2 image build rejects this pair; in custom runtimes set `HF_HUB_DISABLE_XET=1` or upgrade before retrying |
| `git ls-remote` fails | Source URL wrong or GitHub auth missing for private fork | Check `NPA_COSMOS3_SOURCE_REPO` and selected GitHub token env var |
| `ModuleNotFoundError: cosmos_framework` | Upstream checkout was not installed | Run `uv sync --all-extras --group=cu130-train` from the upstream checkout |
| PyTorch `_functionalization` import error | NGC container library path conflict | Run `export LD_LIBRARY_PATH=` before Python imports |
| CUDA shared library error | CUDA major version mismatch | Align torch CUDA version with host driver CUDA support |
| Docker `runtime name: nvidia` error | Docker NVIDIA runtime not configured | Run `sudo nvidia-ctk runtime configure --runtime=docker` on the host |

## Upstream Remediation

In an upstream Cosmos framework checkout:

```bash
curl -LsSf https://astral.sh/uv/install.sh | sh
export PATH="$HOME/.local/bin:$PATH"
uv sync --all-extras --group=cu130-train --reinstall
source .venv/bin/activate
export LD_LIBRARY_PATH=
python -c "import cosmos_framework; print('cosmos_framework import ok')"
```

Use `cu128-train` if the driver stack requires older CUDA. Use the inference-only
group only when training dependencies are intentionally not needed.

## Bug Report Template

When the failure remains unresolved, give the user a concise report with:

- NPA commit and branch.
- Exact NPA command or SkyPilot workflow used.
- Redacted `NPA_COSMOS3_*` values.
- OS, Python, torch, and CUDA versions.
- Whether `check` passed and whether `fetch --skip-checkpoint` passed.
- Full traceback with secrets removed.
- Whether guardrails were left on or explicitly disabled.