git:20261021.7e918d7 to git:20261021.9a70e64

106 added, 187 removed. Audit A to A.

---

name: cosmos3-nim-user

- description: Guide customers through Cosmos3 Certified NIM deployment, endpoint verification, API example selection, hardware compatibility, and troubleshooting. Use for operating or calling the NIM from the public cookbooks/cosmos3/nim directory; do not use for maintaining the documentation itself.

+ description: Guide customers through Cosmos3 NIM deployment, endpoint verification, API example selection, hardware compatibility, and troubleshooting. Use for operating or calling the NIM from the public .agents/skills/cookbooks/cosmos3/nim directory; do not use for maintaining the documentation itself.

license: OpenMDW-1.1

---

SPDX-License-Identifier: OpenMDW-1.1 -->

- # Cosmos3 Certified NIM customer workflow

+ # Cosmos3 NIM customer workflow

- Work from `.agents/skills/cookbooks/cosmos3/nim` and read `AGENTS.md`. Use the public pages in

- this directory as the source of truth.

+ Work from `.agents/skills/cookbooks/cosmos3/nim` and follow `AGENTS.md`. The public pages here

+ are the source of truth; the rules below say which page owns each decision.

## 1. Identify the customer's path

- Ask only for information needed to route the workflow:

- 1. Do they already have a reachable NIM endpoint, or must they deploy one?

- 2. Do they need Generator or Reasoner?

- 3. Which task do they want to perform?

- 4. For a new deployment, what GPU count, compute capability, per-device total

- and currently free VRAM, and effective host/container RAM are available? Is

- the device discrete or unified memory? For unified memory, also collect

- `MemFree`, `MemAvailable`, `Cached`, `SReclaimable`, and `Shmem`.

+ Ask only what routes the workflow:

- Do not ask for the value of `NGC_API_KEY`, another token, private input media, or

- unredacted logs. It is sufficient to confirm that required secrets are set.

+ 1. Is there a reachable NIM endpoint, or must one be deployed?

+ 2. Generator or Reasoner, and which task?

+ 3. For a new deployment: GPU model and count, compute capability, per-device

+ total and free VRAM, effective host or container RAM, and whether the GPU is

+ discrete or unified memory (DGX Spark/GB10, Jetson AGX Thor).

## 2. Use an existing endpoint

- Help the customer establish `NIM_URL`, then use the documented client tooling

- and perform the runtime preflight:

```bash

export NIM_URL=${NIM_URL:-http://localhost:8000}

curl -fsS "$NIM_URL/v1/health/ready"

curl -fsS "$NIM_URL/v1/metadata" | python3 -m json.tool

```

- Require metadata to identify the intended runtime and endpoint before choosing

- an example:

- - Generator: `model_type` is `generator`; primary endpoint is `/v1/infer`.

- - Reasoner: `model_type` is `reasoner`; primary endpoint is

- `/v1/chat/completions`.

- Use `/v1/models` only after this runtime check. If metadata identifies the wrong

- runtime, do not rewrite the request for that runtime; correct the deployment or

- `NIM_URL`.

+ Metadata must show the intended runtime before you pick an example: Generator

+ reports `model_type: generator` and `/v1/infer`; Reasoner reports

+ `model_type: reasoner` and `/v1/chat/completions`. If the runtime is wrong, fix

+ the deployment or `NIM_URL`; do not rewrite the request for the other runtime.

## 3. Prepare a new deployment

- Guide the customer through the canonical pages in this order:

- 1. `prerequisites.md` for host and client preparation.

- 2. `support-matrix.md` for model, precision, GPU, VRAM, system-memory, and

- Transfer eligibility.

- 3. `deployment.md` for the exact image, authentication, pre-download profile

- preflight, cache, runtime selector, launch, readiness, logs, and cleanup.

- For Kubernetes, use `helm.md` for the chart, secrets, values file, and

- verification.

- 4. `configuration.md` only when a documented non-default setting is needed.

- 5. The existing-endpoint preflight above before the first inference request.

- Use only the exact image and commands in `deployment.md` or, for Kubernetes,

- `helm.md`; never substitute `latest`. After the image pull, run the documented pre-download profile

- preflight with the intended selectors and GPUs. For a Reasoner on DGX Spark/GB10

- or Jetson AGX Thor, every preflight and launch command shown to the customer,

- starting with the first one, must include

- `-e NIM_GPU_MEMORY_UTILIZATION=0.80` for an image-only workload or

- `-e NIM_GPU_MEMORY_UTILIZATION=0.70` for video or mixed-media workloads. Do not

- show a generic Reasoner command first and add the setting later. Treat preflight

- success as evidence for a candidate profile only, not full host compatibility;

- cold start and representative requests remain required. One service container

- starts one runtime. The preflight container is temporary and `--rm` removes

- only that container. Preserve a failed service container long enough to inspect

- its logs, and ask before removing containers or cached data.

- Before NGC login, explain the documented Docker credential-storage behavior and

- external-helper option without presenting a helper as a prerequisite;

- `--password-stdin` protects shell input, not Docker's stored credential. Include

- `docker logout nvcr.io` when guiding evaluation cleanup, and explain that logout

- does not revoke the NGC personal key.

+ Walk through `prerequisites.md`, `support-matrix.md`, then `deployment.md`

+ (`helm.md` for Kubernetes); open `configuration.md` only for a documented

+ non-default setting. Use the exact pinned image, never `latest`. Apply these

+ rules while doing so:

- For hardware guidance:

+ - **Model choice** ([practical recommendation](../../../support-matrix.md#practical-model-recommendation)):

+ Super on H200- and B200-class GPUs when the workload needs it; Nano on H100,

+ RTX PRO 6000 Blackwell, slower discrete GPUs, and all unified-memory systems.

+ A Super row that fits those systems is compatibility, not a recommendation.

+ - **Precision** ([precision selection](../../../deployment.md#precision-selection)):

+ leave it unset. Generator prefers FP8; Reasoner prefers BF16 on compute

+ capability 8.0–8.8, FP8 on 8.9–9.x, and NVFP4 on 10.0 or newer.

+ - **Transfer** ([headroom](../../../support-matrix.md#transfer-headroom)): size

+ against the Transfer minimum, not the generation minimum, and recommend an

+ RTX PRO 6000 Blackwell 96 GB, H100 80 GB, or faster discrete GPU rather than

+ DGX Spark.

+ - **Reasoner on DGX Spark or Jetson AGX Thor**

+ ([memory share](../../../deployment.md#set-the-reasoner-memory-share-on-unified-memory-systems)):

+ the NIM counts 70% of the shared pool toward profile floors and defaults

+ `NIM_GPU_MEMORY_UTILIZATION` to `0.70`. Suggest a lower value only from host

+ measurements, then include it in every Reasoner launch command. Preflight

+ ignores it and always uses the 70% share.

+ - **NGC login** ([authenticate](../../../deployment.md#authenticate-to-ngc)):

+ mention that Docker stores the login unless a credential helper is configured

+ (the helper is optional) and that `--password-stdin` protects only shell

+ input. Include `docker logout nvcr.io` in evaluation cleanup; logout does not

+ revoke the NGC key.

+ - **Host software:** recommend NVIDIA driver 595.58.03 or newer and cite the

+ validated configuration in `prerequisites.md` as tested versions, not

+ minimums. Do not invent other fixed CPU, RAM, disk, or Docker minimums.

+ - **GPU exposure** ([GPU selection](../../../configuration.md#gpu-selection)):

+ select GPUs with Docker `--gpus`, not `CUDA_VISIBLE_DEVICES`.

+ - **Offline or air-gapped**

+ ([procedure](../../../deployment.md#offline-and-air-gapped-deployment)):

+ fill the cache online, copy it preserving symlinks, and do not set

+ `NIM_DISABLE_MODEL_DOWNLOAD` for Generator.

+ - **Preflight** ([preflight](../../../deployment.md#run-the-pre-download-profile-preflight)):

+ run it after the pull with the intended selectors and GPUs. A pass means a

+ candidate profile; cold start and representative requests confirm the host.

+ After startup, run `uv run python examples/inspect_profile.py` to report the

+ profile actually selected.

- - detect DGX Spark/GB10 and Jetson AGX Thor as unified-memory systems; for an

- unfamiliar device, use the exact-image preflight's unified-memory report

- rather than inferring from a marketing memory label;

- - evaluate every participating device's total and currently free memory against

- the per-device floor, including the documented Reasoner runtime reserve; on a

- discrete-GPU host also require the practical host RAM minimum for the selected

- non-offloading model/precision row rather than treating the 16-GiB profile

- admission tag as sufficient;

- - on unified memory, use `MemAvailable` for current state because it includes

- reclaimable cache; report `MemFree` and the cache components separately,

- never add cache to `MemAvailable`, and do not add the discrete-host practical

- RAM requirement to the single shared pool;

- - never add VRAM across devices;

- - use the Transfer minimum when Transfer must be served, and distinguish

- profile compatibility from the task's practical hardware recommendation:

- prefer an RTX PRO 6000 Blackwell 96-GB, H100 80-GB, or higher-throughput

- compatible discrete GPU for Transfer rather than DGX Spark;

- - use the tested-GPU inventory as the official validation scope, not as a

- compatibility allowlist for every profile or task;

- - recommend Super on H200- and B200-class discrete GPUs when the workload needs

- it; default to Nano for generation on H100, RTX PRO 6000 Blackwell, lower-

- throughput discrete devices, and all unified-memory devices; treat a fitting

- Super row on those systems as compatibility, not a performance

- recommendation;

- - choose runtime, model variant, and Generator latency/throughput selectors

- first, normally leaving precision, offload, tags, and profile ID unset;

- Generator then prefers FP8 when compatible, while Reasoner prefers BF16 on

- compute capability 8.0 through 8.8, FP8 on 8.9 through 9.x, and NVFP4 on 10.0

- or newer; let preflight choose the exact image-specific profile;

- - after cold start, require `examples/inspect_profile.py` to match the selected

- profile from `/v1/metadata` to the YAML embedded in `/v1/manifest`, and use

- the support matrix for documented requirements;

- - use the current RTX 5090 guidance and thresholds in `support-matrix.md` to

- distinguish ordinary generation from Transfer eligibility;

- - explain that preflight/startup takes one free-memory snapshot and can fall

- back only to an equivalent lower-memory layout; an explicit profile pin never

- falls back;

- - use the published floors to choose a candidate without claiming that they

- define fixed general CPU, RAM, disk, shared-memory, driver, Docker, or

- Container Toolkit requirements; and

- - leave requirements without a documented fixed minimum unresolved and require

- cold-start validation rather than inventing a value.

+ When checking hardware, compare each device's total and free memory with the

+ per-device floor and never add memory across devices. On discrete GPUs, require

+ the support matrix's system-memory floor, which profile selection enforces, as

+ a minimum. The

+ tested-GPU inventory is the validation scope, not an allowlist.

### Unified-memory decision procedure

- 1. Run the read-only memory report in

- `prerequisites.md#inspect-unified-memory-capacity-and-current-state` and

- present `MemFree`, `MemAvailable`, `Cached`, `SReclaimable`, `Shmem`, and the

- approximate reclaimable cache. Explain that `MemAvailable` already includes

- reclaimable memory.

- 2. If the customer needs Reasoner, classify the workload before presenting any

- Docker command. Use `NIM_GPU_MEMORY_UTILIZATION=0.80` for image-only or

- `0.70` for video or mixed media. Include the value explicitly in the first

- preflight command and every later Reasoner launch command; the `0.93` default

- is not reduced automatically on the unified pool.

- 3. Compare the requested profile family's floor with effective total capacity

- after the host reserve. If the floor exceeds effective total, report

- **exceeds total capacity**; cache reclamation cannot change that result. For

- example, Reasoner Super BF16 TP1 requires 135 GiB effective memory and cannot

- fit the DGX Spark shared pool.

- 4. If the floor fits effective total but preflight rejects current free memory,

- report **fits hardware but not the current memory state**. Identify active

- workloads and rerun preflight after the operator stops only those workloads.

- 5. Recommend model-level selectors, not an image-specific profile ID. On

- H200/B200-class hardware, a Generator Super request begins with

- `NIM_MODEL_TYPE=generator`, `NIM_MODEL_VARIANT=super`, and

- `NIM_PERF_PROFILE=latency`. On DGX Spark, Thor, H100, or RTX PRO 6000

- Blackwell, recommend `NIM_MODEL_VARIANT=nano` for practical generation

- turnaround unless the customer has an explicit Super requirement.

- 6. Run preflight with those selectors and the intended GPU visibility. Present

- a pass only as candidate-profile compatibility.

- 7. After startup, check readiness and `/v1/metadata`, then run

- `uv run python examples/inspect_profile.py`; report the model selector and

- the actual profile selected from that image.

+ 1. Run the memory report in

+ [prerequisites](../../../prerequisites.md#inspect-unified-memory-capacity-and-current-state)

+ and show `MemFree`, `MemAvailable`, and the cache components. `MemAvailable`

+ already includes reclaimable cache; do not add cache to it, and do not add a

+ separate host-RAM requirement to the shared pool.

+ 2. For Reasoner, apply the memory-share rule above.

+ 3. Effective capacity is 70% of the device-reported total. If the profile floor

+ exceeds it, report **exceeds

+ total capacity** (for example, Reasoner Super BF16 TP1 on DGX Spark). If it

+ fits but preflight rejects current free memory, report **fits hardware but

+ not the current memory state** and ask the operator to stop only the

+ identified workloads before rerunning preflight.

+ 4. Recommend model selectors, not image-specific profile IDs.

- Never clear host page cache, delete model cache, stop unidentified processes,

- or reboot automatically. Present host page-cache reclamation only as an

- approved diagnostic with its host-wide impact, require explicit operator

- confirmation, and collect before-and-after memory and preflight evidence.

+ Never clear page cache, delete model cache, stop unidentified processes, or

+ reboot automatically. Page-cache reclamation is an approved host-wide

+ diagnostic only, with explicit operator confirmation and before/after evidence.

## 4. Route the task

- Use the canonical task page and its committed example instead of constructing a

- new transport or translating another backend's request:

| --- | --- | --- | --- |

- Initialize the pinned client environment through `prerequisites.md` before

- running `uv run python examples/...`. Do not add ad hoc dependencies. The

- Generator examples reject unknown CLI arguments before endpoint or media work,

- then require metadata to identify the Generator runtime, `/v1/infer`, a

- selected profile, and a model variant compatible with the example before they

- submit inference. Do not bypass this preflight. For Reasoner, use

- `examples/reasoner.py --list-cases` or the JSON `--describe`

- mode to select from the complete catalog without contacting an endpoint. Then

- follow `reasoning.md`, preserve the catalog's exact vLLM user-prompt text and

- explicit effective sampling controls, keep its NIM-specific media transport,

- and use the default prompt-constrained output path unless the customer asks to

- exercise NIM guided output. Confirm the selected profile before treating output

- as representative of that deployment. Literal `<think>` instructions in a

- parity prompt are visible response text. Keep API/format validation separate

- from the catalog's qualitative review criteria.

+ Initialize the client environment from `prerequisites.md` first and add no

+ dependencies. Do not bypass the example clients' argument and metadata checks.

+ For Reasoner, pick a case with `examples/reasoner.py --list-cases` or

+ `--describe <case> --format json`, keep the catalog's prompt text and sampling

+ values, and use `--guided-output` only when the customer asks for enforced

+ structured output. Literal `<think>` text requested by a prompt is ordinary

+ response text.

## 5. Troubleshoot safely

- Use `operations.md` as the owner for health, logs, generic errors, resource

- failures, and diagnostics. Use the task page for task-specific validation.

- Collect the minimum sanitized evidence needed:

- - the exact image identifier, without credentials;

- - intended runtime and model variant;

- - redacted `/v1/metadata` and, when configuration is relevant,

- `/v1/manifest`;

- - GPU count, compute capability, and per-device total and free memory;

- - HTTP status and response error envelope; and

- - the relevant container log excerpt with tokens, private URLs, and media

- removed.

- Before retrying a synchronous Generator or Transfer request, check whether it is

- still active and inspect service health and logs. Treat the documented

- 30-minute general Generator and 60-minute Transfer client values as timeout

- ceilings, not expected latency or an SLO. The Generator backend retains a

- separate 30-minute default unless the operator changes it at container launch.

+ `operations.md` owns health, logs, errors, and diagnostics; `monitoring.md` owns

+ metrics; task pages own task-specific validation. Collect only sanitized

+ evidence: image identifier, runtime and variant, redacted `/v1/metadata` (and

+ `/v1/manifest` when configuration matters), GPU count, compute capability and

+ per-device memory, HTTP status and error body, and a redacted log excerpt.

- Do not use unsafe overrides as fixes. If an approved diagnostic requires one,

- state its risk, limit it to that diagnostic, and restore the default afterward.

+ Generator requests are synchronous

+ ([long-running requests](../../../operations.md#long-running-requests)): the

+ examples use 30-minute (Generator) and 60-minute (Transfer) client timeouts as

+ ceilings, not expected latency, and the backend still stops at 30 minutes unless

+ the container sets `NIM_TRITON_REQUEST_TIMEOUT`. Check whether a request is

+ still running before retrying. Do not recommend an unsafe override as a fix. If

+ the customer chooses one for a diagnostic, state the risk, limit it to that

+ diagnostic, and restore the default.

## 6. State the result

- Tell the customer:

- - what was checked;

- - which runtime, page, and example apply;

- - which commands are safe to run next;

- - which assumptions or unresolved release requirements remain; and

- - whether the conclusion comes from documentation or from observed endpoint

- output.

+ Tell the customer what was checked, which runtime, page, and example apply,

+ which commands are safe to run next, which assumptions remain, and whether the

+ conclusion comes from the documentation or from observed endpoint output.