tao-finetune-cosmos-reason · diff
v0.1.2 to v0.3.1
228 added, 268 removed. Audit A to A.
---
name: tao-finetune-cosmos-reason
- description: Cosmos3-Nano video QA supervised fine-tuning with FSDP parallelism across variable-width GPU configurations. Use when training or evaluating video
- question-answering models, fine-tuning Cosmos3-Nano or compatible Cosmos Reason models with SFT/LoRA, or working with
- Cosmos-RL. Trigger phrases include "fine-tune Cosmos", "Cosmos3 Nano Reasoner", "Cosmos-RL SFT",
- "video QA fine-tune", "Cosmos3-Nano training".
+ description: >-
+ Shared Cosmos3 frontend that explicitly routes Cosmos Framework and
+ Cosmos-RL, validates runtime model/video-dataset/SLURM inputs, builds clean
+ repository-derived images, prepares checkpoints, gates full training on a
+ smoke run, and returns token-weighted losses and task-aware accuracy.
license: Apache-2.0
- compatibility: Requires Docker, NVIDIA driver 580 or newer, CUDA Toolkit 13.0 or newer, and NVIDIA Container Toolkit 1.19.1 or newer. The selected image must pass its native CUDA-stack smoke test on every visible GPU architecture.
+ compatibility: Docker with NVIDIA Container Toolkit, or SLURM with Pyxis/Enroot and a user-supplied shared-storage configuration.
metadata:
author: NVIDIA Corporation
- version: "0.1.2"
+ version: "0.3.1"
allowed-tools: Read Bash
- tags:
- - video
- - qa
- - cosmos
- - sft
- - reasoning
- - vlm
+ tags: [cosmos, vlm, sft, peft, video, reasoning, slurm]
---
- # Cosmos-RL
-
- > **Standalone install?** If this session was not initialized by the TAO skill bank plugin, run the `tao-setup` skill first (host preflight, credentials, cross-skill discovery).
-
- Supervised fine-tuning (SFT) of Cosmos Reason video QA models. The packaged
- default base model is **hf_model://nvidia/Cosmos3-Nano**. Pretrained weights
- are sourced from HuggingFace, not NGC. Gated HuggingFace models require
- `HF_TOKEN`. Some Cosmos-RL images cannot load the native Cosmos3 Omni checkpoint
- format directly; for those images, convert Cosmos3-Nano to a Qwen3-VL HF
- safetensors directory before train/evaluate and use that converted directory as
- the PTM path.
-
- Uses FSDP-based parallelism with `dp_shard_size` for GPU count and `dp_replicate_size` for node count (not the standard `num_gpus`/`num_nodes`).
-
- Requests for "Cosmos Reason 3", "Cosmos3 Nano Reasoner", or
- `nvidia/Cosmos3-Nano` are handled by this skill. There is no separate Cosmos3
- model directory in the skill bank; route those requests here. Override the base
- HuggingFace model only when the user explicitly asks for a different model.
-
- ## Execution Environment — Hard Gate
-
- `references/skill_info.yaml:container_image` is the default runtime for every
- Cosmos-RL action, including AutoML child jobs. Resolve and preflight that image
- before creating training files, installing training dependencies, downloading
- the checkpoint, or launching a smoke test. A user-supplied `image=<override>`
- replaces the resolved default for that run.
-
- Do **not** create or use a host `venv`, `uv` environment, generic NGC PyTorch
- image, or custom Transformers training project merely because the base model is
- on Hugging Face. Venv-based **training** is allowed only when the user explicitly
- selects venv execution. A host Python environment used to run TAO SDK or the
- AutoML controller is control-plane-only and does not change the child-job
- runtime: training and evaluation still run in the resolved Cosmos-RL image.
-
- If this skill and `tao-finetune-huggingface-model` both appear to match, this
- dedicated model skill wins.
-
- Deep detail lives in references; load the smallest one that matches the task:
-
- - `references/cosmos-reason-launch.md` — launch intake, preflight, per-action dataset requirements, spec construction, typical overrides.
- - `references/cosmos-reason-evaluate.md` — evaluate (flat TOML, task types, LoRA eval, selective download, results) and datasets.
- - `references/cosmos-reason-automl.md` — AutoML/HPO policy and search-space guidance.
- - `references/cosmos-reason-parameters.md` — important parameters, hardware, error patterns, DEFT/gap analysis, parent-model inference mappings.
- - `references/cosmos-reason-wts-gb300.md` — deterministic WTS dataset/runtime helpers and single-GB300 launch guards.
-
- ## Quick Start (docker run)
-
- Docker-native launch in the resolved Cosmos-RL image — no TAO SDK and no host
- Python. This satisfies the Execution Environment hard gate above: the container
- *is* the runtime. Use the local Docker/platform skill instead when it gives a
- stricter environment-specific command.
-
- ```bash
- set -a; source /path/to/.env; set +a # omit if already exported
- COSMOS_RL_IMAGE_DEFAULT=nvcr.io/nvidia/tao/tao-toolkit:7.1.0-cosmos-rl # versions-key: images.tao_toolkit.cosmos_rl
- COSMOS_RL_IMAGE="${COSMOS_RL_IMAGE:-$COSMOS_RL_IMAGE_DEFAULT}"
- RUN_ROOT="${RUN_ROOT:-$PWD}"
- DOCKER_COMMON=(
- --rm --gpus all --ipc=host
- --shm-size=64g
- --ulimit memlock=-1
- --ulimit stack=67108864
- -e HF_TOKEN
- -v "$RUN_ROOT/data:/data:ro"
- -v "$RUN_ROOT/specs:/specs:ro"
- -v "$RUN_ROOT/results:/results"
- )
- ```
-
- Train:
-
- ```bash
- docker run "${DOCKER_COMMON[@]}" "$COSMOS_RL_IMAGE" \
- cosmos-rl --config /specs/train.toml /opt/cosmos_rl/tao_sft_example.py
- ```
-
- Evaluate:
-
- ```bash
- docker run "${DOCKER_COMMON[@]}" "$COSMOS_RL_IMAGE" \
- cosmos-rl-evaluate --config /specs/evaluate.toml
- ```
-
- Specs are TOML for this skill, not YAML. `inference` and `quantize` are driven
- by spec-reading wrappers rather than a single flag-for-flag command; run them
- through the same image and see `references/cosmos-reason-launch.md` for the
- exact form. Pass `HF_TOKEN` as an environment variable — never in a spec or log.
-
- ## Dataclass Schemas
-
- Generated TAO Core schemas are packaged in `schemas/<action>.schema.json`, with `schemas/manifest.json` listing available actions. Each generated schema also emits `references/spec_template_<action>.yaml` from the schema top-level `default` field. AutoML enablement is declared at the model layer in `references/skill_info.yaml` via `automl_enabled`. Runnable AutoML for any selected action requires that action's schema and spec template to exist and parse. Use the packaged selected-action schema for `automl_default_parameters`, `automl_disabled_parameters`, defaults, min/max bounds, enums, option weights, math conditions, dependencies, and popular parameters. Do not expect `~/tao-core` at runtime; maintainers regenerate schemas/templates before packaging the skill bank.
+ # Cosmos3 TAO training
- ## Train Action Policy
+ Keep this as one shared model-facing frontend. Shared concepts live here;
+ backend-native runtime contracts remain separate in
+ `references/cosmos-framework-backend.yaml` and
+ `references/cosmos-rl-backend.yaml`. Never translate one backend's TOML into
+ the other backend's schema.
- This model is AutoML-enabled at the model layer. Before handling any train-stage request, read `references/skill_info.yaml` and resolve the run override from either an explicit `automl_policy` value or the user's workflow request. Use `automl_policy: on` by default and only expose `on` / `off` in new launch prompts. Treat phrases like "turn off AutoML", "disable AutoML", "no HPO", or "plain training" as `automl_policy: off` for this run only. When `automl_policy: on`, `automl_enabled: true`, and both `schemas/train.schema.json` and `references/spec_template_train.yaml` are packaged, route the train action through `tao-skill-bank:tao-run-automl` by default with this model's `skill_dir`. Preserve workflow/application overrides for datasets, specs, output directories, GPU/platform settings, parent checkpoints, and `automl_policy`. Use direct model training only when `automl_policy: off` or the packaged train schema/template is missing; in the missing-schema case, report that AutoML is enabled but not runnable for this model until schemas are generated.
+ ## Mandatory runtime intake
- For non-train actions, run this model skill directly unless the user explicitly asks to optimize that action or sets `automl_policy: on`. When an action schema/template is packaged, route that optimization through `tao-skill-bank:tao-run-automl` with `action=<selected-action>`. The Cosmos `evaluate` action has default AutoML search parameters for zero-shot prompt/config tuning: `dataset.system_prompt`, `vision.nframes`, and generation controls. The per-run `automl_policy` override does not change model metadata.
+ Before planning training, collect all of the following. Do not infer a path
+ from history, another user, a prior job, an image, or a developer checkout.
- ## Credentials
+ - `base_model_path_or_uri`; require `base_model_revision` for a URI/model ID.
+ - `base_model_format`; Nano URI inputs use `qwen3_vl` or `cosmos3_omni`.
+ Cosmos3-Edge is inferred as `cosmos3_edge` from the resolved model ID.
+ - optional `prepared_checkpoint_path`; validate it instead of silently
+ replacing it.
+ - training/validation annotation paths and media roots for conversation-style
+ or task-aware video supervision, plus optional task selection.
+ - explicit `backend` for a comparison; `cosmos-framework` or `cosmos-rl`.
+ - `training_mode`; `dense` or `peft`. PEFT also requires rank, alpha, dropout,
+ target modules, bias, RS-LoRA, modules-to-save, and adapter precision.
+ - user-owned `results_dir`, `checkpoint_dir`, `cache_dir`, and, for SLURM,
+ `sqsh_cache_dir`, `sqsh_path`, `ssh_key_path`, container mounts, and all
+ scheduler settings.
+ - clean repository paths and exact commits for the selected native backend,
+ TAO integration, DAFT, and TAO Core; image tags/base images/build context are
+ runtime inputs.
- - **HF_TOKEN** (required for gated models): HuggingFace access token. For the
- packaged default, the user must accept the model agreement at
- <https://huggingface.co/nvidia/Cosmos3-Nano> and provide a token with read
- access. If the user explicitly overrides the base model, they must accept
- that target model's agreement too. Passed to the container as a
- `docker_env_var`.
+ The planner preserves each original path and reports an accessible `realpath`.
+ Missing paths fail. No fallback dataset, checkpoint, cache, result directory,
+ image, partition, account, mount, SSH key, or shared-storage root is allowed.
- ## Cosmos3 Checkpoint Conversion
+ ### Public Cosmos3-Edge checkpoint contract
- When a selected image cannot load the native Cosmos3 checkpoint format
- (`model_type="cosmos3_omni"` or `Cosmos3ForConditionalGeneration`), do not patch
- QwenVL, Transformers, or vLLM first. Use the upstream Cosmos Framework VLM
- conversion path to produce a Qwen3-VL HF safetensors directory, then point
- Cosmos-RL specs at that converted directory.
+ For Cosmos3-Edge, accept the public Hugging Face model ID plus an immutable
+ revision, or a complete local snapshot of that same public model. Do not ask
+ the user for a second workload-specific checkpoint and do not copy or edit
+ `processor_config.json` or `video_preprocessor_config.json` to create one.
+ Keep the base-model fingerprint separate from the processor-profile
+ fingerprint.
- The model skill packages a helper:
+ The model-aware TAO profile supplies Edge runtime defaults: 6 sampled video
+ frames, a 1280 x 720 per-frame reference budget (5,529,600 aggregate video
+ pixels), sequence length 16,000, and `flash_attention_2`. These are runtime
+ settings, not checkpoint contents. Record whether each value came from the
+ skill profile or an explicit user override, include it in parity and cache
+ keys, and require the normal compute-node smoke gate before full training.
+ Framework receives the pixel budget through `TAO_VIDEO_MAX_PIXELS`; Nano keeps its native processor limit unless the user
+ explicitly overrides it.
- ```bash
- set -a; source /path/to/.env; set +a # omit if already exported
- python skills/models/tao-finetune-cosmos-reason/scripts/prepare_cosmos3_vlm_checkpoint.py \
- --checkpoint-path /abs/path/Cosmos3-Nano \
- --output-path /abs/path/Cosmos3-Nano-VLM \
- --validate-with-image <cosmos-rl-image>
- ```
+ ## Backend selection
- The helper forwards `HF_TOKEN` / `HUGGING_FACE_HUB_TOKEN` from the session by
- name, so `--secrets-env` is unset by default. A path given there is handed to
- docker `--env-file` (bare `KEY=value` lines only) and its whole contents enter
- the conversion container, so it must hold nothing but the HuggingFace token.
+ Run `scripts/cosmos_workflow.py resolve` first.
- After conversion, use the converted directory consistently as the PTM:
+ | Request | Automatic selection |
+ |---|---|
+ | Cosmos3-Nano plain train | Cosmos-RL (compatibility default) |
+ | Cosmos3-Nano AutoML/HPO | Cosmos-RL |
+ | Nano Framework-DCP export | Cosmos Framework |
+ | Nano evaluate/inference/microservice with no explicit backend | Cosmos-RL |
+ | Nano quantize | Cosmos-RL |
+ | Cosmos3-Edge train/export/evaluate/inference/microservice | Cosmos Framework |
- ```text
- train: policy.model_name_or_path=/abs/path/Cosmos3-Nano-VLM
- evaluate: model.model_name=/abs/path/Cosmos3-Nano-VLM
- evaluate: model.base_model_path=/abs/path/Cosmos3-Nano-VLM
- ```
+ An explicit supported backend wins, so users can select Cosmos Framework for
+ Nano training without changing model ownership. Comparative runs reject
+ `auto`, so both sides of an experiment are deliberately forced.
+ Framework-trained checkpoints use the native exact-key exporter, then the
+ repository-backed TAO evaluation adapter. That does not make Framework a
+ Cosmos-RL version.
- For local Docker, mount the converted directory read-only into the Cosmos-RL
- container and set the spec to the container path. If a converted copy already
- exists and validates, reuse it for PTM baseline evaluation, AutoML
- recommendations, and final best-checkpoint evaluation rather than converting
- again.
+ ## Framework checkpoint pre-action
- ## Training Requirements
+ Before every Framework `evaluate`, `inference`, or `inference_microservice`
+ action, invoke `scripts/framework_checkpoint_action.py plan`. Do not ask the
+ user to find or run an exporter manually. The helper preserves the supplied
+ checkpoint path, detects a complete HF safetensors directory versus a native
+ Framework DCP, infers the saved Framework config only from the checkpoint's
+ standard run layout, and otherwise requires an explicit `config_file`.
- The model-level GPU host requirements in `references/skill_info.yaml` override
- the TAO-wide platform defaults for this workflow. Apply that profile through
- the shared host setup check before launch, as shown in
- `references/cosmos-reason-launch.md`. Treat the versions as lower bounds, not
- exact pins, and retain the selected-image native GPU smoke test.
+ For DCP input, run the helper's `prepare` verb in the newly built Framework
+ TAO action image before starting the requested action. Its runner may be local
+ Docker or an `srun` Pyxis command supplied through `--command-prefix`. On
+ SLURM, stage this checked-in helper and `cosmos_common.py` with checksums in the
+ job input directory; mount only that job directory, not a source checkout.
+ The helper invokes the repository-owned exact-key exporter, verifies the DCP
+ metadata/config/base-model/revision/exported weights and manifest, writes
+ `.tao_export_complete`, and returns `action_model_path`. Put that verified path
+ into `model.model_name` for evaluation or `model_path` for inference and the
+ microservice. Run `verify` once more from the target compute frame before the
+ action child starts.
- - **Dataset type:** vlm
- - **Formats:** llava, daft
- - **Accepted dataset intents:** training, evaluation, testing
- - **Monitoring metric:** `BERTScore_F1` for the default evaluation-backed AutoML free-form text workflow; maximize it and use the same evaluator metric for the baseline, every recommendation, and final best-checkpoint evaluation.
- - **Training-only monitoring metrics:** `val/avg_loss`, `val/reward_avg`, `val/loss`; these are useful for training observability but are not emitted by the evaluate action and must not replace `BERTScore_F1` in the default AutoML workflow.
- - **Evaluate task metrics:** binary `accuracy`/balanced accuracy/precision/recall/F1, or text `BLEU`, `ROUGE*`, and `BERTScore_F1`. Monitor the exact scalar `BERTScore_F1` for free-form text; `BERTScore` is not an emitted key. The evaluate action does not emit `val/avg_loss`.
- - **AutoML metric contract:** do not use an evaluate job as a baseline for `val/avg_loss`. Either optimize an evaluate task metric consistently for baseline, every recommendation (`eval_fn`), and final evaluation, or obtain explicit approval for a training-loss proxy run without an impact baseline.
- - **Dataset URI examples:** `s3://bucket/cosmos/train`, `s3://bucket/cosmos/eval`, `/lustre/fsw/tao_datasets/cosmos_rl/train`, `/lustre/fsw/tao_datasets/cosmos_rl/eval`
- - **Input modes:** accept either dataset roots or direct spec-key paths. Root mode maps `<root>/annotations.json` plus `<root>` as the media path. Direct spec mode is valid when annotations and media live in different locations, for example `custom.train_dataset.annotation_path=/lustre/.../train.json` and `custom.train_dataset.media_path=/lustre/.../videos.tar.gz`.
- - **Media handling:** do not ask the user to choose `videos.tar.gz` vs `images.tar.gz` unless they are using direct spec mode or the model/action requires a single media archive. In root mode, pass the dataset root as the media path.
- - **Annotation validation:** before launching train/AutoML/evaluate, verify the
- annotation JSON is readable and the referenced media path or archive is
- visible from the selected platform. Do not block, patch, or mutate
- annotations solely because optional fields are absent.
- - **Per-record video FPS:** the packaged train template uses
- `custom.vision.nframes`, so per-record `video_fps` is not required by
- default for SFT training. The `7.0.1-cosmos-rl` evaluate action still reads
- per-record `video_fps` during video input preparation even when
- `vision.nframes` is set, so require `video_fps` for evaluate-backed baseline,
- per-recommendation, and final AutoML metrics with that image. If the user
- switches training to `custom.vision.fps`, selects a dataset
- profile that requires per-record timing, or uses an image/version that
- requires `video_fps`, make it a preflight requirement with
- `--json-required-field train_annotation=video_fps` and
- `--json-required-field val_annotation=video_fps` before any download or
- job launch.
+ An already verified export is reused without conversion. A stale or partial
+ export is never silently accepted: `prepare` creates and verifies a sibling
+ temporary export, preserves the invalid directory under an `.invalid-*` name,
+ then atomically installs the replacement. Capture the pre-action JSON, child
+ exit code, export manifest checksum, source checkpoint/config fingerprints,
+ and final action child code in the job metadata. A failed export blocks the
+ evaluate or inference allocation/action and emits terminal `FAILURE`.
- ## Spec construction
+ ## Required gates
- cosmos-rl is `mode: config`. **Always start from the packaged
- `references/spec_template_<action>.yaml` for the requested action** — load it
- as your base spec via `yaml.safe_load(...)` and apply user overrides on top.
- Don't rebuild from scratch.
+ Execute these stages in order and persist their outputs.
- ```python
- import yaml
- from pathlib import Path
+ 1. Resolve model/backend/action and load the selected backend contract.
+ 2. Check credentials by presence only. Never read or persist credential
+ values. Require a token only for the operation that needs it.
+ 3. Validate host tools, clean repository commits/trees, build context, free
+ storage, every original/resolved path, and image build inputs.
+ 4. Build the selected native image and TAO action image from clean commits.
+ Inspect `/opt/tao/image-provenance.json`; reject dirty, missing, or mismatched
+ source. Never mount a host source checkout into training.
+ 5. If needed, prepare the model with the converter packaged in the clean
+ Framework image. URI downloads require immutable revisions. Validate exact
+ tensor/config keys and fingerprint model, tokenizer, and processor files.
+ 6. Validate every annotation and referenced media file, record counts,
+ duplicates, train/validation overlap, task selection, and fingerprints.
+ Verify the resolved inputs again from an allocated compute node.
+ When SLURM storage is not mounted on the launch host, let
+ `cosmos_workflow.py` stream its checked-in `cosmos_common.py` inspector to a
+ login host over SSH. It runs from stdin, preserves remote `realpath` values,
+ and creates no remote script or source overlay. Do not require local Lustre,
+ `sbatch`, or `srun` on an SSH-based launch host. Run this expensive input
+ inspection exactly once with the `plan` verb and pass a local
+ `--plan-artifact <path>` so the resolved request and inspection results are
+ sealed for the remaining launch verbs.
+ 7. Prepare the decoder input selected by the structural dataset contract.
+ Cosmos-RL defaults to direct on-demand sample processing so training starts
+ without a dataset-cache prewarm phase. Conversation-style runs prewarm only
+ when the user explicitly selects `--rl-dataset-cache-mode prewarm`; that
+ opt-in uses separate deterministic train and validation keys. Task-aware
+ runs decode directly on GPU and require a fingerprinted override artifact
+ from the packaged
+ `cosmos_rl.utils.video_override_artifacts` builder. Pair every annotation
+ with its actual media root, scan the full input set for the NVDEC macroblock
+ limit, force every validation annotation with `--force-annotation`, and add
+ only independently diagnosed train streams with `--force-video`. Validate
+ the artifact with the packaged
+ `cosmos_rl.utils.validate_video_override_artifacts` command, including full
+ validation-media coverage and GPU random-access decoding, before smoke.
+ The JSON plan emits exact `decoder_artifact.preparation_command` and
+ `decoder_artifact.validation_command` values for the selected clean image;
+ re-plan once with their map, manifest, and artifact fingerprint outputs,
+ seal that plan, and never reuse another run's cache or override artifact.
+ 8. Generate backend-native TOML, environment, topology, preflight commands,
+ parity data, and machine-readable job metadata. Full specs must contain no
+ sample limit. `plan` and `preflight` are read-only with respect to the
+ target compute frame; the explicitly requested controller-side plan artifact
+ is their handoff. Invoke `preflight`, post-review `materialize`, and
+ `render-slurm` with that same `--plan-artifact` and no repeated original
+ input arguments. `materialize` atomically creates the TOML and any
+ merged/smoke manifest in the verified compute frame. For SSH-based
+ SLURM, the checked-in helper is streamed to the verified login host and the
+ generated files are written directly to user-supplied shared storage; do
+ not read a remote annotation through the launch host or copy a temporary
+ source patch to the cluster. The planner derives all in-container runtime
+ paths from explicit mount mappings and rejects an explicit mapping mismatch.
+ 9. Convert the newly built image to a new SQSH when SLURM is selected. Record
+ image ID/digest and SQSH SHA256; verify Pyxis/Enroot, mounts, non-root Python,
+ packages, decoder, GPU memory/type, CUDA/PyTorch, NCCL, and storage on the
+ allocated node.
+ 10. Run a smoke job for every distinct backend × structural dataset family × training
+ mode × checkpoint/evaluator path. Continue only on child exit zero,
+ structured `SUCCESS`, finite global train/validation loss, checkpoint
+ completion, and evaluator accuracy coverage.
+ 11. Create one fresh sealed full plan after the smoke gate, with all smoke
+ limits removed. Materialize its full spec once and verify its SHA256 in the
+ compute frame before rendering the job from the same plan artifact. Launch with
+ `afterok` only after the smoke gate, monitor scheduler and structured TAO
+ state to a terminal result, and preserve the child exit code independently
+ of scheduler state.
+ 12. Export/evaluate the selected checkpoint with identical prompt,
+ preprocessing, generation, normalization, and task scoring. Extract final
+ metrics with `scripts/extract_cosmos_metrics.py`.
- skill = Path.home() / "tao-sdk/tao-skills-external/skills/models/tao-finetune-cosmos-reason"
- action = "train" # train, evaluate, inference, or quantize
- specs = yaml.safe_load((skill / f"references/spec_template_{action}.yaml").read_text())
- # Now apply your overrides on top of `specs`.
- ```
+ ## Dataset contracts
- The reference TOML (and the spec the model actually consumes) is **nested
- dicts**, not flat dotted keys. Dotted notation in override examples denotes
- *paths into the nested spec* — walk the path and assign at the leaf, not store
- the dotted string as a literal key. Data source overrides are **mandatory for
- every action**.
+ Resolve datasets by structure, not by project, benchmark, directory, or file
+ name. The supported families are:
- The packaged template keeps `custom.vision.nframes=8` for bounded 1-GPU memory;
- switch to `fps` only after checking token budget and GPU memory, and delete
- `custom.vision.nframes` from the spec when you do.
+ - `video_conversation`: a JSON array with media and at least two ShareGPT,
+ LLaVA, or OpenAI-style conversation turns;
+ - `task_aware_video_reasoning`: one or more item-envelope or array annotation
+ files with media, task identity, and conversation/response targets.
- ### Epoch-based cadence policy
+ Default `dataset_family` to `auto`, inspect every annotation, and require train
+ and validation to resolve to the same family. Capture record count, unique
+ media count, media reuse, extensions, byte-size distribution, task/metric
+ metadata, and any declared width, height, FPS, and duration. Select processor,
+ cache, smoke-size, and resource profiles from those characteristics and the
+ model tier. Never branch on a customer dataset name.
- Training duration, checkpointing, and validation are epoch-based by default:
+ Tasks declaring accuracy participate in deterministic accuracy; common binary
+ and multiple-choice task types are recognized. Generative tasks report their
+ declared metrics and are excluded from aggregate accuracy with a reason.
+ Aggregate accuracy is example-weighted over records with an accuracy definition.
- - Set training duration with `train.epoch`.
- - Keep `train.ckpt.save_freq_in_epoch=1` and do not disable it or replace it
- with `train.ckpt.save_freq` unless the user explicitly requests step-based
- checkpointing.
- - Keep `validation.freq_in_epoch=1` and do not replace it with
- `validation.freq` unless the user explicitly requests step-based validation.
+ ## Dense and PEFT contracts
- Dataset, GPU, topology, memory, and runtime-image guidance must not change
- these cadence defaults. A hardware- or dataset-specific reference may explain
- how to implement an explicitly requested step cadence, but must never infer or
- select step cadence on the user's behalf.
+ Dense SFT must have no active LoRA block and must report trainable, frozen, and
+ total parameter counts. PEFT must represent the same rank, alpha, dropout,
+ target modules, bias, RS-LoRA, modules-to-save, precision, and trainable count
+ on both backends. Reject a paired PEFT run when semantics cannot be matched.
- See `references/cosmos-reason-launch.md` for launch intake, the full
- `check_tao_launch_preflight.py` slurm/local-Docker examples, the
- `video_fps` preflight example, S3 staging, the GPU resource/architecture gate,
- the per-action dataset requirements table, the `/workspace` mount caveat,
- the quantize compatibility shim, and the full typical-overrides list.
+ For fair comparisons, force the same logical model, train/validation records,
+ media, prompt, frames, sequence length, precision, seed, epochs, effective
+ global batch, optimizer, learning rate, schedule, warmup, weight decay,
+ clipping, loss masking, validation/checkpoint cadence, evaluated checkpoint,
+ and generation/normalization settings. Classify differences as equivalent
+ syntax, unavoidable implementation difference, or invalid mismatch; an invalid
+ mismatch blocks the full pair.
- ## Critical Overrides (Train)
+ ## Metrics and completion
- These are the keys whose template defaults are wrong or where omission flips the run into a different mode:
+ The required primary metrics are:
- | Parameter | Template Default | Required Value | Why |
- |---|---|---|---|
- | `policy.model_name_or_path` | `hf_model://nvidia/Cosmos3-Nano` | Direct Docker: `nvidia/Cosmos3-Nano`, `hf_model://nvidia/Cosmos3-Nano`, or a local HF snapshot path. SDK/managed platform predownload: `hf_model://nvidia/Cosmos3-Nano`. | Keep the train and evaluate base model aligned. |
- | `policy.model_max_length` | 40960 | Keep at 40960 or higher | Smaller than ~40k causes `vision_embeds` shape mismatch on video inputs |
- | `train.train_batch_per_replica` | 32 | Any multiple of `train.train_policy.mini_batch` | Mismatch raises an immediate AssertionError |
- | `train.train_policy.type` | `"sft"` | Keep as `"sft"` for SFT workflows | If dropped during agent regeneration, cosmos-rl flips to RL mode → rollout replica allocated → multi-node attempted → hostname errors when `num_nodes=1` |
+ - complete-run globally reduced token-weighted training loss, with numerator
+ and valid-label denominator;
+ - final-validation globally reduced token-weighted loss, with numerator and
+ valid-label denominator;
+ - repository-evaluator validation accuracy, with correct/total, coverage,
+ per-task metrics, aggregation definition, exclusions, and evaluator version.
- ## Evaluate
+ Do not average console lines or rank means. A step loss is not the average
+ training loss. A validation heartbeat is not final validation loss. A
+ generative exact match is not accuracy unless the task defines it.
- The evaluator reads a flat TOML config (`dataset`, `model`, `task`,
- `evaluation`, `vision`, `generation`, `metrics`, `results`, `num_gpus`,
- `results_dir`); the `actions.evaluate` block in `references/skill_info.yaml`
- declares inputs and outputs. See `references/cosmos-reason-evaluate.md` for the
- flat-TOML config detail, task types (`""` General Evaluator vs
- `"its_directionality"`), LoRA evaluation via spec_overrides, selective download,
- results/metrics, and the datasets section.
+ The native Framework callback and native Cosmos-RL logger own early failure,
+ checkpoint, progress, metric, and terminal events. Do not stage a status bridge
+ or patch status at container startup. `COMPLETED` from SLURM is failure when the
+ child exit code is nonzero or the terminal TAO state is not successful.
- ## AutoML / HPO Notes
+ ## SLURM invariants
- **Baseline eval is required AutoML setup, not an optional question for the user.**
- Before launching AutoML for an accuracy objective, run the model's evaluate
- action once after preflight and before any recommendation jobs, on the same
- validation subset — using the selected base model or starting checkpoint,
- `task=""`, and the same prompt/metric setup planned for per-recommendation
- evaluation. Report that eval job id, result path, and accuracy in the launch
- review before asking for confirmation to start recommendations. The final AutoML
- summary must compare this baseline accuracy, every recommendation's accuracy, and
- the selected best recommendation.
+ Generated jobs use `#!/usr/bin/env bash` and are syntax-checked by Bash. They
+ use SQSH via Pyxis, disable requeue by default, run one launcher task per node,
+ write stdout/stderr to runtime-supplied paths, and exit with the training child
+ code. Framework topology is shard=`gpus_per_node`, replica=`nodes`.
+ Cosmos-RL uses one controller on node zero and its policy-worker topology.
+ Asynchronous distributed checkpointing is rejected for multi-node runs.
- The packaged default base model is `hf_model://nvidia/Cosmos3-Nano`; apply it
- consistently to train (`policy.model_name_or_path`) and post-training evaluation
- (`model.base_model_path`) unless the user provides a different model. See
- `references/cosmos-reason-automl.md` for accuracy-vs-`val/avg_loss` objective
- selection, the `eval_fn` per-recommendation evaluate flow, the knob mapping
- (learning rate, batch size, epochs, weight decay, warmup ratio), example
- `custom_param_ranges`, `train_sample_count` batch-size capping,
- `ordered_int` requirements, zero-shot evaluate AutoML prompt/config search,
- and the pre-launch recommendation summary.
+ Every job metadata record must validate against
+ `schemas/cosmos-job-metadata.schema.json`. It records supplied/resolved paths,
+ model/data/image fingerprints, source commits, config and SQSH checksums,
+ requested/allocated topology, scheduler and child states, logs/results,
+ timestamps, and terminal TAO status without credentials.
- ## Parameters, Hardware, Errors, DEFT, Inference
+ ## Source-affecting recovery
- For parallelism, set `policy.parallelism.dp_shard_size` = GPUs per node and
- `policy.parallelism.dp_replicate_size` = node count (1 for single node).
- Cosmos-RL handles distributed init internally via FSDP and does not rely on
- platform-level `MASTER_ADDR`/`WORLD_SIZE`; submit with
- `gpu_count=<gpus_per_node>` and `num_nodes=<N>` and the spec keys drive
- sharding. SFT requires at least 256 GB of cumulative visible GPU memory rather
- than a fixed device count or per-device capacity. Set `dp_shard_size` and the
- platform GPU request to the actual visible GPU count, and use
- `dp_replicate_size=1` for a single node. The selected image must support every
- visible GPU architecture and pass its runtime CUDA-stack smoke test.
+ If a run exposes a code or image defect, stop the affected path, change the
+ owning repository, add a test, commit it, rebuild both image and SQSH from a
+ clean checkout, rerun smoke, and rerun every affected full job. Never edit a
+ running container, patch an existing image, reuse an old SQSH after a source
+ change, or rely on a temporary launch script as the implementation.
- See `references/cosmos-reason-parameters.md` for important parameters (training
- loop, model/policy, parallelism incl. multi-node FSDP, optimization, vision
- encoders, checkpointing incl. the `best` symlink/`epoch_*` resolution,
- validation, logging), hardware sizing, the full error-pattern catalog (CUDA OOM,
- LoRA-eval OOM, NaN loss, `vision_embeds` mismatch, quantize token mismatch,
- batch-size divisibility and per-rank limits, stale cache, scheduler-None,
- gated-repo `HF_TOKEN`, GPU resource/architecture gate, status-logging warnings),
- DEFT support and `scripts/analyze_gaps.py` gap analysis, and the parent-model
- inference mapping table.
+ Use `references/cosmos-reproducibility-gates.md` as the source-owner and test
+ map before proposing a workaround in a fresh session.