v0.1.0 to v0.1.0

---

name: tao-finetune-video-clip

description: >-

InternVideo2-CLIP L14 (TAO video_clip) for video-text retrieval, zero-shot classification,

embedding extraction, LoRA fine-tuning, and ONNX export. Use when the user asks to

"fine-tune IV2CLIP", "run video_clip train/evaluate/inference/export", "InternVideo2-CLIP

on KPI chunks", or "TAO video_clip on vadr1_chunks JSON".

license: Apache-2.0

compatibility: >-

Requires docker + nvidia-container-toolkit and the pinned TAO video_clip container (see

references/skill_info.yaml), or a local tao-pytorch checkout + tao-cli venv for virtualenv

runs. MobileCLIP + InternVideo2 weights must be on disk for offline eval (HF LFS may be

blocked in CI). Metadata JSON uses vadr1_chunks with absolute video_path entries.

metadata:

author: NVIDIA Corporation

version: "0.1.0"

allowed-tools: Read Bash

tags:

- video

- vision-language

- vlm

- multimodal

- retrieval

- embedding

- internvideo2

- iv2clip

- fine-tuning

---

# InternVideo2-CLIP (TAO video_clip)

> **Standalone install?** If this session was not initialized by the TAO skill bank plugin, run the `tao-setup` skill first (host preflight, credentials, cross-skill discovery).

TAO task **`video_clip`** wraps OpenGVLab **InternVideo2-CLIP L14**. The CLI entrypoint is `video_clip` with actions `train`, `evaluate`, `inference`, `export`, and `default_specs`.

Container image and per-action commands are in `references/skill_info.yaml`. Starting specs are in `references/spec_template_*.yaml`.

> **Release note:** The pinned image is an interim nvstaging build until [tao-pytorch !620](https://gitlab-master.nvidia.com/nvidia-tao-toolkit/tao-pytorch/-/merge_requests/620) merges and TAO publishes an official `video_clip` image tag. After release, `versions.yaml` `images.tao_toolkit.video_clip` will be re-stamped.

>

> **Known-broken images:** interim builds cut before tao-pytorch commit `0cc31de4` (`v7.0.1-pyt2.1.0-py3-01` and `-py3-02`, both superseded by the pinned `-py3-03`) ship a `video_clip` package with no `model.backbones` submodule, so `train`/`evaluate`/`inference` all die at import with `ModuleNotFoundError: nvidia_tao_pytorch.multimodal.video_clip.model.backbones` — while `video_clip --help` still exits 0. Run the import smoke check in the preflight below before pulling data or launching a run.

## Train Action Policy

AutoML is not packaged for this model skill. Always use direct `video_clip` actions even when a higher-level request mentions AutoML. Non-train actions stay in this skill.

## Quick Start (local Docker)

Use the pinned TAO container declared in `references/skill_info.yaml`. Pull with `NGC_KEY` when the image is not cached locally.

```bash

VIDEO_CLIP_IMAGE_DEFAULT="nvcr.io/nvstaging/tao/tao-toolkit-pyt:v7.0.1-pyt2.1.0-py3-03" # versions-key: images.tao_toolkit.video_clip

VIDEO_CLIP_IMAGE="${VIDEO_CLIP_IMAGE:-$VIDEO_CLIP_IMAGE_DEFAULT}"

docker pull "$VIDEO_CLIP_IMAGE"

```

Expected workspace layout (host paths bind-mounted into the container):

```text

workspace/

├── data/

│ ├── train.json # vadr1_chunks metadata; video_path = /data/videos/<name>.mp4

│ ├── val.json

│ ├── prompts.txt # one text prompt per line (inference)

│ └── videos/ # mp4 clips referenced by train.json / val.json

├── model/

│ ├── mobileclip_blt.pt # MobileCLIP weights for model.text_encoder

│ └── hf/ # offline InternVideo2_distillation_models snapshot (HF_HUB_OFFLINE=1)

├── specs/

│ ├── train.yaml

│ ├── evaluate.yaml

│ ├── inference.yaml

│ └── export.yaml

└── results/

```

Docker options for all actions (skill-eval CI uses the same `$WORKSPACE_DIR` bind-mount pattern):

```bash

VIDEO_CLIP_IMAGE_DEFAULT="nvcr.io/nvstaging/tao/tao-toolkit-pyt:v7.0.1-pyt2.1.0-py3-03" # versions-key: images.tao_toolkit.video_clip

VIDEO_CLIP_IMAGE="${VIDEO_CLIP_IMAGE:-$VIDEO_CLIP_IMAGE_DEFAULT}"

RUN_ROOT="${RUN_ROOT:-$PWD}"

DOCKER_COMMON=(

--rm --gpus all --ipc=host --network=host

--shm-size=64g

--ulimit memlock=-1

--ulimit stack=67108864

-e WANDB_DISABLED=true

-e WANDB_MODE=disabled

-e HF_HUB_OFFLINE=1

-e HUGGINGFACE_HUB_CACHE=/model/hf

-e TRANSFORMERS_OFFLINE=1

-e TORCH_FORCE_NO_WEIGHTS_ONLY_LOAD=1

-v "$RUN_ROOT/data:/data:ro"

-v "$RUN_ROOT/model:/model:ro"

-v "$RUN_ROOT/specs:/specs:ro"

-v "$RUN_ROOT/results:/results"

)

```

Preflight (host):

```bash

[ -f "$RUN_ROOT/model/mobileclip_blt.pt" ] || echo "MISSING: MobileCLIP weights"

[ -f "$RUN_ROOT/data/train.json" ] || echo "MISSING: train metadata"

[ -d "$RUN_ROOT/model/hf" ] || echo "MISSING: offline HF snapshot under model/hf"

docker run --rm "$VIDEO_CLIP_IMAGE" video_clip --help >/dev/null || echo "MISSING: video_clip in container"

docker run --rm "$VIDEO_CLIP_IMAGE" \

python -c "import nvidia_tao_pytorch.multimodal.video_clip.model.adapters.internvideo2clip" \

>/dev/null 2>&1 || echo "BROKEN IMAGE: video_clip package is incomplete (missing model.backbones) — stop, see Release note"

docker run --rm "$VIDEO_CLIP_IMAGE" python -c "import decord" \

>/dev/null 2>&1 || echo "NO DECODE BACKEND: decord is not in the image and the container ffmpeg/opencv are codec-disabled — build a derived image (FROM \$VIDEO_CLIP_IMAGE + RUN pip install decord==0.6.0) and run every action from it"

nvidia-smi >/dev/null 2>&1 || echo "note: no GPU visible"

```

Train:

```bash

docker run "${DOCKER_COMMON[@]}" "$VIDEO_CLIP_IMAGE" \

video_clip train -e /specs/train.yaml results_dir=/results

```

Evaluate:

```bash

docker run "${DOCKER_COMMON[@]}" "$VIDEO_CLIP_IMAGE" \

video_clip evaluate -e /specs/evaluate.yaml results_dir=/results

```

Inference:

```bash

docker run "${DOCKER_COMMON[@]}" "$VIDEO_CLIP_IMAGE" \

video_clip inference -e /specs/inference.yaml results_dir=/results

```

Export:

```bash

docker run "${DOCKER_COMMON[@]}" "$VIDEO_CLIP_IMAGE" \

video_clip export -e /specs/export.yaml results_dir=/results

```

## Quick Start (virtualenv — local dev hosts)

On hosts with a `tao-pytorch` checkout and `tao-cli` venv (for example rtdetr-pytorch), run through **`tao-run-on-virtualenv`** instead of Docker:

```bash

export VENV="${VENV:?set to your tao-cli virtualenv (must contain bin/video_clip)}"

export TAO_PYTORCH_ROOT="${TAO_PYTORCH_ROOT:?set to your tao-pytorch checkout with multimodal/video_clip}"

export PATH="$VENV/bin:$PATH"

export PYTHONPATH="$TAO_PYTORCH_ROOT:$TAO_PYTORCH_ROOT/tao-core:$PYTHONPATH"

export HF_HOME="${HF_HOME:-$PWD/hf_cache}"

export HF_HUB_OFFLINE="${HF_HUB_OFFLINE:-1}"

export WANDB_DISABLED=true

export WANDB_MODE=disabled

```

Copy a template spec from `references/spec_template_*.yaml`, fill checkpoint paths and metadata, then run:

```bash

video_clip train -e /path/to/train.yaml

video_clip evaluate -e /path/to/evaluate.yaml

video_clip inference -e /path/to/inference.yaml

video_clip export -e /path/to/export.yaml

```

## Credentials

- **NGC_KEY** — pull the pinned TAO container from `nvcr.io` when it is not cached locally.

- **HF_TOKEN** (only when weights are not already cached on the host):

HuggingFace read token used to resolve the InternVideo2 snapshot named by

`model.internvideo2clip_hf_id`. With `HF_HUB_OFFLINE=1` and local MobileCLIP +

InternVideo2 weights already on disk, no token is required.

Treat tokens as secrets. Export them into the environment or pass them through an

## Data format (vadr1_chunks)

Metadata is a **top-level JSON list** of video records. Each record has `video_path`, `split`, and nested `chunks[]` with caption fields (`queries`, `action_queries`, `anomaly_queries`, `dense_caption`, `scene_caption`).

Point `dataset.*.video_text.metadata` at the user's JSON files (for example `/data/train.json` and `/data/val.json` in the templates). Each `video_path` must be an **absolute** path resolvable inside the runtime — for Docker, remap host clips to container paths like `/data/videos/<name>.mp4` and set `data_root: null` unless using `path_prefix_mapping`.

## Smoke overrides

For a short functional check (for example 2 epochs, 1 GPU, small batch):

```yaml

train.num_epochs: 2

train.num_gpus: 1

train.gpu_ids: [0]

train.optim.warmup_steps: 10

dataset.train.batch_size: 2

dataset.val.batch_size: 2

dataset.train.num_workers: 4

dataset.val.num_workers: 4

dataset.train.video_text.caption_fields: [queries, action_queries, anomaly_queries]

dataset.train.video_text.caption_mode: first

dataset.metrics.mode: classification

```

Use `dataset.metrics.mode: retrieval` only when `dataset.val.video_text.relevance_file` is provided.

## Inference

- `inference.mode: embeddings` writes `video_embeddings.h5` and `text_embeddings.h5` under `results_dir`.

- Provide `inference.query.text_file` (one prompt per line) and/or `inference.query.input_texts`.

- Gallery videos come from `dataset.inference.video_text.metadata`.

## Export

`export.encoder_type: separate` writes vision/text ONNX under `export.onnx_file` (base path). Default opset is **23** on the vendor branch. Requires a trained `.pth` at `export.checkpoint`.

## LoRA

For vision-LoRA runs, start from `tao-pytorch` `experiment_spec_lora.yaml` or add a top-level `peft:` block (see shipped spec comments). Merge LoRA before export when checkpoints contain `lora_*` keys.

## Common pitfalls

- **`PATH` must prefer `$VENV/bin`** on virtualenv hosts so child processes resolve the venv Python.

- **`evaluate` uses `dataset.val`**, not a separate test split — Lightning stage `"test"` still loads val metadata.

- **Eval precision**: match `train.precision` (typically `bf16`) or flash-attn paths may fail under fp32 eval.

- **Empty `action_queries` on normal chunks** become literal `"Normal"` positives during training; exclude `Normal`/`Abnormal` in `dataset.metrics.exclude_categories` for classification eval.

- **No hard-negative / explicit-neg training** on the vendor branch unless the spec and branch explicitly enable it.

- **`video_clip --help` is not a health check.** It exits 0 on an image whose `video_clip` package is missing `model.backbones`; only the import smoke check in the preflight catches it.

- **The pinned image has no working video-decode backend.** `decord` is not installed, the bundled `ffmpeg` is codec-disabled (no `pipe` protocol, no `rawvideo` muxer, no `select` filter), and the `opencv` build has `WITH_FFMPEG=OFF`, so `train` dies at data-loading with `All video decoding backends failed`. Until the base image is rebuilt from [tao-pytorch !620](https://gitlab-master.nvidia.com/nvidia-tao-toolkit/tao-pytorch/-/merge_requests/620) — which pins `decord==0.6.0` in `docker/requirements-pip.txt` — build a derived image (`FROM $VIDEO_CLIP_IMAGE` + `RUN pip install decord==0.6.0`) and run every action from it. A wheel-only release rebuild does **not** fix this: `release/docker/Dockerfile` consumes the base by hardcoded digest, so base-level dependency pins only land after a base-image rebuild.

- **PyTorch ≥ 2.6 defaults to `torch.load(weights_only=True)`** and rejects the TAO checkpoint’s numpy dtype objects with `_pickle.UnpicklingError`. `TORCH_FORCE_NO_WEIGHTS_ONLY_LOAD=1` is set in `DOCKER_COMMON` above; keep it for `evaluate`, `inference`, and `export`.

- **`model/hf/` is a snapshot, not an HF hub cache.** Offline packs stage InternVideo2 weights at repo-relative paths (`stage1/L14/L14_dist_1B_stage2/pytorch_model.bin`, `clip/L14/pytorch_model.bin`), while `HUGGINGFACE_HUB_CACHE` expects a `models--<org>--<repo>/snapshots/<sha>/` tree. Leaving `model.vision_encoder` / `model.clip_head` at `null` sends asset resolution to `hf_hub_download` and fails under `HF_HUB_OFFLINE=1` — point both at the files directly.

- **CI / skill-eval**: stage weights + remapped JSON under `$WORKSPACE_DIR` from S3; do not rely on HuggingFace LFS downloads at eval time.

## References

- Shipped defaults: `tao-pytorch` `nvidia_tao_pytorch/multimodal/video_clip/experiment_specs/` (when developing from source)