tao-run-deft-cr-its-mining v0.1.0A

Immutable. This exact content never changes and is served at /api/v1/blob/97de210ef09785b2.

---
name: tao-run-deft-cr-its-mining
description: >-
  Run the mining-based DEFT improvement workflow for ITS Cosmos-Reason binary video questions,
  focused on the non-reasoning classification/evaluation path. Use when the user asks for a
  DEFT CR ITS mining workflow, traffic-camera Cosmos Reason improvement loop, collision-identification workflow with data mining, or iterative Cosmos-RL refinement driven by gap analysis.
license: Apache-2.0
compatibility: Requires docker + nvidia-container-toolkit. Workflows declare additional requirements.
metadata:
  author: NVIDIA Corporation
  version: "0.1.0"
allowed-tools: Read Bash Write Task
tags:
- application
- workflow
- deft
- its
- cosmos-reason
- cosmos-rl
- mining
---

# Skill: TAO Run DEFT CR ITS Mining

## Bundled Resources

Resolve `DEFT_SKILL_ROOT` to the absolute directory containing this installed `SKILL.md`. The agent or plugin runtime resolves this path; it is not a user input. Run bundled helpers with `run_script("scripts/<name>.py", ...)` when the runtime provides it. Otherwise invoke them directly with `python3 "$DEFT_SKILL_ROOT/scripts/<name>.py"`. Never require a `tao-skills-external` checkout or change the user's working directory to a repository root.

This workflow invokes `tao-finetune-cosmos-reason`, `tao-finetune-cosmos-embed`, `tao-analyze-gaps-vlm-bcq`, `tao-mine-nearest-neighbors`, and the selected platform skill by registered skill name. Those skills own their commands, credentials, and bundled assets. This workflow overrides only the Cosmos Reason runtime image: resolve `images.tao_toolkit.deft_cosmos_reason` from `versions.yaml` and pass that image as the planner `image_tag` and submitted action image for every Cosmos Reason train and evaluate launch instead of the image declared by `tao-finetune-cosmos-reason`. This workflow owns only the helpers under its own `scripts/` and `assets/` directories.

## User Inputs (DEFT Workspace and Workflow Configuration Yaml)

The user should provide a DEFT workspace absolute path that will be used for the entire run. This path should not be `/workspace` because that conflicts with the Cosmos Reason skills.

The layout of the DEFT workspace should be as follows:

```text
<deft_workspace>/
├── data/
├── hf_cache/
├── model/
├── specs/
└── results/
```

| Path | Purpose |
| --- | --- |
| `<deft_workspace>/data/` | User-provided KPI and training datasets, including LLaVA annotations and media directories. |
| `<deft_workspace>/hf_cache/` | Persistent Hugging Face cache used when workflow steps download or reuse HF models. |
| `<deft_workspace>/model/` | Local model/checkpoint inputs for the workflow, including the baseline Cosmos Reason checkpoint and any user-provided Cosmos Embed checkpoint. |
| `<deft_workspace>/specs/` | User-provided configuration files, including templates for Cosmos Reason train and evaluate, Cosmos Embed inference, and TAO Data Services mining. |
| `<deft_workspace>/results/` | All workflow-generated run artifacts, including state, logs, baseline evaluation, Cosmos Embed outputs, mining outputs, and iteration directories. |

Each workflow run writes under a run-specific directory inside `<deft_workspace>/results/`. If `run.name` is set in `workflow.yaml`, that exact value is used as the directory name; otherwise the workflow creates a timestamped directory. For example, `run.name: deft_cr_its_mining_test` writes baseline evaluation under `<deft_workspace>/results/deft_cr_its_mining_test/baseline/evaluate`.

The run directory layout is:

```text
<deft_workspace>/results/<run_name_or_timestamp>/
├── workflow.yaml
├── deft_state.json
├── loop_log.jsonl
├── bcq_accuracy_report.md
├── bcq_accuracy_summary.json
├── baseline/
│   └── evaluate/
│       └── bcq_accuracy_metrics.json
├── cosmos_embed_output/
│   ├── kpi/
│   └── train/
├── embedding_parquets/
│   ├── kpi/
│   │   └── embeddings.parquet
│   └── train/
│       └── embeddings.parquet
├── iter_1/
│   └── evaluate/
│       └── bcq_accuracy_metrics.json
├── iter_2/
└── ...
```

`workflow.yaml` is a copy of the high-level config that produced the run. `loop_log.jsonl` is the append-only event source for resume, and every stage-log append atomically refreshes `deft_state.json` as its current snapshot. `baseline/evaluate/` stores the initial evaluation of the baseline model. Each completed evaluation also stores `bcq_accuracy_metrics.json`. The run-level `bcq_accuracy_report.md` and `bcq_accuracy_summary.json` compare the baseline with every completed iteration. `cosmos_embed_output/kpi/` and `cosmos_embed_output/train/` store raw Cosmos Embed preparation and inference outputs. `embedding_parquets/kpi/` and `embedding_parquets/train/` store the fixed mining-ready parquet artifacts computed once from the KPI and train datasets. Each `iter_<N>/` directory stores the artifacts for one DEFT improvement iteration.

### Workflow Configuration

There should be a file called `<deft_workspace>/specs/workflow.yaml`, which will be the high-level workflow config for the full DEFT run:

```yaml
run:
  name: null
  max_iterations: 1

kpi_dataset:
  annotations_path: /abs/path/to/deft_workspace/data/kpi/annotations.json
  media_dir: /abs/path/to/deft_workspace/data/kpi/media

train_dataset:
  annotations_path: /abs/path/to/deft_workspace/data/train/annotations.json
  media_dir: /abs/path/to/deft_workspace/data/train/media

cosmos_reason:
  baseline_model_path: /abs/path/to/deft_workspace/model/baseline
  base_evaluate_toml: /abs/path/to/deft_workspace/specs/cr_base_evaluate.toml
  base_train_toml: /abs/path/to/deft_workspace/specs/cr_base_train.toml
  continual_model: false

mining:
  embeddings_spec_template: /abs/path/to/deft_workspace/specs/cosmos_embed_inference_template.yaml
  embeddings_modality: both
  cosmos_embed_checkpoint_path: null
  embedding_parquets:
    kpi: null
    train: null
  mining_spec_template: /abs/path/to/deft_workspace/specs/mining_spec_template.yaml
  mine_unique_only: true
```

`run`, `kpi_dataset`, `train_dataset`, `cosmos_reason`, and `mining` are required. All configured paths should be absolute and inside `<deft_workspace>`. `cosmos_reason.continual_model` is optional and defaults to `false`; when false, every iteration trains from the baseline checkpoint. `mining.embeddings_modality` selects KPI target modalities; train/source embeddings always contain both text and video. `mining.mine_unique_only` is optional and defaults to `true`; when true, previously mined train/source filepaths are filtered out before later mining runs.

All workflow outputs go under `<deft_workspace>/results`; do not add a separate output root to `workflow.yaml`. `run.name` is not just a display label: if it is set, use `<deft_workspace>/results/<run.name>` as the run directory and explain that all stage outputs, including `baseline/`, `cosmos_embed_output/`, `embedding_parquets/`, and `iter_<N>/`, are nested under that run directory. If `run.name` is `null`, create `<deft_workspace>/results/run_<YYYYMMDD_HHMMSS>` and record the resolved run directory in `deft_state.json` so resume never recomputes it.

Before running any workflow stage, validate the config:

```bash
python3 "$DEFT_SKILL_ROOT/scripts/verify_workflow_yaml.py" \
  --workspace "$WORKSPACE" \
  --workflow-yaml "$WORKSPACE/specs/workflow.yaml"
```

The validator checks that required fields are present, configured paths are absolute and inside `<deft_workspace>`, `embeddings_modality` is `text`, `video`, or `both`, every configured path exists, and the KPI/train LLaVA annotations are compatible with their configured `media_dir`. Every annotation must have a unique non-empty `id`; its `video` field must resolve to an existing file against that dataset's `media_dir`. The Cosmos Embed template must declare a positive `inference.num_gpus`; preflight prints that count so the agent can verify available hardware before launch. `mining.cosmos_embed_checkpoint_path` may be `null`, an absolute local path inside `<deft_workspace>`, or a Hugging Face model id such as `nvidia/Cosmos-Embed1-224p` or `hf_model://nvidia/Cosmos-Embed1-224p`. Optional `mining.embedding_parquets.kpi` and `.train` values must be absolute existing mining-ready Parquet files inside the workspace with `filepath`, `embedding`, and `modality` columns. The KPI file must contain exactly the selected modality or modalities; the train file must contain both text and video. The removed `text_embeddings` and `video_embeddings` fields are invalid. This validates the workflow input contract; each stage still follows the mount behavior of the underlying skill, container, or platform runner.

If the user does not provide custom Cosmos Reason or Cosmos Embed templates, copy `$DEFT_SKILL_ROOT/assets/cr_base_evaluate.toml`, `$DEFT_SKILL_ROOT/assets/cr_base_train.toml`, and `$DEFT_SKILL_ROOT/assets/default_cosmos_embed_inference.yaml` into `<deft_workspace>/specs/`. Use `tao-mine-nearest-neighbors` to copy its bundled `assets/default_nearest_neighbors.yaml` when the user does not provide a mining template. Point `workflow.yaml` at the workspace copies. The bundled Cosmos Embed template requests 8 GPUs; do not launch until preflight confirms the selected platform can satisfy that request, or the user approves a modified template.

Initialize a run once after validation:

```bash
python3 "$DEFT_SKILL_ROOT/scripts/initialize_workflow.py" \
  --workspace "$WORKSPACE" \
  --workflow-yaml "$WORKSPACE/specs/workflow.yaml"
```

Use the printed `run_dir` as `RUN_DIR` for every later command. If `run.name` is `null`, never ask another preparation script to derive the run directory again; pass `--run-dir "$RUN_DIR"` so all stages write into the initialized run.

Do not use `--force` for an ordinary resume. It rewrites the state/config snapshot but intentionally leaves `loop_log.jsonl` and all stage artifacts in place. Use it only to repair the snapshot for the same run; choose a new `run.name` for a clean restart. Before resuming, run `$DEFT_SKILL_ROOT/scripts/resume_position.py --run-dir "$RUN_DIR"` and continue from the reported stage.

## Baseline Evaluation

Run baseline evaluation once before the DEFT iteration loop. This evaluates `cosmos_reason.baseline_model_path` on the KPI dataset and produces the first `results.json` used by gap analysis. Immediately run `scripts/compute_bcq_accuracy_metrics.py` on that file and store `baseline/evaluate/bcq_accuracy_metrics.json`. Its parser recognizes `yes` and `no` in short or free-form responses, including capitalization and punctuation. It reports false positives, false negatives, accuracy, balanced accuracy, and unparseable predictions. Use `references/mining-loop.md` for the exact commands, completion check, and logging rule.

## Initialize Mining Embeddings

Initialize mining embeddings once before the DEFT iteration loop. The KPI and train datasets are fixed for the run, so their Cosmos Embed outputs are reusable across all iterations.

Use `scripts/prepare_cosmos_embed_inference.py` to prepare the Cosmos Embed inference specs from `workflow.yaml`. The KPI dataset requires the modality or modalities selected by `mining.embeddings_modality`; the train dataset always requires text and video. If the user provides a complete dataset Parquet under `mining.embedding_parquets.kpi` or `.train`, the preparation script stages it at `$RUN_DIR/embedding_parquets/<dataset>/embeddings.parquet` and generates no inference specs for that dataset. While staging, it remaps text embedding identifiers from the source run's question files to the current run's lookup by question content; embeddings and video identifiers are preserved. If the dataset value is `null` or omitted, the script generates every required modality spec for that dataset. Partial per-modality reuse is not supported. If `mining.cosmos_embed_checkpoint_path` is an absolute local path, the script uses it for generated specs. If it is a remote Hugging Face model id, with or without the `hf_model://` prefix, the script downloads or reuses it under `<deft_workspace>/hf_cache` and writes the local downloaded checkpoint path into the generated inference specs. If the field is `null`, the script uses `nvidia/Cosmos-Embed1-224p`.

The script downloads remote HF checkpoints before Cosmos Embed runs because Cosmos Embed startup can race when `inference.num_gpus > 1` and multiple workers try to download the same model at once. If both combined dataset Parquets are provided, the preparation script does not need the checkpoint and does not download it.

Use `scripts/prepare_cosmos_embed_inference.py` once. It prepares both the KPI dataset and the train dataset:

```bash
python3 "$DEFT_SKILL_ROOT/scripts/prepare_cosmos_embed_inference.py" \
  --workspace "$WORKSPACE" \
  --workflow-yaml "$WORKSPACE/specs/workflow.yaml" \
  --run-dir "$RUN_DIR"
```

The preparation script writes only the missing embedding specs under the initialized `RUN_DIR`. Initialization owns the `$RUN_DIR/workflow.yaml` snapshot; this stage does not derive a new run or overwrite that snapshot.

For each dataset without a supplied combined Parquet, the preparation script writes all required modality-specific inference specs under `cosmos_embed_output/<dataset>/specs/`, creates raw Cosmos Embed result directories, stages text questions as files for stable joins, and writes `lookup.parquet`. After inference, the conversion script gathers those separate modality outputs into one mining-facing file per dataset. Each row contains `filepath`, `embedding`, and `modality`:

```text
$RUN_DIR/embedding_parquets/kpi/embeddings.parquet
$RUN_DIR/embedding_parquets/train/embeddings.parquet
```

The KPI parquet contains only the selected modality or modalities. The train parquet always contains both text and video rows.

### Run Cosmos Embed Inference

Use the `tao-finetune-cosmos-embed` skill's inference action for each generated inference spec. That model skill owns the Cosmos Embed container image, action command, credentials, and platform-specific mount behavior. This workflow skill owns only the generated specs and the downstream conversion into mining-ready parquet files. Do not run inference or conversion for a dataset whose combined Parquet was supplied and staged during preparation. See `references/mining-loop.md` for exact commands, permission repair, and conversion.

The generated lookup Parquet uses `annotation_id` for the original LLaVA `id` and `video_path` for resolved media. Do not rename either column to `video_id`; that external field belongs to Cosmos Reason and gap-analysis records.

## DEFT Mining Loop

Run iterations `1..run.max_iterations`. The loop is mining-only: no PAIDF or generated-video branch runs in this skill. Before gap analysis, each iteration rewrites Cosmos Reason result `video_id` values from LLaVA annotation ids to resolved KPI video paths and stores the prepared predictions in the iteration's `gaps/` directory. It then runs gap analysis, builds one selected-modality target parquet, mines it against the combined text/video train source in one `tao-mine-nearest-neighbors` run, converts mined rows to LLaVA annotations, merges annotations, trains Cosmos Reason, evaluates the trained checkpoint, computes that evaluation's BCQ accuracy metrics, and removes the resumable training checkpoints while preserving the exported safetensors. At loop termination, generate the run-level accuracy report and include its baseline/iteration table in the agent's final response. See `references/mining-loop.md` for exact commands and completion criteria.

## Completion Criteria

| Stage | Complete when |
| --- | --- |
| `validate_workflow` | `verify_workflow_yaml.py` exits successfully. |
| `initialize_workflow` | `$RUN_DIR/workflow.yaml` and `$RUN_DIR/deft_state.json` exist. |
| `baseline_evaluate` | The evaluate job exits successfully, exactly one baseline `results.json` is found, and `baseline/evaluate/bcq_accuracy_metrics.json` exists. |
| `prepare_cosmos_embed_inference` | Each dataset has a lookup parquet and either all required Cosmos Embed specs or a staged combined embedding Parquet. |
| `cosmos_embed` | Every generated Cosmos Embed inference spec has completed successfully through the underlying skill. |
| `convert_embeddings` | `embedding_parquets/{kpi,train}/embeddings.parquet` exist; train contains both modalities and KPI contains the selected modalities. |
| `gap_analysis` | `$RUN_DIR/iter_<N>/gaps/predictions.json` exists and the container exits successfully. The workflow counts valid rows directly from `kpi_gaps.jsonl`; missing or empty output after successful completion means zero weak samples. |
| `prepare_nearest_neighbor_mining` | One target parquet, optional filtered source parquet, and one nearest-neighbor spec exist. |
| `mine_nearest_neighbors` | One mined-neighbor parquet and mining summary exist. |
| `record_mined_paths` | When `mine_unique_only` is true, `$RUN_DIR/mining/mined_paths_log.parquet` exists; otherwise the stage is logged as skipped. |
| `prepare_cosmos_reason_train` | Mined and accumulated LLaVA annotations plus `train/specs/train.toml` exist. |
| `train` | The Cosmos Reason training job reaches terminal success. |
| `evaluate` | Evaluation preparation finds the latest completed training checkpoint, the evaluation job exits successfully, exactly one iteration `results.json` is found, and its `bcq_accuracy_metrics.json` exists. |
| `cleanup_cosmos_reason_training` | `train/checkpoint_cleanup.json` exists, raw checkpoint directories are gone, and its listed safetensors exports remain. |
| `loop_stop` | Stop reason is logged and `bcq_accuracy_report.md` plus `bcq_accuracy_summary.json` compare the baseline with every completed iteration. |

## Troubleshooting

**Prediction id does not match an annotation**: Confirm the evaluation used the same KPI annotations configured by `kpi_dataset.annotations_path`. Every Cosmos Reason result `video_id` must match a LLaVA annotation `id` or an already-resolved annotation `video` path.

**No target embeddings matched**: Check that `gaps/predictions.json` and gap-analysis `video_id` paths match the KPI LLaVA media paths used during embedding preparation. Text mode also requires the question text to match after removing `<video>` and trailing “Answer with yes or no.”

**Mined neighbors do not join to train lookup**: Confirm every mined `filepath` exists in `embedding_parquets/train/embeddings.parquet`. Its `modality` determines whether the path must match train `lookup.parquet` `filepath` for text or `video_path` for video.

**Run directory looks nested under `results/<run.name>`**: This is expected. `run.name` is the run directory name under `<deft_workspace>/results`; all baseline, embedding, and iteration artifacts are nested there.

**Docker-created output is not writable**: Use `restore_docker_mount_permissions.py` on the affected run subdirectory after informing the user which container produced root-owned files.

**Unparseable prediction count is nonzero**: Report the count to the user and inspect the corresponding raw `results.json` responses. These predictions count as incorrect in accuracy and class recall; the metrics script does not silently drop them. An unparseable ground truth stops metric computation because the expected class is undefined.

**Cosmos Reason reports `No space left on device` for `/dev/shm/nccl-*`**: This is container shared memory, not ordinary disk capacity. For local Docker, run the Cosmos Reason action through `tao-run-on-docker` and require `--ipc=host --ulimit memlock=-1 --ulimit stack=67108864`. Confirm these flags in the submitted job before retrying. Other platforms must provide equivalent shared-memory and memlock settings.