tao-train-dino · diff
v0.1.0 to v0.1.0
86 added, 11 removed. Audit A to A.
---
name: tao-train-dino
description: DINO (DETR with Improved DeNoising Anchor Boxes) for 2D object detection. Transformer-based detector with
denoising training, multi-scale features, and optional distillation support. Use when training, evaluating, exporting,
distilling, quantizing, or running inference for a TAO DINO detector. Trigger phrases include "train DINO", "DETR object
detection", "TAO 2D detection", "DINO with distillation".
license: Apache-2.0
compatibility: Requires docker + nvidia-container-toolkit.
metadata:
version: "0.1.0"
author: NVIDIA Corporation
allowed-tools: Read Bash
tags:
- object
- detection
---
# DINO
DINO (DETR with Improved DeNoising Anchor Boxes) for 2D object detection. Transformer-based detector with denoising training, multi-scale features, and optional distillation support.
Uses pretrained backbone weights (e.g. ResNet-50 ImageNet). Set `model.pretrained_backbone_path` for backbone-only or `train.pretrained_model_path` for full model.
+ ## When To Use
+
+ Train, evaluate, export, distill, quantize, or run inference for a TAO DINO 2D object detector.
+
For TAO Deploy TensorRT actions (`gen_trt_engine`, TensorRT `evaluate`, and
TensorRT `inference`), read `references/tao-deploy-dino.md` first. Deploy spec templates live
in this skill's `references/` folder with the `spec_template_deploy_*.yaml`
prefix.
- Generated TAO Core schemas are packaged in `schemas/<action>.schema.json` (with `schemas/manifest.json` listing actions); each schema emits a matching `references/spec_template_<action>.yaml`. See `references/sdk_orchestration.md` for the full dataclass-schema, spec-template, data-sources, and parent-model inference details used by SDK orchestration.
+ ## Reference Map
+ - `references/dino-data-specs.md` — dataset contracts, per-action dataset requirements, per-action spec-override examples (train, evaluate, export, deploy/gen_trt_engine, inference, quantize, distill), data-source arrays, checkpoint inference, and dataset layout.
+ - `references/dino-actions-errors.md` — important parameters, default values, evaluate/export defaults, hardware, and the full error-pattern catalog.
+ - `references/dino-tuning-multigpu.md` — full AutoML/HPO notes (metrics, hyperparameters, extractor) and multi-GPU spec consistency.
+ - `references/dino-automl-sdk.md` — AutoML metrics, SDK orchestration internals, data-source gap, and spec-param/parent-model inference.
+ - `references/tao-deploy-dino.md` — TensorRT deploy workflow.
+ - `references/detailed-guide.md` — map to the detailed model guide.
+
+ ## Dataclass Schemas
+
+ Generated TAO Core schemas are packaged in `schemas/<action>.schema.json`, with `schemas/manifest.json` listing available actions. Each generated schema also emits `references/spec_template_<action>.yaml` from the schema top-level `default` field. AutoML enablement is declared at the model layer in `references/skill_info.yaml` via `automl_enabled`. Runnable AutoML still requires `schemas/train.schema.json` and `references/spec_template_train.yaml` to exist and parse. Use the packaged train schema for `automl_default_parameters`, `automl_disabled_parameters`, defaults, min/max bounds, enums, option weights, math conditions, dependencies, and popular parameters. Do not expect `~/tao-core` at runtime; maintainers regenerate schemas/templates before packaging the skill bank.
+
## Train Action Policy
- This model is AutoML-enabled at the model layer. Before handling any train-stage request, read `references/skill_info.yaml` and resolve the run override from either an explicit `automl_policy` value or the user's workflow request. Treat phrases like "turn off AutoML", "disable AutoML", "no HPO", or "plain training" as `automl_policy: off` for this run only; otherwise default to `auto`. When `automl_policy: auto`, `automl_enabled: true`, and both `schemas/train.schema.json` and `references/spec_template_train.yaml` are packaged, route the train action through `tao-skill-bank:tao-run-automl` by default with this model's `skill_dir`. Preserve workflow/application overrides for datasets, specs, output directories, GPU/platform settings, parent checkpoints, and `automl_policy`. Use direct model training only when `automl_policy: off` or the packaged train schema/template is missing; in the missing-schema case, report that AutoML is enabled but not runnable for this model until schemas are generated.
+ This model is AutoML-enabled at the model layer. Before handling any train-stage request, read `references/skill_info.yaml` and resolve the run override from either an explicit `automl_policy` value or the user's workflow request. Use `automl_policy: on` by default and only expose `on` / `off` in new launch prompts. Treat phrases like "turn off AutoML", "disable AutoML", "no HPO", or "plain training" as `automl_policy: off` for this run only. When `automl_policy: on`, `automl_enabled: true`, and both `schemas/train.schema.json` and `references/spec_template_train.yaml` are packaged, route the train action through `tao-skill-bank:tao-run-automl` by default with this model's `skill_dir`. Preserve workflow/application overrides for datasets, specs, output directories, GPU/platform settings, parent checkpoints, and `automl_policy`. Use direct model training only when `automl_policy: off` or the packaged train schema/template is missing; in the missing-schema case, report that AutoML is enabled but not runnable for this model until schemas are generated.
Non-train actions such as `evaluate`, `inference`, `export`, and deploy flows stay in this model skill. The per-run `automl_policy` override does not change model metadata.
## Training Requirements
The agent MUST read this section before generating any training or AutoML script for DINO.
- **Dataset type:** object_detection
- **Formats:** coco, coco_raw
- **Accepted dataset intents:** training, evaluation, testing, calibration
- - **Monitoring metric:** val_mAP50
+ - **Monitoring metric:** mAP50 for quick operational checks; `val_mAP` for
+ COCO/paper-style benchmark comparisons.
**Required datasets — MUST resolve both:**
| Dataset | Required | Why |
|---|---|---|
| Train dataset URI | Yes | Training data (COCO format) |
| Validation dataset URI | **Yes — ALWAYS** | DINO unconditionally builds a val dataloader. Omitting `val_data_sources` causes `FileNotFoundError` at startup regardless of the metric or workflow. If the user has no separate eval split, reuse the train URI. |
**Required inputs before generating any training spec:**
1. **Train dataset URI** — S3 path to COCO-format training data
2. **Validation dataset URI** — S3 path to COCO-format val data (can be same as train)
3. **`num_classes`** — How many object classes? Default 91 (COCO). Must be >= `max(category_id) + 1`. Too low causes `CUDA error: device-side assert triggered`.
Resolve these from the user request or the default profile below. Prompt only
for values that are still missing after applying the profile rules.
**Bankable local default profile for DINO AutoML smoke runs:**
Use this profile only when the user asks to run DINO AutoML and does not provide
dataset or class-count inputs. This profile is intentionally small and local to
this skill bank; it is for smoke/iteration runs, not a production benchmark.
Do not search previous runners, logs, session state, shell history, or the home
directory to recover these values.
```python
DINO_AUTOML_PROFILE = {
"train_dataset_uri": "s3://nvcf-storage-handling/data/tao_od_synthetic_subset_train_no_convert",
"validation_dataset_uri": "s3://nvcf-storage-handling/data/tao_od_synthetic_subset_val_no_convert",
"object_classes": 4,
"dataset_num_classes": 5,
"image_archive": "images.tar.gz",
"annotation_file": "annotations.json",
"max_recommendations": 10,
"train_num_epochs": 10,
"train_checkpoint_interval": 10,
"train_validation_interval": 1,
"train_num_gpus": 1,
}
```
If the user supplies any dataset URI or class-count value, prefer the user value
and ask for any remaining required DINO value. Do not partially mix a user's
custom dataset with this profile's class count unless the user confirms it.
**Do not prompt for image layout for the standard DINO dataset.** The standard
TAO DINO dataset artifact is `images.tar.gz` plus `annotations.json`. Use
`images.tar.gz` in the remote `image_dir` spec override. The SDK downloads the
archive and rewrites the runtime spec to the extracted folder named after the
archive stem (`images.tar.gz` -> `images`). Only deviate if the user explicitly
provides a different image artifact name.
- ## Spec Overrides, Datasets, and Parameters
+ ## Core Workflow
- Data source overrides are **mandatory for every action** — DINO's `config.json` has empty `data_sources` because the runner cannot auto-resolve array-of-objects spec keys. The agent MUST build data source paths and include them in `spec_overrides`.
+ DINO supports train, evaluate, export, distill, quantize, and inference. Data-source
+ overrides are **mandatory for every action** — DINO's `config.json` has empty
+ `data_sources` because the runner cannot auto-resolve array-of-objects spec keys.
+ The agent MUST construct data source paths and include them in `spec_overrides`.
- See `references/spec_overrides.md` for: the per-action dataset requirements table; the mandatory `spec_overrides` blocks for `train`, `evaluate`, `export`, `gen_trt_engine`, `inference`, `quantize`, and `distill`; checkpoint resolution via `parent_model` inference and the `results_dir/train/dino_model_latest.pth` fallback; the COCO dataset format and `images.tar.gz` archive-stem rules; per-action data-source layouts; the full **Important Parameters** list (num_classes, backbone and its supported values, lr/lr_backbone, num_epochs, lr_steps, num_queries, batch_size); **Default Values** (num_epochs 10, batch_size 4, learning_rate 2e-4, lr_backbone 2e-5, num_classes 91, backbone resnet_50); **Evaluate Defaults**; **Export Defaults** (input 640x640, opset 17, trt_data_types [FP32, FP16, INT8], trt_workspace_size_mb 1024); and **Hardware** guidance (1 GPU minimum, 4 recommended, 24GB+ A100). Full TAO Deploy reference: [tao-deploy-dino](references/tao-deploy-dino.md).
+ See `references/dino-data-specs.md` for the per-action dataset requirements table,
+ the standard dataset artifact (`images.tar.gz` + `annotations.json`) and runtime
+ folder rewrite rules, and the complete per-action `spec_overrides` examples for
+ train, evaluate, export, deploy/gen_trt_engine, inference, quantize, and distill —
+ including checkpoint inference via `parent_model`, the `results_dir/train/`
+ checkpoint location, and the distillation FAN-teacher / student rules.
- When generating an `evaluate` spec, carry forward the winning AutoML rec's structural model settings (`model.backbone`, `model.num_queries`, `model.dropout_ratio`, `dataset.num_classes`) so the checkpoint shape matches the evaluation model.
+ ## Important Parameters And Defaults
+ Key defaults: `num_epochs=10`, `batch_size=4`, `learning_rate=2e-4`,
+ `lr_backbone=2e-5`, `num_classes=91`, `backbone=resnet_50`.
+
+ - **dataset.num_classes**: Default 91 (COCO). Must be >= `max(category_id) + 1`. Too low causes `CUDA error: device-side assert triggered`. Set as `<num_classes> + 1` in spec overrides.
+ - **num_epochs**: default 10 (quick iteration); real datasets typically need 30-50+ epochs for good mAP.
+
+ See `references/dino-actions-errors.md` for the full parameter list (backbone
+ options, `train.optim.lr`/`lr_steps`, `model.num_queries`, `batch_size`),
+ default values, evaluate defaults, export defaults (input 960x544, opset 17,
+ TRT data types, workspace 1024 MB), and hardware requirements.
+
+ ## Multi-GPU And AutoML / HPO
+
+ When increasing `train.num_gpus`, also set `train.gpu_ids` to the same visible
+ device range, or distributed startup can be inconsistent.
+
+ AutoML runs training — all **Training Requirements** above apply. For no-input
+ local smoke runs, use `DINO_AUTOML_PROFILE`. Recommended metric is `mAP50`
+ (`val_mAP` for benchmark comparisons) with `direction="maximize"` and a custom
+ `metric_extractor`.
+
+ See `references/dino-tuning-multigpu.md` for the full multi-GPU spec-consistency
+ rule (8-GPU example, NCCL timeout note) and the full AutoML/HPO notes (metric
+ selection, `metric_extractor`, recommended hyperparameters, `weight_decay`
+ behavior, dense-dataset resume guidance). See `references/dino-automl-sdk.md` for
+ AutoML metric extractor code, SDK orchestration internals, and parent-model
+ inference mappings.
+
## Error Patterns
- Common failures include CUDA OOM, `num_select < num_queries * num_classes`, spec/schema merge errors, dataset-smaller-than-batch, `return_interm_indices` vs `num_feature_levels` mismatch, `FileNotFoundError` on images or missing val data, `CUDA device-side assert` from low `num_classes`, S3 inputs not downloaded, and evaluate checkpoint not found at the result root. See `references/troubleshooting.md` for each error pattern and its fix.
+ Common failures include CUDA OOM (reduce `batch_size`), missing `val_data_sources`
+ (`FileNotFoundError` at startup — always supply val), `num_classes` too low (`CUDA
+ device-side assert`), and the parent `dino gen_trt_engine` / `dino convert` PyT-CLI
+ restrictions.
- ## AutoML / HPO Notes
+ See `references/dino-actions-errors.md` for the complete error-pattern catalog
+ with diagnostics and fixes.
- AutoML runs training — all **Training Requirements** above apply, and the no-input case uses `DINO_AUTOML_PROFILE`. Do not inspect previous AutoML runs to infer dataset URIs, `num_classes`, recommendation count, or interval settings. Use explicit `metric="mAP50"` with `direction="maximize"` and a custom `metric_extractor` reading `Validation mAP50` rather than `metric="kpi"`. See `references/automl.md` for the recommended metric extractor, hyperparameter list, `custom_param_ranges`, the `train.optim.weight_decay` note, and the backbone-constraint guidance.
+ ## Spec Param / Parent Model Inference
+ Model-specific inference mappings belong in this MD file, not in `config.json`.
+ Generated runners read the mappings and apply them with SDK helpers before
+ `create_job()`. For `parent_model`/`parent_model_folder`, pass the upstream
+ train/export/AutoML child job id as `parent_job_id`; the SDK lists the parent
+ result folder, filters checkpoint artifacts, and returns the selected model.
+
+ See `references/dino-automl-sdk.md` for the full inference-mapping table (per
+ action: `parent_model`, `key`, `output_dir`, `ptm_if_no_resume_model`,
+ `resume_model`, `create_onnx_file`) and the TensorRT-mapping note. TensorRT
+ mappings live in the deploy workflow, not the PyT model skill.
+
## Optional: running via the TAO SDK
- The SDK `script_runner` orchestration, S3 I/O wrapping, AutoML internals, spec-template generation, the data-sources gap, the `config.json` `inputs` declarations, and the full per-action spec-param / parent-model inference mapping table are documented in `references/sdk_orchestration.md`. Skip this when running locally with `docker run`.
+ When running DINO through the TAO SDK (`script_runner` orchestration, S3 I/O
+ wrapping, AutoML), skills read `references/skill_info.yaml` for input and
+ spec-param mappings. See `references/dino-automl-sdk.md` for SDK orchestration
+ internals, including the data-sources gap and the `[0]`-indexed `inputs`
+ declarations. Skip this when running locally with `docker run`.
+
+ ## Deployment
+
+ - [tao-deploy-dino](references/tao-deploy-dino.md)