tao-train-sparse4d · diff
v0.1.0 to v0.2.0
64 added, 204 removed. Audit A to A.
---
name: tao-train-sparse4d
- description: Sparse4D for multi-camera temporal 3D object detection and tracking. Uses sparse queries with deformable
- attention across camera views and time for end-to-end 3D perception, with an instance bank for temporal tracking. Use when
- training, evaluating, exporting, quantizing, or running inference for a TAO Sparse4D model. Trigger phrases include
- "train Sparse4D", "multi-camera 3D detection", "temporal 3D tracker", "sparse query 3D perception".
+ description: Prepare calibrated multi-camera data and train, evaluate, run inference, export, or quantize TAO Sparse4D 3D detection and tracking models. Includes 2D-to-3D geometric distillation with released LTT weights. Use for Sparse4D adaptation and temporal multi-camera perception workflows.
license: Apache-2.0
- compatibility: Requires docker + nvidia-container-toolkit.
+ compatibility: Requires a compatible TAO runtime; Docker execution requires docker and nvidia-container-toolkit.
metadata:
- version: "0.1.0"
+ version: "0.2.0"
author: NVIDIA Corporation
allowed-tools: Read Bash
tags:
- temporal
- 3d
- detection
- tracking
---
# Sparse4D
- > **Standalone install?** If this session was not initialized by the TAO skill bank plugin, run the `tao-setup` skill first (host preflight, credentials, cross-skill discovery).
+ > **Standalone install?** If the TAO skill bank plugin did not initialize the session, run `tao-setup` for host preflight, credentials, and cross-skill discovery.
- Sparse4D for multi-camera temporal 3D object detection and tracking. Uses sparse queries with deformable attention across camera views and time for end-to-end 3D perception. Includes instance bank for temporal tracking.
+ Sparse4D predicts 3D boxes and tracking IDs from synchronized, calibrated multi-camera images. Start with the selected model's experiment specification and checkpoint. The [canonical TAO documentation](https://docs.nvidia.com/tao/tao-toolkit/latest/text/cv_finetuning/pytorch/sparse4d/sparse4d.html) covers the model workflow.
- Use a pretrained ResNet-101 backbone when one is available by setting
- `train.pretrained_model_path`. For local smoke validation, Sparse4D training
- can run with an empty `train.pretrained_model_path`, but production runs should
- still use a compatible PTM.
+ ## Instructions
- ## Dataclass Schemas
+ 1. Identify the requested action, data supervision, selected checkpoint, and platform. Read [the action contract](references/skill_info.yaml) and the selected platform skill. Preserve the user's platform, model, data, and training-budget choices.
+ 2. Match the backbone, ordered classes, image preprocessing, anchors, and temporal dimensions to the selected checkpoint. Follow **Backbone and checkpoint selection** below before preparing the specification.
+ 3. Use existing compatible info PKLs when supplied. For raw data, follow [local conversion and artifact checks](references/local_docker_conversion.md). For calibrated real scenes with 2D targets, read [geometric distillation](references/geometric_distillation.md); that route does not require 3D annotations or conversion-generated anchors.
+ 4. Start from `references/spec_template_<action>.yaml` and the selected release's experiment specification. Resolve paths against actual staged artifacts. Bind `results_dir` and any action-specific `results_dir` to the current job output directory; released specifications can contain old output paths. Write nested YAML, not flat dotted keys; do not fabricate train/val/test files or scene suffixes. Check the installed runtime exposes the fields used by the requested workflow.
+ 5. Resolve the model-level image for model actions and the **action-level Data Services image** for `dataset_convert`. For a requested TAO release, use the matching skill-bank release branch and its image pins; do not substitute the default branch's release. Follow `tao-launch-workflow` and the chosen platform's `submit` / `status` / `logs` / `cancel` contract, retaining its job record and explicit output directory. Existing user authorization applies; request missing launch approval only for work not already authorized.
+ 6. Verify the produced artifacts and model metrics. A successful process or short training run is a smoke result, not evidence of improved 3D accuracy. Do not report unpublished experimental fixes or another run's results as validation of the selected image.
- Generated TAO Core schemas are packaged in `schemas/<action>.schema.json`, with `schemas/manifest.json` listing available actions. Each generated schema also emits `references/spec_template_<action>.yaml` from the schema top-level `default` field. AutoML enablement is declared at the model layer in `references/skill_info.yaml` via `automl_enabled`. Runnable AutoML for an action requires `schemas/<action>.schema.json` and `references/spec_template_<action>.yaml` to exist and parse. Use the packaged selected-action schema for `automl_default_parameters`, `automl_disabled_parameters`, defaults, min/max bounds, enums, option weights, math conditions, dependencies, and popular parameters. Do not expect `~/tao-core` at runtime; maintainers regenerate schemas/templates before packaging the skill bank.
+ ## Backbone and checkpoint selection
- ## Train Action Policy
+ - Respect an explicit ResNet-50 or ResNet-101 choice. For an existing Sparse4D checkpoint, use the accompanying experiment specification and release metadata to determine `model.backbone.type` (`resnet_50` or `resnet_101`). If they conflict with the requested backbone, resolve that mismatch before launch; do not change the architecture to make a checkpoint load partially.
+ - If no checkpoint/backbone has been selected, ask which compatible released model to use. The schema default `resnet_101` is a configuration default, not a recommendation or evidence about the user's checkpoint.
+ - Preserve the selected checkpoint's class order, anchors, input dimensions, and temporal/query dimensions. A filename alone is insufficient when its provenance is unclear.
+ - The LTT MLP is a separate geometry adapter. Downloading it from the ResNet-50 bundle does **not** select the Sparse4D backbone. Its ordered taxonomy must match the training data and model.
+ - Fine-tuning uses `train.pretrained_model_path`. Continuing a run uses `train.resume_training_checkpoint_path` and a compatible saved training state. Random initialization with an empty pretrained path is suitable only when explicitly requested, for example a wiring smoke test.
- This model is AutoML-enabled at the model layer. Before handling any train-stage request, read `references/skill_info.yaml` and resolve the run override from either an explicit `automl_policy` value or the user's workflow request. Use `automl_policy: on` by default and only expose `on` / `off` in new launch prompts. Treat phrases like "turn off AutoML", "disable AutoML", "no HPO", or "plain training" as `automl_policy: off` for this run only. When `automl_policy: on`, `automl_enabled: true`, and both `schemas/train.schema.json` and `references/spec_template_train.yaml` are packaged, route the train action through `tao-skill-bank:tao-run-automl` by default with this model's `skill_dir`. Preserve workflow/application overrides for datasets, specs, output directories, GPU/platform settings, parent checkpoints, and `automl_policy`. Use direct model training only when `automl_policy: off` or the packaged train schema/template is missing; in the missing-schema case, report that AutoML is enabled but not runnable for this model until schemas are generated.
+ ## Train action policy
- Non-train actions such as `evaluate`, `inference`, `export`, and deploy flows stay in this model skill. The per-run `automl_policy` override does not change model metadata.
+ Read `references/skill_info.yaml` before train requests. This model is AutoML-enabled. Preserve an explicit `automl_policy: on` / `off`; requests for plain training, no HPO, a fixed-specification reproduction, a bounded wiring smoke test, or an exact continuation mean `off` for that run. Otherwise default to `on` and route through `tao-skill-bank:tao-run-automl` when both `schemas/train.schema.json` and `references/spec_template_train.yaml` are available. Preserve dataset, checkpoint, specification, GPU/platform, and budget overrides. If the schema/template is missing, report that AutoML is unavailable and use direct training within the user's request.
+ AutoML's monitoring name is `val_mAP`; Sparse4D emits `img_bbox_NuScenes/mAP` and `mAP` as aliases. A promoted resume job may carry the source rung's metric without a new evaluation: report its provenance and verify the resumed checkpoint independently. Follow the AutoML skill's baseline-evaluation requirement. Evaluate, inference, export, and quantize remain in this skill.
+
## Training Requirements
- - **Dataset type:** sparse4d
- - **Formats:** ovpkl
- - **Monitoring metric:** val_mAP
- - Current TAO Sparse4D training emits this value in status/logs as
- `img_bbox_NuScenes/mAP` and `mAP`; AutoML metric
- extractors should treat those emitted keys as aliases for `val_mAP`.
- Multi-fidelity AutoML algorithms such as Hyperband, ASHA, and BOHB may
- promote a checkpoint to a resume job that completes without emitting a fresh
- `val_mAP` alias. In that case, compare AutoML's carried metric to the source
- rung job that emitted `img_bbox_NuScenes/mAP` or `mAP`, while still verifying
- that the promoted job resumed from the explicit epoch/step checkpoint,
- produced a real checkpoint, and is usable for evaluate/inference.
+ Training needs converted OVPKL data, held-out labeled validation data, matching anchors, and the selected pretrained checkpoint or resume state. Preserve the selected backbone and ordered taxonomy. Geometric distillation additionally needs the released LTT checkpoint and prepared supervision caches. Choose the GPU count and training budget from the user's request and available resources.
### Per-Action Dataset Requirements
- | Action | Spec Key | Source | Files | List? |
- |---|---|---|---|---|
- | dataset_convert | aicity.root | id | | No |
- | evaluate | dataset.data_root | eval_dataset | (from convert job, spec: aicity.split) | No |
- | evaluate | model.head.instance_bank.anchor | train_datasets | /results/{dataset_convert_job_id}/anchor_init.npy | No |
- | evaluate | dataset.train_dataset.ann_file | train_datasets | (from convert job, spec: aicity.split) | No |
- | evaluate | dataset.val_dataset.ann_file | eval_dataset | (from convert job, spec: aicity.split) | No |
- | evaluate | dataset.test_dataset.ann_file | inference_dataset | (from convert job, spec: aicity.split) | No |
- | export | model.head.instance_bank.anchor | train_datasets | /results/{dataset_convert_job_id}/anchor_init.npy | No |
- | inference | dataset.data_root | inference_dataset | (from convert job, spec: aicity.split) | No |
- | inference | model.head.instance_bank.anchor | train_datasets | /results/{dataset_convert_job_id}/anchor_init.npy | No |
- | inference | dataset.train_dataset.ann_file | train_datasets | (from convert job, spec: aicity.split) | No |
- | inference | dataset.val_dataset.ann_file | eval_dataset | (from convert job, spec: aicity.split) | No |
- | inference | dataset.test_dataset.ann_file | inference_dataset | (from convert job, spec: aicity.split) | No |
- | quantize | dataset.data_root | train_datasets | (from convert job, spec: aicity.split) | No |
- | quantize | model.head.instance_bank.anchor | train_datasets | /results/{dataset_convert_job_id}/anchor_init.npy | No |
- | quantize | dataset.train_dataset.ann_file | train_datasets | (from convert job, spec: aicity.split) | No |
- | quantize | dataset.val_dataset.ann_file | eval_dataset | (from convert job, spec: aicity.split) | No |
- | quantize | dataset.test_dataset.ann_file | inference_dataset | (from convert job, spec: aicity.split) | No |
- | quantize | dataset.quant_calibration_dataset.images_dir | train_datasets | | No |
- | train | dataset.data_root | train_datasets | (from convert job, spec: aicity.split) | No |
- | train | model.head.instance_bank.anchor | train_datasets | /results/{dataset_convert_job_id}/anchor_init.npy | No |
- | train | dataset.train_dataset.ann_file | train_datasets | (from convert job, spec: aicity.split) | No |
- | train | dataset.val_dataset.ann_file | eval_dataset | (from convert job, spec: aicity.split) | No |
- | train | dataset.test_dataset.ann_file | inference_dataset | (from convert job, spec: aicity.split) | No |
-
- ### Typical Spec Overrides
-
- Data source overrides are **mandatory for every action** — the agent MUST construct data source paths from the Per-Action Dataset Requirements table above and include them in `spec_overrides`.
-
- ```python
- S3_TRAIN = "s3://bucket/data/train"
- S3_EVAL = "s3://bucket/data/eval"
- CONVERTED_SCENE = "<scene-from-converter>" # e.g. "subsetscene+bev-sensor-random-0"
- ```
-
- **train (mandatory data sources):**
- ```python
- CONVERTED = "s3://bucket/results/<dataset_convert_job_id>"
- {
- "train.num_epochs": 30,
- "train.checkpoint_interval": 10,
- "train.validation_interval": 10,
- "train.num_gpus": 1,
- "dataset.sequences.split_num": 90,
- "dataset.train_dataset.sequences_split_num": 90,
- "dataset.data_root": f"{S3_TRAIN}/train",
- "model.head.instance_bank.anchor": f"{CONVERTED}/anchor_init.npy",
- "dataset.train_dataset.ann_file": f"{CONVERTED}/train/{CONVERTED_SCENE}_infos_train.pkl",
- "dataset.val_dataset.ann_file": f"{CONVERTED}/val/{CONVERTED_SCENE}_infos_val.pkl",
- "dataset.test_dataset.ann_file": f"{CONVERTED}/test/{CONVERTED_SCENE}_infos_test.pkl",
- }
- ```
-
- **evaluate (mandatory data sources):**
- ```python
- CONVERTED = "s3://bucket/results/<dataset_convert_job_id>"
- {
- "dataset.data_root": f"{S3_EVAL}/val",
- "model.head.instance_bank.anchor": f"{CONVERTED}/anchor_init.npy",
- "dataset.train_dataset.ann_file": f"{CONVERTED}/train/{CONVERTED_SCENE}_infos_train.pkl",
- "dataset.val_dataset.ann_file": f"{CONVERTED}/val/{CONVERTED_SCENE}_infos_val.pkl",
- "dataset.test_dataset.ann_file": f"{CONVERTED}/test/{CONVERTED_SCENE}_infos_test.pkl",
- }
- ```
-
- **export (mandatory data sources):**
- ```python
- CONVERTED = "s3://bucket/results/<dataset_convert_job_id>"
- {
- "model.head.instance_bank.anchor": f"{CONVERTED}/anchor_init.npy",
- }
- ```
-
- **inference (mandatory data sources):**
- ```python
- CONVERTED = "s3://bucket/results/<dataset_convert_job_id>"
- {
- "dataset.data_root": f"{S3_EVAL}/test",
- "model.head.instance_bank.anchor": f"{CONVERTED}/anchor_init.npy",
- "dataset.train_dataset.ann_file": f"{CONVERTED}/train/{CONVERTED_SCENE}_infos_train.pkl",
- "dataset.val_dataset.ann_file": f"{CONVERTED}/val/{CONVERTED_SCENE}_infos_val.pkl",
- "dataset.test_dataset.ann_file": f"{CONVERTED}/test/{CONVERTED_SCENE}_infos_test.pkl",
- }
- ```
-
- **quantize (mandatory data sources):**
- ```python
- CONVERTED = "s3://bucket/results/<dataset_convert_job_id>"
- {
- "dataset.data_root": f"{S3_TRAIN}/train",
- "model.head.instance_bank.anchor": f"{CONVERTED}/anchor_init.npy",
- "dataset.train_dataset.ann_file": f"{CONVERTED}/train/{CONVERTED_SCENE}_infos_train.pkl",
- "dataset.val_dataset.ann_file": f"{CONVERTED}/val/{CONVERTED_SCENE}_infos_val.pkl",
- "dataset.test_dataset.ann_file": f"{CONVERTED}/test/{CONVERTED_SCENE}_infos_test.pkl",
- "dataset.quant_calibration_dataset.images_dir": f"{S3_TRAIN}",
- }
- ```
-
- See `references/local_docker_conversion.md` for local-docker conversion roots and mounts, H5 depth-path normalization, converted annotation filenames, smoke-run `max_num_cams`/anchor contracts for export compatibility, and converted-artifact verification before train/evaluate/inference.
-
- ## Eval Dataset
-
- Optional. Val/test splits configured via dataset ann_file paths.
-
- ## Important Parameters
-
- - **model.backbone**: Backbone. Default resnet_101.
- - **model.neck.out_channels**: FPN output channels. Default 256. num_outs=4.
- - **model.input_shape**: Input image shape [W, H]. Default [1408, 512].
- - **model.head.num_output**: Number of detection output queries. Default 300.
- - **model.head.num_decoder**: Number of decoder layers. Default 6.
- - **model.head.temporal**: Enable temporal reasoning. Default True.
- - **model.head.instance_bank.num_anchor**: Instance bank anchors. Default 900.
- - **model.head.instance_bank.num_temp_instances**: Temporal instance count. Default 600.
- - **model.depth_branch.loss_weight**: Depth supervision loss weight. Default 0.2.
- - **dataset.batch_size**: Per-GPU batch size. Default 2.
- - **dataset.num_frames**: Sequence length. Default 200.
- - **dataset.classes**: Detection classes. Default [person, gr1_t2, agility_digit, nova_carter]. num_ids=70 for tracking.
- - **train.optim.lr**: Learning rate. Default 5e-5. img_backbone lr_mult=0.2.
- - **train.lr_scheduler**: Cosine scheduler with linear warmup (500 iters, ratio 0.333).
- - **train.grad_clip.max_norm**: Gradient clipping. Default 25.
- - **train.precision**: Options: bf16, fp16, fp32. Default bf16.
- - **evaluate.metrics**: Eval metrics. Default ["detection"]. Optional tracking evaluation.
- - **evaluate.tracking.enabled**: Enable tracking evaluation. tracking_threshold=0.2.
-
- ## Multi-GPU / Multi-Node
-
- **Launch method:** Lightning-managed (single `python` process, Lightning spawns workers).
-
- | Spec Key | Description | Default |
- |----------|-------------|---------|
- | `train.num_gpus` | Number of GPUs | 1 |
- | `train.gpu_ids` | GPU device indices | [0] |
- | `train.num_nodes` | Number of nodes | 1 |
+ The model uses OVPKL annotations. The action contract declares staged inputs, commands, outputs, and optional artifacts. Paths embedded in info PKLs, split lists, and lazy indexes must also remain visible inside the selected runtime; staging a split file alone does not stage its referenced data.
- - Multi-GPU strategy: `ddp_find_unused_parameters_true` (no fsdp support)
- - `sync_batchnorm` is always enabled (True)
- - Iterations per epoch computed as: `num_frames * num_bev_groups / (num_nodes * num_gpus * batch_size)`
- - **Scaling:** When increasing GPUs, effective batch size grows and iterations-per-epoch shrinks proportionally
+ | Action | Runtime command | Required data / model artifacts |
+ |---|---|---|
+ | `dataset_convert` | `annotations convert -e SPEC` (Data Services) | Raw scene root, calibration, images; 3D annotations for labeled conversion only |
+ | `train` | `sparse4d train -e SPEC` | Training and held-out validation PKLs/splits, image root, matching anchors, selected pretrained or resume checkpoint; optional LTT/cache artifacts for the chosen supervision route |
+ | `evaluate` | `sparse4d evaluate -e SPEC` | `evaluate.checkpoint`, matching anchors, labeled held-out PKLs at **`dataset.test_dataset.ann_file`**, image root |
+ | `inference` | `sparse4d inference -e SPEC` | `inference.checkpoint`, matching anchors, input PKLs at `dataset.test_dataset.ann_file`, image root |
+ | `export` | `sparse4d export -e SPEC` | `export.checkpoint`, matching anchors and architecture, explicit `export.onnx_file`; no training datasets or external LTT checkpoint |
+ | `quantize` | `sparse4d quantize -e SPEC` | `quantize.model_path` and the selected backend's calibration inputs; check runtime backend support |
- **Multi-node env vars** (set by orchestrator): `WORLD_SIZE`, `NODE_RANK`, `MASTER_ADDR`, `MASTER_PORT`, `NUM_GPU_PER_NODE`.
+ The evaluation CLI calls the test dataloader. Setting only `dataset.val_dataset.ann_file` does not select its evaluation data; that field is used during training validation. Keep the unused dataset sections from the complete specification without inventing or staging unused annotation files.
- ## Hardware
+ Resolve `model.head.instance_bank.anchor` as an explicit model artifact from the selected bundle or a compatible labeled conversion. It is not inferred from a mandatory training dataset for evaluation or export.
- Minimum 2 GPU(s), recommended 8 GPU(s). 40GB+ (A100 recommended) VRAM per GPU. Multi-camera temporal model is memory intensive. bf16 required for practical training. Multi-GPU strongly recommended. Instance bank requires substantial memory for temporal reasoning.
+ Use [checkpoint and output resolution](references/spec_param_inference.md) for exact artifact handoff. Do not choose the first checkpoint found or rely on a removed SDK job resolver.
- ## Error Patterns
+ ### Typical Spec Overrides
- **dataset_convert required**: Must run dataset_convert first to produce annotation pickles and anchor_init.npy.
+ For AutoML, construct `spec_overrides` from concrete staged paths in the table above and `skill_info.yaml`'s `data_sources`; dotted keys are an AutoML override map, not the container YAML format. Set only the inputs needed by the selected action.
- **dataset_convert container/command**: Sparse4D conversion is an AICity to
- OVPKL annotations conversion. Launch `dataset_convert` with the action-level
- `tao_toolkit.data_services` image and `annotations convert -e {config_path}`;
- do not use the PyTorch `sparse4d` CLI for conversion. Train/evaluate/export/
- inference still use the model-level PyTorch image.
+ | Scope | Overrides to resolve |
+ |---|---|
+ | Training data | `dataset.data_root`, `dataset.train_dataset.ann_file`, `dataset.val_dataset.ann_file` |
+ | Model initialization | `model.backbone.type`, `model.head.instance_bank.anchor`, `dataset.classes`, and `train.pretrained_model_path` or `train.resume_training_checkpoint_path` |
+ | Training resources | `train.num_gpus`, `train.num_epochs`, `dataset.batch_size`, and `dataset.num_workers`, preserving the requested budget |
+ | Geometric distillation | The LTT/cache paths and route fields in [the training fragment](references/geometric_distillation_train.yaml); preserve their validated class order and supervision policy during HPO |
+ | Evaluation and inference | `dataset.data_root`, `dataset.test_dataset.ann_file`, matching anchors, and the action's checkpoint |
+ | Output | `results_dir` and the action-specific output paths bound to the current job |
- **Stable raw-data path**: The AICity to OVPKL converter writes image paths into
- the generated pickle files. Keep `aicity.root` at `/data/aicity_root` during
- conversion, then point `dataset.data_root` at the split folder, for example
- `/data/aicity_root/train` for training or `/data/aicity_root/val` for
- evaluation. This preserves the converter's absolute RGB paths and relative
- depth paths.
+ With `dataset.lazy_load: true`, stage the split's parent directory as a folder, including `<split>_lazy_index.pkl` beside the split and `_pkl_cam_counts.pkl` when used. Point `ann_file` into that staged directory; do not let file-only staging relocate it away from its index. Keep referenced PKLs and images visible, and rebuild the index after relocation. `dataset.pkl_cam_counts_path` is an optional camera-count override, not the lazy index.
- **H5 depth tuple mismatch**: If training fails with an H5 path error where the
- trainer tries to open a camera directory such as
- `/data/aicity_root/train/<scene>/Camera`, run
- `models/tao-train-sparse4d/scripts/normalize_depth_paths.py --data-root <host-aicity-root>/train <converted-ann-dir>`
- after `dataset_convert` and before train/evaluate/inference. The helper rewrites
- converted `depth_map_path` tuples to point at
- `<scene>/depth_maps/<camera>.h5` with the H5 dataset key basename.
+ ## Configuration and resource checks
- **Missing anchor file**: Set model.head.instance_bank.anchor to the anchor_init.npy path from dataset_convert results.
+ Generated schemas and templates describe the packaged contract; `schemas/manifest.json` lists actions. Do not require a TAO Core checkout on the user's machine. Use the runtime's matching release specification for launch: a schema that knows a new field does not make an older image support it.
- **Temporal OOM**: Reduce dataset.num_frames or dataset.batch_size if running out of memory during temporal training.
+ - `model.backbone.type`: match the selected checkpoint.
+ - `train.optim.lr`, `train.optim.lr_scheduler`, and `train.optim.grad_clip`: optimizer settings are nested under `optim`.
+ - `dataset.batch_size`: per-GPU batch size. `dataset.max_cameras` can reduce sampled training cameras when supported; validate the resulting supervision coverage.
+ - `train.precision`: `bf16`, `fp16`, or `fp32`, subject to hardware/runtime support. BF16 is not mandatory.
+ - `train.num_gpus`, `train.gpu_ids`, `train.num_nodes`: use the selected platform's allocation. One-GPU smoke training is supported; memory needs depend on the backbone, resolution, cameras, batch size, and temporal configuration.
+ - `dataset.num_frames` and `dataset.num_bev_groups` are scheduling/data-size settings, not an activation-memory sequence-length knob. Inspect actual dataloader length and iteration counts before comparing training budgets across datasets or GPU counts.
+ - `dataset.eval_dist_fcn` and `dataset.eval_hota`: select supported detection/tracking evaluation. Keep thresholds, class order, dataset, and raw-versus-EMA checkpoint choice identical for comparisons.
- **Quantize image compatibility**: The model-skill wiring should pass
- `quantize.model_path` through the parent-model resolver, and checkpoint handoff
- should select the exact epoch/step checkpoint just like evaluate, inference,
- export, and resume. TorchAO checkpoint quantization passes in the
- `validation-fixes-20260525` PyT image and writes
- `quantized_model_torchao.pth`. Older 7.0.0-rc PyT images may fail inside the
- Sparse4D quantize entrypoint or lack ONNX quantization dependencies; do not
- remove or skip the advertised `quantize` action if that occurs. Report the
- container/image failure and keep the exact checkpoint path visible.
+ Lightning manages the local workers. Do not add a second distributed launcher. Use the platform's multi-node environment contract; TAO's `WORLD_SIZE` launcher variable denotes node count, not the global GPU count. Mixed 3D/2D distributed training additionally needs the synchronized route settings in the geometric-distillation reference.
- ## Spec Param / Parent Model Inference
+ ## Error patterns
- See `references/spec_param_inference.md` for the model-specific inference mappings from TAO Core `sparse4d.config.json` (the per-action spec-field to inference-function table) and the `parent_model`/`parent_job_id` checkpoint-resolution rules that generated runners apply with SDK helpers before `create_job()`.
+ - **Missing annotation or image:** inspect the converter outputs and embedded paths. Preserve the emitted BEV-group suffix, camera IDs, frame IDs, and mount roots. A conversion of `train` does not also create `val` and `test` outputs.
+ - **Missing anchor:** reuse the selected pretrained bundle's matching anchors, or use labeled conversion's generated anchors when compatible. Annotation-free conversion intentionally skips anchor initialization.
+ - **Legacy HDF5 depth tuple:** inspect the affected labeled PKL and use the optional normalization helper in the conversion reference only when its documented layout matches. Unlabeled scenes have no depth paths to normalize.
+ - **Temporal OOM:** first reduce per-GPU batch size or supported camera sampling; preserve checkpoint architecture. Do not blindly shrink anchor/temporal dimensions or treat `num_frames` as a memory fix.
+ - **LTT or teacher cache mismatch:** verify ordered classes, scene/camera/frame joins, and container paths. The released LTT route supports its original seven classes; changing class names does not adapt the weights.
+ - **Export reshape:** check the exporter contract in the conversion reference before training a reduced smoke model. Current export uses 20 dummy cameras and 600 cached temporal instances; successful inference alone does not establish export compatibility.
+ - **Unsupported quantize backend/image:** report the selected image, backend, exact checkpoint, and failure. Do not claim a result from a historical validation image proves the current runtime works, or remove the action to hide the failure.