v0.1.0 to v0.1.0

---

name: tao-run-deft-od-aoi

description: >-

Run a binary industrial-inspection DEFT loop with TAO RT-DETR: measure on

frozen KPI/test roles, mine gap-similar real and clean images with SigLIP,

accumulate admitted COCO data, retrain from one base checkpoint, and select

by KPI AP50. Use for iterative AOI defect detection, not generic multiclass OD.

license: Apache-2.0

compatibility: Requires TAO RT-DETR and Data Services images, CUDA GPUs, and normalized COCO roles.

metadata:

author: NVIDIA Corporation

version: "0.1.0"

allowed-tools: Read Bash Write

tags: [application, workflow, deft, object-detection, aoi, rtdetr]

---

# TAO DEFT OD AOI

This application is a disk-backed RT-DETR loop for one foreground class,

`defect`. Its core is real-data-only; AnomalyGenNext synthesis is an optional

route with separate preparation, generation, and admission gates.

## References

Read only the references needed for the current stage:

- intake and launch: `references/defaults.md`, `references/data-contract.md`,

`references/source-manifest.md`, and `references/preflight.md`;

- orchestration: `references/pipeline.md` and

`references/scripts-and-agents.md`;

- gaps and retrieval: `references/gap-routing.md` and

`references/tao-analyze-gaps-od-map.md`;

- training and selection: `references/training-policy.md`;

- optional synthesis: `references/anomalygen-pool.md`.

## Start

Select an installed platform, read its skill, then invoke

`tao-launch-workflow`. The single launch review must include the four normalized

COCO roles, trainable RT-DETR base checkpoint, maximum iterations, image and

Data Services containers, GPU shape, and expected runtime. After approval,

prepare source COCO files when needed:

```bash

scripts/prepare_deft_od_aoi_sources.py \

--manifest /data/dataset_sources.json --check-only

scripts/prepare_deft_od_aoi_sources.py \

--manifest /data/dataset_sources.json \

--output-dir /new/results/normalized

```

`--check-only` runs without a container. Materialization must run in the pinned

TAO Data Services image because it delegates each role's canonical COCO merge

to the existing `annotations merge` action. The application then validates the

merged role contracts before emitting `sources.json`.

The user-facing manifest calls the held-out input `benchmark`. The second

command maps it to the existing internal `kpi` role and emits `sources.json`

with the canonical `kpi`, `test`, `real`, and `clean` mapping. Copy

`assets/default_policy.yaml`, use that mapping for `sources`, fill the other

required values, and initialize once:

```bash

scripts/init_deft_od_aoi.py \

--config /workspace/deft_policy.yaml \

--output-dir /new/results/deft_contract

```

Never reinitialize an existing result. The validator requires disjoint KPI,

test, defective-real, and verified-clean roles; every COCO must declare only

`defect`. KPI and test may mix boxed and boxless images because they never enter

## Loop boundary

The loop composes existing bank actions:

1. Baseline gap initialization in explicit `cold_start` or `checkpoint` mode;

later measurements use `tao-train-rtdetr` inference on KPI and test.

2. Two `tao-analyze-gaps-od-map` actions: loose confidence for FP routing and

strict confidence for FN routing.

3. `tao-generate-image-embeddings` with one frozen SigLIP encoder, followed by

`tao-mine-od-images` unique-neighbor matching against the real or clean role.

4. Application-owned admission and cumulative binary COCO assembly.

5. Direct `tao-train-rtdetr` training from the same frozen base checkpoint,

then KPI-only checkpoint selection. Test remains report-only.

All specs are nested YAML dictionaries. Every GPU/Data Services action uses the

selected platform's `submit/status/logs/cancel` contract and a job record.

Stop on missing artifacts, role overlap, empty enabled mining, class drift, or

failed training. Never infer live state from the JSON record alone.

## Retrieval preparation

Build the reusable candidate cache once, then prepare queries after each pair

of loose/strict gap jobs:

```bash

scripts/prepare_deft_od_aoi_retrieval.py candidates \

--policy "$RESULTS/deft_od_aoi_policy.yaml" \

--output-dir "$RESULTS/candidates"

scripts/prepare_deft_od_aoi_retrieval.py queries \

--policy "$RESULTS/deft_od_aoi_policy.yaml" \

--strict-gaps "$ITER/strict/box_gaps.parquet" \

--loose-gaps "$ITER/loose/box_gaps.parquet" \

--iteration 1 --candidate-root "$RESULTS/candidates" \

--output-dir "$ITER/retrieval"

```

Strict FNs and loose near-miss FPs route to real data. Background-like loose

## Admission and cumulative COCO

```bash

scripts/admit_deft_od_aoi_coco.py \

--policy "$RESULTS/deft_od_aoi_policy.yaml" \

--candidate-root "$RESULTS/candidates" \

--retrieval-root "$ITER/retrieval" \

--previous-coco "$PREVIOUS/train.json" \

--output-dir "$ITER/training_data"

```

It retains every prior image and box and emits one binary COCO with explicit

zero-annotation clean images. Use `--link-mode hardlink` only when source and

output share a filesystem; portable staging should keep the copy default.

## Measurement specs

For the baseline, pass `--baseline`. The default `cold_start` mode creates an

empty prediction file for every KPI image, making all KPI ground-truth boxes

initial false negatives without loading an incompatible multiclass checkpoint

into the binary inference head. It also emits cardinality-matched empty test

prediction evidence so the measurement manifest retains the same KPI/test

completion contract without running inference. Explicit `checkpoint` mode

emits normal baseline KPI/test inference specs and therefore requires a

binary-compatible checkpoint. Neither mode changes training initialization.

```bash

scripts/prepare_deft_od_aoi_measurement.py \

--policy "$RESULTS/deft_od_aoi_policy.yaml" \

--checkpoint "$CHECKPOINT" \

--kpi-predictions "$MEASURE/kpi/inference/labels" \

--results-root "$MEASURE" --output-dir "$MEASURE/specs" \

--baseline

```

Submit only the inference specs listed in `measurement_manifest.json`; a

cold-start baseline lists no inference specs. Then submit `gap_loose.yaml` and

`gap_strict.yaml` through

`tao-analyze-gaps-od-map`. The helper projects the frozen KPI COCO to KITTI,

keeps the two-line inference class map durable, and emits only nested specs.

Test inference is report-only and its output cannot influence routing or

checkpoint selection.

After training, omit `--baseline`; every iteration measurement uses the

selected binary checkpoint and emits KPI/test inference regardless of the

baseline mode.

## RT-DETR training specs

Build nested specs from the current cumulative COCO:

```bash

scripts/prepare_deft_od_aoi_training.py \

--policy "$RESULTS/deft_od_aoi_policy.yaml" --iteration 1 \

--train-coco "$ITER/training_data/train.json" \

--train-images "$ITER/training_data/images" \

--results-root "$ITER/training" --output-dir "$ITER/train_specs"

```

Invoke the RT-DETR leaf as direct training with `automl_policy: off`; the AOI

application already owns its frozen probe policy. Iterations 1–2 use 36 epochs.

Later iterations use the clipped size-adaptive budget. When probes are enabled

and the iteration reaches the frozen `training.probes_start_iteration`,

the helper emits independent incumbent, data-growth-scaled, and deterministic

jitter specs, all initialized from the same frozen base checkpoint. Run all

three before selecting and materializing `train.yaml`; never initialize the

next iteration from a previous iteration checkpoint.

Select all three probes before main training:

```bash

scripts/select_deft_od_aoi_training.py probes \

--manifest "$SPECS/training_manifest.json" \

--main-template "$SPECS/main_template.yaml" \

--status "$P0/status.json" --status "$P1/status.json" --status "$P2/status.json" \

--output-dir "$SPECS/selected"

```

After main training, select the KPI-best checkpoint from structured status

rows. Repeat `--status` for resumed phases. If the best epoch is within the

frozen late window, the first call emits one same-iteration `extension.yaml`

from the terminal checkpoint; submit it, then reselect with

`--extension-applied`. The checkpoint carried to measurement is always the

maximum KPI `val_mAP50`, never the latest checkpoint or test result.

After each stage succeeds, commit at least one completion artifact with

`commit_deft_od_aoi_stage.py`. It accepts only the next frozen stage, verifies

and hashes every named file, atomically updates `deft_state.json`, and appends

`loop_log.jsonl`. The final iteration becomes `COMPLETE` only after its gap

artifacts are committed. Poll native backends for live state; this record is

durable workflow history, not a scheduler substitute.

## Optional synthesis with existing task weights

Enable `synthesis` only when every KPI sample retains its real, nonempty

`dataset_id`; a missing ID is malformed KPI metadata and is never an opt-out.

`synthesis.routes` is the synthesis allowlist. Dataset IDs absent from it skip

synthesis but remain in the separate normal real-data DEFT path. Every FN in a

configured route must also carry `texture_id`, `defect_class`, and a pixel

`fn_mask_source`, and the route must provide an existing AnomalyGenNext

checkpoint and matching recipe. After strict gap analysis, normalize exact

FN/annotation matches:

```bash

scripts/prepare_deft_od_aoi_synthesis.py \

--policy "$RESULTS/deft_od_aoi_policy.yaml" \

--strict-gaps "$MEASURE/gap_strict/box_gaps.parquet" \

--output-dir "$ITER/synthesis_request"

```

Pass the emitted filtering YAML through `tao-prepare-anomalygennext-inputs`,

`iteration_synthesis` before training. Re-run admission with the generated

the frozen cumulative fraction cap is applied against admitted real defects.

Boxes alone never substitute for the required pixel mask. If the request has no

routed FNs, or preparation reports no mask-eligible FNs, commit that typed skip

contract as `iteration_synthesis` and continue to training without running AMP

or fabricating generation/admission success. Missing or empty AnomalyGen clean

reference pools likewise skip only affected FN/types with

`no_clean_reference_images`; eligible types and real/clean retrieval continue.

Initialization rejects synthesis when the entire configured reference pool has

no supported image under any `TEXTURE/clean_image` directory.

If no FN remains eligible, the preparation-level reason is

`no_eligible_false_negatives` while per-FN evidence retains the specific cause.

If every producer is unavailable, the typed skip commits convergence instead of

training on an unchanged iteration.

## Missing AnomalyGenNext task weights

A synthesis route may replace `checkpoint` and `recipe` with a `finetune` block

containing `dataset_root`, frozen `validation_testcase`, Cosmos3-Nano

optional user `recipe_template`/`defect_spec`. These are AnomalyGenNext inputs;

the application policy is not an upstream training recipe.

Run `resolve_deft_od_aoi_synthesis.py` before the candidate cache. If a handoff

is absent, it emits `finetune_requests.json`; execute each request through

`tao-finetune-anomalygennext` once. Resolve again into a new directory, verify

the hash-bound handoff, commit `synthesis_bootstrap`, and use the emitted

`resolved_synthesis_policy.yaml` for all later synthesis preparation. Never

start or resume AnomalyGenNext training inside a DEFT iteration.