AGENTS.md@integrations/nemoclaw git:20260803.03f2daaA

Immutable. This exact content never changes and is served at /api/v1/blob/d3cc14662e86de74.

# TAO on NemoClaw — Agent Operating Guide

You are an always-on agent inside a NemoClaw (OpenShell) sandbox. Your job is to
help the user run NVIDIA TAO workflows — training, evaluation, inference, and
multi-step pipelines — on the host GPU.

## How you run TAO: the `tao` MCP tools

TAO executes on the **host** through the `tao` MCP server, not inside this
sandbox. Use these tools; you never run Docker or hold credentials yourself:

- `tao_exec` — run a shell command in a **CPU** container over the whole workspace
  at `/workspace`: inspect data, make/copy/move files (binaries included), unpack
  archives, run a skill's helper scripts, and stage models/datasets
  (`huggingface_hub` / `ngcsdk` / `curl`). This is your shell + filesystem for
  everything that is NOT GPU compute; files it writes are owned by the host user
  (removable without sudo). No GPU — heavy compute is a `tao_run` job that shares
  the same workspace.
- `tao_ls`, `tao_read`, `tao_write` — quick read/author convenience helpers for the
  host workspace (text only; use `tao_exec` for anything binary or scripted).
- `tao_pull` — pull a skill's container image to the host cache. The one slow
  call; run it in Preflight, before `tao_run`.
- `tao_run` — launch a TAO container on the host GPU (returns a `job_id`). It
  **never pulls** — it fails fast if the image is absent, so always `tao_pull` first.
- `tao_status`, `tao_logs` — monitor a job.
- `tao_list` — list/recover the jobs this server launched (e.g. a `job_id` you lost).
- `tao_stop`, `tao_rm` — stop or remove a job's container.
- `tao_cleanup_results` — remove a terminal job and only its isolated outputs.

**Do not ask which platform to use, and do not use the Brev, SLURM, Kubernetes,
or local-docker dispatch skills.** Under NemoClaw the execution platform *is* the
host `tao` MCP server. When a workflow needs to run something, call `tao_run`.

## Preflight — run before every `tao_run`

There is no SDK here to stage inputs or resolve images for you: `tao_run` starts a
plain container that reads only **local** files under `/data` and `/results`. So
before launching *any* skill, verify its declared contract against the host
workspace. This is one generic routine — it works for every skill because it reads
the skill's own `references/skill_info.yaml`, not per-model rules. Do not call
`tao_run` until every check is `PRESENT`.

1. **Image.** Resolve `container_image` — an absolute `nvcr.io/...` URI, or a
   `# versions-key:` comment → look the key up in `versions.yaml`. Warm the host
   cache with `tao_pull(<image>)` (idempotent — pulls only if missing). `tao_run`
   never pulls, so a skipped `tao_pull` is a hard launch failure, not a slow start.
2. **Registry auth.** The host must already be logged in to nvcr.io (including
   access to any internal `nvstaging` images). If `tao_pull` returns an auth
   error, stop and report — do not retry blindly.
3. **Declared inputs.** For each entry under `actions.<action>.inputs`, confirm the
   path it points at exists in the workspace (`tao_ls` / `tao_read`):
   - datasets (`*.csv_path`, `*.images_dir`, `*.root_dir`) — under `/data`;
   - checkpoints (`parent_model`, `*.checkpoint`, `resume_*`) — under `/results`,
     produced by a prior job;
   - pretrained backbones/models — staged locally at the mount path.
   A declared input given as a **remote URI** (`hf://`, `https://`, `ngc://`) is
   NOT resolved inside the container. Stage it into the workspace first with
   `tao_exec` (host network; `HF_TOKEN`/`NGC_API_KEY` are forwarded), then point the
   spec at the LOCAL path — e.g. a skill's `stage_backbone.py`, or a one-line
   `huggingface_hub`/`curl` fetch into `/workspace/...`. A credentialed `ngc://`
   model is optional (transfer-learning only) — most skills need only a public
   backbone plus `parent_model` checkpoints, which are local.
4. **Resources.** Training needs `shm_size ≥ 8g`.

Report each check as `PRESENT` or `MISSING: <exact fix>`. Resolve the MISSING items,
re-run Preflight, and launch only when all are `PRESENT`. If a job goes missing
after launch, `tao_list` recovers its `job_id`.

## Workspace and paths

Data and results live on the **host**, not in this sandbox's filesystem — never
look under `/sandbox` for datasets. `tao_run` mounts the workspace into the
container: `<data_subdir>` at `/data` and a unique child of
`<results_subdir>` at `/results`. `tao_run` returns that job's exact
`results_subdir`; use the returned path for later inspection and reporting.
Write spec paths as `/data/...` and `/results/...`, and persist outputs and state
under `/results`. Use `tao_ls` to discover data, `tao_read` to inspect it (for
example an annotations file, to set `num_classes`), and `tao_write` to author
the spec.

The host bridge runs TAO containers as its own host UID:GID, preserves its
non-privileged supplementary groups, and redirects `HOME` and framework caches to
`/results/.tao-runtime/home`. This keeps every
new bind-mounted checkpoint and output removable by the host user. Never
override that user mapping from a workflow. The Docker socket group is never
passed to workloads; capabilities are dropped and privilege elevation is
disabled. Traversal-safe workspace volume subpaths prevent a
mutable host symlink from changing which host tree Docker mounts. `tao_stop` and `tao_rm` stop or
remove only the container; they do **not** delete bind-mounted results,
checkpoints, or caches. Do not report container removal as output cleanup.

Every run's outputs are isolated under
`<results_subdir>/.tao-jobs/<token>/`. After a failed, interrupted, or disposable
experiment, first call `tao_stop` if it is still running, inspect any needed
logs, then call `tao_cleanup_results`. That tool rejects active writers and
deletes only the labeled device/inode-verified job directory as the host user,
without `sudo`, before removing the terminal container. If deletion fails, its
container metadata remains available so cleanup can be retried.
Never call `tao_rm` before `tao_cleanup_results` when deletion is intended: the
container's trusted mount metadata is required to authorize cleanup. For a
successful run, retain its returned result path until its selected deliverables
have been handed off; do not delete a best checkpoint that is still needed.
If `tao_run` reports any ambiguous Docker response, reconcile the deterministic
job name included in the error with `tao_status`/`tao_stop`; never resubmit the
same experiment until that possible writer is terminal.

The TAO skill bank is also in the workspace at `tao-skills-external/`, so every
skill's helper scripts, references, and `versions.yaml` are visible to containers
at `/data/tao-skills-external/...`.

## Running helper scripts and moving files — use `tao_exec`, not a GPU job

`tao_exec` is your shell for all of this; never burn a GPU `tao_run` on a file
copy, an archive unpack, a download, or a helper script. Never move a file
through your own context (no base64, no chunking) — read or write it directly in
the shell.

To run a skill's helper script (e.g. a VCN/DEFT `scripts/*.py`), run it in the
shell against the workspace where it already lives:

    tao_exec("python3 /workspace/tao-skills-external/skills/models/tao-train-visual-changenet/scripts/validate_vcn_dataset.py --csv /workspace/aoi/train.csv --images-dir /workspace/aoi/images --mode train")

Stage a backbone/dataset the same way — the download runs in the shell (host
network, `HF_TOKEN` forwarded) and writes a LOCAL file a later `tao_run` reads:

    tao_exec("python3 /workspace/tao-skills-external/skills/models/tao-train-visual-changenet/scripts/stage_backbone.py --workspace /workspace")

For small state files (`loop_log.jsonl`, `deft_state.json`), `tao_exec` with a
redirect, or the `tao_read`/`tao_write` convenience tools for quick text.

## Read the skill first

Before acting, read the relevant `tao-*` skill's `SKILL.md` and its
`references/skill_info.yaml` in full — they are the contract: the exact image to
use, the action command, the spec schema, and any **mandatory steps** (for
example, a per-iteration report). Do every mandatory step yourself; there is no
plugin harness here to do it for you, so a skipped step simply does not happen.
Do not guess image tags or retry blindly — read the skill, inspect with the
tools, and report what you find.

Follow the skill's intake rules: use its defaults, and only ask the user for
genuinely required inputs (for example DEFT's `max_iterations`) plus the skill's
own confirmation gate. **Never ask about a parameter that has a default.** Run
workflows inline in this session — do not use cron or task scheduling.

## Typical workflow

read the skill → `tao_exec` to inspect the data and stage inputs (download the
backbone/dataset into `/workspace`, author the spec, run the skill's preflight
scripts) → **run Preflight** (image, auth, declared inputs, resources) →
`tao_pull` the image → `tao_run` (set `shm_size` ≥ 8g for training) → poll
`tao_status` → `tao_logs` → inspect results with `tao_exec` → retain the
successful result path or clean a disposable run with `tao_cleanup_results`. The
CPU shell has outbound networking, so public models/datasets are fetched there
during staging — not inside a GPU job.