--- name: build-and-push-image description: Use when building, tagging, validating, or pushing NPA workbench container images for Nebius registry-backed workflows. --- # Build And Push Image ## When To Use Use this skill when a task changes Dockerfiles, image manifests, registry tags, or workflow image references for NPA workbench tools. ## Procedure For SkyPilot workflow images, build/test the versioned bootstrap contract and record it in OCI config. Preflight consumes the selected digest, not its tag. Keep live-validation tags unique, scan built bytes for restricted payloads, and push only to the authorized project registry unless shared publication is explicitly requested. 1. Resolve runtime registry settings with `npa configure` or `npa.clients.config.resolve_container_registry` or `npa configure`. 2. Build from the checked-in Dockerfile for the tool; do not invent a detached image source outside the repo. 3. Tag images with the configured registry prefix and a version that matches the tool manifest or release plan. 4. Inspect the image or manifest before pushing. 5. Update image manifests, workflow YAML, and skill guidance together when a command starts depending on the new image. ## Three-Tier Contract - CLI: use `docker buildx build`, tool-specific `deploy --container-image`, and command help from the affected workbench tool. - SDK: resolve registry and project settings through `npa.clients.config` instead of hardcoded env reads. - YAML: workflow `image_id` values should come from variables or manifests, especially SONIC's `npa/src/npa/deploy/sonic_image_manifest.json`. ## Packaging Contract Before changing Dockerfiles, read `docs/workbench/container-packaging.md` and update `npa/docker/workbench/packaging-contract.yaml` when adding an image or changing its tier (`service` / `job` / `interactive`). Security baseline: non-root final USER, no secrets in layers, digest-pinned bases where possible, Trivy scan coverage. Service images should expose ports and prefer a `HEALTHCHECK` or K8s probe on `/health`. ## Redistribution Class Every image in the packaging contract also declares `redistribution: public | restricted`, which decides whether it may leave the owning org: - `public` — OSS-redistributable, may be mirrored to a public registry. Every canonical image in `CONTAINER_IMAGE_NAMES` is currently `public`. - `restricted` — bakes a runtime we may not redistribute. The four Isaac images (`isaac-lab`, `sonic`, `sonic-mujoco`, `groot`) used to be restricted because they baked Omniverse Kit, and were re-architected to fetch Isaac Sim / Isaac Lab at first run under the operator's own EULA acceptance instead. The separately contracted `cosmos3-serving` image is now restricted/build-your-own because its pinned vendor base carries derived-distribution conditions that anonymous GHCR does not establish. When adding an image, set its class. `npa/tests/docker/test_packaging_contract.py` fails the build if a Dockerfile **bakes** Omniverse Kit (or is built `FROM` a restricted image) while claiming `public`. Keep `images.OMNIVERSE_RESTRICTED_TOOLS` in sync; it is what `npa.deploy.publish_public` uses to decide what may be mirrored publicly. Note the distinction the guard encodes: **baked at build time** vs. **fetched at run time**. Mentioning `isaacsim` in bootstrap plumbing is fine; installing it in a `RUN` layer is not. Two of its patterns exist specifically because the runtime-fetch design created new ways to bake by accident — `RUN isaac-bootstrap ensure` and `RUN /isaac-sim/python.sh ...` both materialise the whole install into a layer. If you need a build-time interpreter in an Isaac image, use the image's own venv python (see `Dockerfile.mujoco`'s `ARG NPA_IMAGE_PYTHON`), never the shim. ### Building an Isaac image No NGC credentials are needed — nothing credentialed is left to pull: ```bash npa/docker/workbench/isaac-lab/build.sh --registry cr..nebius.cloud/ --push npa/docker/workbench/sonic/build.sh --registry cr..nebius.cloud/ --push --variant baked ``` Two practical notes from doing this on the dev VM: an Isaac image build can peak at ~90 GB of scratch, so `docker builder prune -af` between builds, and prefer `--push` (buildx streams to the registry) over a local build, which additionally unpacks ~30 GB into the image store. Verify the result with `npa/.venv/bin/python npa/scripts/scan_image_omniverse_payload.py `. ## GPU Architecture Coverage An image only runs on a GPU whose architecture it was compiled for, and there are two independent knobs: - A prebuilt torch wheel ships a fixed fat-binary arch set that `TORCH_CUDA_ARCH_LIST` cannot change; only the wheel index does. cu128/cu130 include `sm_100` and `sm_120`, cu124/cu126 stop at `sm_90`. Check with `torch._C._cuda_getArchFlags()`. `torch.cuda.get_arch_list()` returns `[]` when the build host has no visible GPU and is not valid build-time evidence. - Source-compiled extensions (flash-attn from source, Taichi, natten, custom ops) obey `TORCH_CUDA_ARCH_LIST` at build time; omitting an arch fails at runtime with `no kernel image is available for execution on the device`. `npa-base` (`base/cuda13-b300`) builds `8.0 9.0 10.0 10.3 12.0` and asserts the wheel reports `sm_80 sm_90 sm_100 sm_120`; override with `build.sh --arch-list` / `--require-archs`. Validate an image with `npa/scripts/validate_blackwell_image.sh --target b200|b300 --gpu`. Use additive tags that name the architectures (`cuda13-b300-sm80-sm90-sm100-sm103-sm120-`) and record the per-image verdict in `npa/docker/workbench/blackwell-dc-images.json`. Background: `docs/workbench/blackwell-datacenter-image-compatibility.md`. The current alias for that contract is `cuda13-b300-sm80-sm90-sm100-sm103-sm120-v2-latest`. Its `v2` generation is additive: it was introduced after physical B300 testing proved that the first five-architecture alias baked an older validator which incorrectly demanded a literal `sm_103` wheel entry. The v2 image bakes the same-major `sm_100` → `sm_103` SASS rule. Never repoint either historical alias; introduce a new truthful generation if another published contract must be superseded. Retain `cuda13-b300-sm80-sm90-sm120-latest` and the first five-architecture alias only where historical evidence refers to them. ### Registry-native parent rebases A registry-native parent rebase is an allowed recovery when the dev VM cannot materialize a very large, otherwise unchanged child image. Use it only when all of these are true: - the child Dockerfile instructions and application payload are unchanged except for selecting the replacement parent image; - the child does not compile CUDA extensions during its build; - the old child's layer list starts with the exact old parent layer list; - the rebased child's layer list starts with the exact new parent layer list, and its remaining child-layer suffix is byte-for-byte identical; - provenance labels are updated, the tag is additive, both registry digests match, and the exact final tag passes its baked validators plus a real GPU capability smoke. Describe this result as **rebased**, not rebuilt. Never use this shortcut for a changed Dockerfile, a child that compiles extensions, or as a substitute for physical GPU execution. ## Gotchas - Do not commit concrete registry IDs or private image digests from a live account unless the repo already treats that value as public. - Nebius registry auth expires; a push or pull failure may require a refreshed token rather than an image change. - For GPU-specific images, verify the target GPU family before changing defaults. ## Verify ```bash npa/.venv/bin/python -m pytest npa/tests/guardrails/test_skills_index.py -q ``` The smoke test verifies current deploy command help and the image-manifest file paths used by the skill.