49 added, 142 removed. Audit A to A.
---
name: build-and-push-image
- description: Use when building, tagging, validating, or pushing NPA workbench container images for Nebius registry-backed workflows.
+ description: Use when building, tagging, validating, or publishing NPA workbench container images through immutable full-SHA public development tags and digest-identical supported GHCR releases.
---
# Build And Push Image
- ## When To Use
-
- Use this skill when a task changes Dockerfiles, image manifests, registry tags,
- or workflow image references for NPA workbench tools.
-
- ## Procedure
-
- For SkyPilot workflow images, build/test the versioned bootstrap contract and
- record it in OCI config. Preflight consumes the selected digest, not its tag.
- Keep live-validation tags unique, scan built bytes for restricted payloads, and
- push only to the authorized project registry unless shared publication is
- explicitly requested.
-
- 1. Use the anonymous GHCR default for published images. For a private or locally
- modified image, select the destination explicitly with `NPA_REGISTRY`, a
- build-script `--registry`, or the relevant command's image option. Existing
- saved `container_registry` overrides still resolve for compatibility.
- 2. Build from the checked-in Dockerfile for the tool; do not invent a detached
- image source outside the repo.
- 3. Tag images with the configured registry prefix and a version that matches the
- tool manifest or release plan.
- 4. Inspect the image or manifest before pushing.
- 5. Update image manifests, workflow YAML, and skill guidance together when a
- command starts depending on the new image.
-
- ## Three-Tier Contract
-
- - CLI: use `docker buildx build`, tool-specific `deploy --container-image`, and
- command help from the affected workbench tool.
- - SDK: resolve registry and project settings through `npa.clients.config` instead of
- hardcoded env reads.
- - YAML: workflow `image_id` values should come from variables or manifests,
- especially SONIC's `npa/src/npa/deploy/sonic_image_manifest.json`.
-
- ## Packaging Contract
-
- Before changing Dockerfiles, read `docs/workbench/container-packaging.md` and
- update `npa/docker/workbench/packaging-contract.yaml` when adding an image or
- changing its tier (`service` / `job` / `interactive`).
-
- Security baseline: non-root final USER, no secrets in layers, digest-pinned
- bases where possible, Trivy scan coverage. Service images should expose ports
- and prefer a `HEALTHCHECK` or K8s probe on `/health`.
-
- ## Redistribution Class
-
- Every image in the packaging contract also declares
- `redistribution: public | restricted`, which decides whether it may leave the
- owning org:
+ Load and obey `skills/atomic/secure-image-build/SKILL.md` before every image
+ build, tag, push, copy, or promotion. That skill defines the mandatory order and
+ refusal conditions; this skill records NPA-specific build and GPU details.
- - `public` — OSS-redistributable, may be mirrored to a public registry. Every
- canonical image in `CONTAINER_IMAGE_NAMES` is currently `public`.
- - `restricted` — bakes a runtime we may not redistribute. The four
- Isaac images (`isaac-lab`, `sonic`, `sonic-mujoco`, `groot`) used to be restricted
- because they baked Omniverse Kit, and were re-architected to fetch Isaac Sim / Isaac
- Lab at first run under the operator's own EULA acceptance instead. The separately
- contracted `cosmos3-serving` image is now restricted/build-your-own because its
- pinned vendor base carries derived-distribution conditions that anonymous GHCR does
- not establish.
+ ## Build Contract
- When adding an image, set its class. `npa/tests/docker/test_packaging_contract.py` fails
- the build if a Dockerfile **bakes** Omniverse Kit (or is built `FROM` a restricted image)
- while claiming `public`. Keep `images.OMNIVERSE_RESTRICTED_TOOLS` in sync; it is what
- `npa.deploy.publish_public` uses to decide what may be mirrored publicly.
+ 1. Build only a checked-in Dockerfile under `npa/docker/workbench/<tool>/`.
+ 2. Resolve the exact 40-character commit and use the one official namespace,
+ `ghcr.io/nebius/nebius-physical-ai`. A pre-release tag is exactly
+ `dev-<full-git-sha>` on the normal `npa-<tool>` package.
+ 3. Require `redistribution: public` before any official push. Build restricted
+ images, including `cosmos3-serving`, only into an operator-controlled
+ registry; neither a private package nor a development tag changes licensing.
+ 4. Run every pre-publication security, packaging, payload, provenance, SBOM,
+ vulnerability, secret, non-root, base-pin, and bootstrap-contract gate before
+ pushing the public development tag.
+ 5. After push, resolve the immutable digest, repeat exact-digest checks, and
+ verify anonymous pullability. Use that digest for functional GPU validation.
+ 6. Promote only the real-GPU-validated digest to the supported release tag with
+ `npa.deploy.publish_public`; verify digest parity and anonymous pull afterward,
+ then record the accepted release digest for anonymous read-only health checks.
- Note the distinction the guard encodes: **baked at build time** vs. **fetched at run
- time**. Mentioning `isaacsim` in bootstrap plumbing is fine; installing it in a `RUN`
- layer is not. Two of its patterns exist specifically because the runtime-fetch design
- created new ways to bake by accident — `RUN isaac-bootstrap ensure` and
- `RUN /isaac-sim/python.sh ...` both materialise the whole install into a layer. If you
- need a build-time interpreter in an Isaac image, use the image's own venv python (see
- `Dockerfile.mujoco`'s `ARG NPA_IMAGE_PYTHON`), never the shim.
+ Use `NPA_PUBLIC_REGISTRY` only to select the configured official namespace.
+ `NPA_REGISTRY` remains the generic execution override. Do not introduce a second
+ official source registry or a separate pre-release package naming convention.
- ### Building an Isaac image
+ ## Packaging And Runtime Fetch
- No NGC credentials are needed — nothing credentialed is left to pull:
+ Read `docs/workbench/container-packaging.md` and update
+ `npa/docker/workbench/packaging-contract.yaml` when an image or tier changes.
+ Make the final stage non-root, pin resolvable bases by digest, keep credentials
+ and generated data out of layers, and make service health machine-checkable.
- ```bash
- npa/docker/workbench/isaac-lab/build.sh --registry cr.<region>.nebius.cloud/<id> --push
- npa/docker/workbench/sonic/build.sh --registry cr.<region>.nebius.cloud/<id> --push --variant baked
- ```
+ Runtime-fetch images must prove absence on the built artifact. Use
+ `scan_image_omniverse_payload.py` for Isaac-family images and
+ `scan_image_ltx_payload.py` for LTX. LTX source and weights both remain runtime
+ fetches under the operator's entitled `HF_TOKEN`; never fetch either at build
+ time or cache acceptance in the image.
- Two practical notes from doing this on the dev VM: an Isaac image build can peak at ~90 GB
- of scratch, so `docker builder prune -af` between builds, and prefer `--push` (buildx
- streams to the registry) over a local build, which additionally unpacks ~30 GB into the
- image store. Verify the result with
- `npa/.venv/bin/python npa/scripts/scan_image_omniverse_payload.py <ref>`.
+ For SkyPilot workflow images, prove the versioned bootstrap behavior before
+ adding its OCI label. A label is an attestation of tested behavior, not a switch
+ that makes the behavior true.
## GPU Architecture Coverage
- An image only runs on a GPU whose architecture it was compiled for, and there
- are two independent knobs:
-
- - A prebuilt torch wheel ships a fixed fat-binary arch set that
- `TORCH_CUDA_ARCH_LIST` cannot change; only the wheel index does. cu128/cu130
- include `sm_100` and `sm_120`, cu124/cu126 stop at `sm_90`. Check with
- `torch._C._cuda_getArchFlags()`. `torch.cuda.get_arch_list()` returns `[]`
- when the build host has no visible GPU and is not valid build-time evidence.
- - Source-compiled extensions (flash-attn from source, Taichi, natten, custom
- ops) obey `TORCH_CUDA_ARCH_LIST` at build time; omitting an arch fails at
- runtime with `no kernel image is available for execution on the device`.
-
- `npa-base` (`base/cuda13-b300`) builds `8.0 9.0 10.0 10.3 12.0` and asserts the
- wheel reports `sm_80 sm_90 sm_100 sm_120`; override with `build.sh --arch-list`
- / `--require-archs`. Validate an image with
- `npa/scripts/validate_blackwell_image.sh <image> --target b200|b300 --gpu`.
- Use additive tags that name the architectures
- (`cuda13-b300-sm80-sm90-sm100-sm103-sm120-<UTC>`) and record the per-image
- verdict in `npa/docker/workbench/blackwell-dc-images.json`. Background:
- `docs/workbench/blackwell-datacenter-image-compatibility.md`.
-
- The current alias for that contract is
- `cuda13-b300-sm80-sm90-sm100-sm103-sm120-v2-latest`. Its `v2` generation is
- additive: it was introduced after physical B300 testing proved that the first
- five-architecture alias baked an older validator which incorrectly demanded a
- literal `sm_103` wheel entry. The v2 image bakes the same-major `sm_100` →
- `sm_103` SASS rule. Never repoint either historical alias; introduce a new
- truthful generation if another published contract must be superseded. Retain
- `cuda13-b300-sm80-sm90-sm120-latest` and the first five-architecture alias only
- where historical evidence refers to them.
-
- ### Registry-native parent rebases
-
- A registry-native parent rebase is an allowed recovery when the dev VM cannot
- materialize a very large, otherwise unchanged child image. Use it only when all
- of these are true:
-
- - the child Dockerfile instructions and application payload are unchanged
- except for selecting the replacement parent image;
- - the child does not compile CUDA extensions during its build;
- - the old child's layer list starts with the exact old parent layer list;
- - the rebased child's layer list starts with the exact new parent layer list,
- and its remaining child-layer suffix is byte-for-byte identical;
- - provenance labels are updated, the tag is additive, both registry digests
- match, and the exact final tag passes its baked validators plus a real GPU
- capability smoke.
-
- Describe this result as **rebased**, not rebuilt. Never use this shortcut for a
- changed Dockerfile, a child that compiles extensions, or as a substitute for
- physical GPU execution.
-
- ## Gotchas
+ Choose the GPU with `skills/atomic/gpu-selection/SKILL.md`. Prebuilt Torch wheels
+ ship fixed architecture sets; `TORCH_CUDA_ARCH_LIST` affects only source-built
+ extensions. Check wheel flags with `torch._C._cuda_getArchFlags()` and validate
+ custom kernels on the target hardware.
- - Do not commit concrete registry IDs or private image digests from a live
- account unless the repo already treats that value as public.
- - Nebius registry auth expires; a push or pull failure may require a refreshed
- token rather than an image change.
- - For GPU-specific images, verify the target GPU family before changing defaults.
+ `npa-base`'s CUDA 13 contract covers `sm_80`, `sm_90`, `sm_100`, `sm_103`, and
+ `sm_120`; its wheel must report `sm_80`, `sm_90`, `sm_100`, and `sm_120`.
+ Validate datacenter Blackwell images with
+ `npa/scripts/validate_blackwell_image.sh <image> --target b200|b300 --gpu` and
+ record truthful additive tags in `npa/docker/workbench/blackwell-dc-images.json`.
## Verify
```bash
- npa/.venv/bin/python -m pytest npa/tests/guardrails/test_skills_index.py -q
+ npa/.venv/bin/python -m pytest \
+ npa/tests/docker/ npa/tests/deploy/ \
+ npa/tests/guardrails/test_secure_image_build_skill.py \
+ npa/tests/guardrails/test_skills_index.py -q
```
-
- The smoke test verifies current deploy command help and the image-manifest file
- paths used by the skill.