--- name: fleet description: Use to deploy or operate a fleet of Nebius Managed Kubernetes (k8s-training) clusters across one or many projects in a tenant from an npa.fleet/v0.0.1 spec — including strict capacity-block-backed GPU pools, identical and/or custom clusters, create-on-demand projects, and a k8s-training recipe source that can consume the latest upstream changes. --- # Fleet (multi-cluster, multi-project Kubernetes) ## When To Use Use when a customer wants **more than one** Managed Kubernetes cluster stood up across **one or many projects** in a single Nebius tenant — e.g. per-team or per-tenant training clusters — and wants npa to drive it from one declarative file. `npa fleet` wraps the public `nebius/nebius-solutions-library` `k8s-training` recipe (the same recipe `npa cluster up` uses) once per cluster, and can create the target projects on demand. For a single cluster, prefer `npa cluster up`. For Slurm-on-Kubernetes, use `npa soperator`. Three-tier contract: - **CLI**: `npa fleet plan|deploy|destroy|status|verify-storage|verify-graphics --spec `. - **SDK**: `npa.sdk.fleet.deploy(spec)` / `destroy` / `plan` / `status` with `FleetSpec` / `ProjectSpec` / `ClusterSpec` / `NodePoolSpec`. - **YAML / agent**: `apiVersion: npa.fleet/v0.0.1` spec; workflow `toolRef: infra.fleet.deploy` (config key `fleet_spec`). ### Opt-in CPU RayCluster Use `kuberay: {enabled: true, worker_replicas: 2, worker_cpus: 2, worker_memory_gib: 4}` in a cluster/default profile for a fixed CPU worker group. The shared backend validates strict types, explicit CPU placement and the exact reviewed recipe before provisioning. It also rejects head or worker pods larger than one declared CPU node and fixed requests above the pool's nominal aggregate CPU or memory; operators must still leave room for Kubernetes system pods. `KubeRaySpec` is exported by `npa.sdk.fleet`. A disabled block replaces enabled defaults atomically. Omitted policy remains off. Enabled deployments require the default local Terraform workspace without `TF_CLI_ARGS*` or `TF_DATA_DIR` overrides. Reapply rejects extra effective inputs and destination symlinks while preserving exact state/backup and provider caches. Retained backend metadata is unsupported; require managed implicit local state. The module manifest is rebuilt from the reviewed recipe before Terraform init. Historical opt-in provenance survives disabled/omitted policy reapply. Teardown checks the same effective-input restrictions and recorded materialized digest, then requires empty managed state before removing recovery files. Preserve state on refusal or incomplete teardown; missing provenance needs a reviewed reapply. See `docs/fleet-kuberay.md` and `npa/examples/fleet/kuberay/cpu-raycluster.yaml` for the one-entry fleet and native Ray Jobs journey. The separate CPU template pins Ray 2.58, disables autoscaling and bundled monitoring, uses non-root pods and installs namespace ingress isolation. GPU workers, RayService, arbitrary images and alternate recipe bytes are unsupported; do not recreate a Jobs submitter/controller or enable the old privileged vendor template. Preserve application outputs before owned teardown. Run `npa/tests/e2e/test_fleet_kuberay_live.py` with `NPA_FLEET_KUBERAY_LIVE_CONFIG` pointing at an owner-private JSON object with `spec`, `kubeconfig`, and `evidence_dir`. It inspects deployed values and executes real deterministic work on every Ray worker through native Jobs status/logs. ### Verify an existing shared filesystem Run `npa fleet verify-storage --spec --output json` to qualify every CPU and GPU worker. The shared implementation is `npa.fleet.storage_verification.verify_storage`, also exported as `npa.sdk.fleet.verify_storage`. Use `--only-projects`, `--only-clusters`, `--project-prefix`, and `--profile` to preserve Fleet selection and identity semantics. Unknown selectors and missing or stale registered identity fail closed; explicitly disabled filesystems are skipped. The verifier proves the exact read-write virtiofs source/path, reboot-safe `nofail` entry, capacity in binary GiB, unique host-file checksums, CSI health, and one RWX PVC shared across pods pinned to every exact worker. Every pod checks every worker's unique payload. It reads no pre-existing customer entries. Cleanup removes only owned probe paths and temporary resources with identity labels and UID preconditions, then proves absence on every node using server-synchronized Linux `statx` attributes after owned writers stop. Cached positive directory entries cannot substitute for fresh link-count evidence; unsupported synchronization and persistent linked entries fail closed. Partial evidence or cleanup failure cannot pass. Do not replace this with the vendored single-node shell smoke or infer storage health from node readiness. Use `--evidence-dir ` outside the repository for exact receipts. Publication surfaces receive only sanitized counts, requested capacity, categories, hashes, and cleanup counts. This operation does not redeploy, resize, change IAM, or alter customer workloads. See `docs/fleet-storage-verification.md` for prerequisites and the installed SDK. Run `npa fleet verify-graphics --spec --output json` after an RTX driver, node, or runtime change. It uses the registered Fleet identity, waits for the declared health stability interval, and runs CUDA vectorAdd plus GLX, EGL, and Vulkan qualification on every selected 8-GPU RTX worker. All targets must declare `gpu_workload_profile: rtx-rendering`; a partial target or partial per-node result fails closed. Use `--concurrency` for independent clusters and the standard Fleet selectors to narrow scope. Exact provider and node evidence goes only to the owner-private `--evidence-dir` outside Git; CLI output remains safe to publish. The shared SDK entry point is `npa.sdk.fleet.verify_graphics`. The all-worker live regression is `npa/tests/e2e/test_fleet_storage_verification_live.py`. Supply `NPA_INTEGRATION_E2E=1`, `NPA_FLEET_STORAGE_VERIFY=1`, an owner-private `NPA_FLEET_STORAGE_VERIFY_SPEC`, and `NPA_FLEET_STORAGE_EVIDENCE_DIR`. The daily runner reaches it only with that explicit opt-in. Its expected target, worker, and binary-GiB totals derive from the complete selected declaration. RTX PRO 6000 hardware MIG is an additive cluster policy. Use `mig: {enabled: true, strategy: mixed, config: all-balanced}` only with two strict reserved-capacity `gpu-rtx6000` / `1gpu-24vcpu-218gb` workers and 128 GiB boot disks. NPA pins and live-verifies GPU Operator `v26.3.3`, driver `580.173.02`, device plugin/GFD `v0.19.3`, MIG Manager `v0.14.2`, exact per-node resources, and zero whole-GPU capacity/allocatable. See `docs/fleet-rtx-pro-6000-mig.md` and `npa/examples/fleet/rtxpro-mig.yaml`. RTX rendering without MIG uses the separate explicit cluster settings: ```yaml gpu_workload_profile: rtx-rendering gpu_graphics_smoke_image: '/graphics@sha256:<64-hex-digest>' ``` The graphics image is a reviewed operator-supplied immutable digest. The governed `tool://sonic` default is quarantined, so omitting this field makes the graphics gate refuse before deployment rather than selecting an implicit image. See `docs/workbench/mk8s-gpu-driver-strategy.md` for the direct CLI, SDK, and standalone-RTX forms. The profile selects RTX PRO 6000 (including an explicitly resolved zonal platform such as `gpu-rtx6000-a`), the supported single-GPU preset by default or accepts the explicit `8gpu-192vcpu-1744gb` RTX PCIe preset, uses GPU Operator mounted drivers, and requires per-node GLX/EGL/Vulkan readiness. The 8-GPU RTX shape remains non-fabric and does not opt into the NVSwitch unsafe-operator path. The profile does not alter the empty/default managed-image contract and cannot be generalized to NVSwitch targets. The rendering profile supplies the exact platform **and preset** to the marketplace GPU Operator's `nebius.nvidiaDriverCRDPatch.profiles` values. Its default `gpu-rtx6000` selector does not match zonal `gpu-rtx6000-a` workers. The override contains only the requested RTX pool, keeps RDMA disabled, and uses a revision hash so Terraform reconciles write-only values on existing releases. Alternate recipes must expose and wire `gpu_operator_rtx_driver_profile`; incompatible recipes fail before cloud mutation. Do not relabel workers to hide a selector mismatch. An RTX cluster may declare `gpu_driver_package_repositories` as a mapping of safe filenames to package-source text. Fleet passes it through the Operator's `driver.repoConfig` and an owned ConfigMap, retaining signature verification. Match the sources to the actual driver-image distribution. The five-cluster example supplies Ubuntu 24.04 HTTPS sources for networks that permit HTTPS but block HTTP. Empty configuration preserves the image defaults. Before workloads, verify the driver pod actually mounts the configured files and can fetch its kernel packages; do not infer network reachability from the operator VM. The marketplace may disable automatic upgrades and use an `OnDelete` driver DaemonSet. A successful Helm update then changes the template without replacing existing pods. Compare actual pod mounts with the template. Replacing stale driver pods is maintenance: first verify no application GPU workloads are active, cordon each affected node, recheck workloads, and delete only the identified stale pod with a UID precondition. Wait for its ready replacement and restore the node's prior scheduling state. Retain private receipts and rerun the full CUDA/graphics health gate; never delete application pods or relax readiness to finish an upgrade. Read-only live coverage checks exact selectors, per-node GPU quantities, ConfigMap contents, ready driver containers, and actual repository mounts: ```bash NPA_INTEGRATION_E2E=1 NPA_FLEET_RTX_VERIFY_SPEC= \ NPA_FLEET_RTX_KUBECONFIGS= \ npa/.venv/bin/python -m pytest \ npa/tests/e2e/test_fleet_rtx_driver_config_live.py -q ``` The mapping is `{project_key: {cluster_name: kubeconfig_path}}`; keep its exact values outside Git. This check creates no pods and complements the real CUDA/graphics and representative workload validation. For qualification on **every individual eight-GPU RTX cluster**, additionally run `npa/tests/e2e/test_fleet_rtx_workload_live.py` with the same private spec/mapping, `NPA_FLEET_RTX_RUN_WORKLOADS=1`, and an owner-only `NPA_FLEET_RTX_WORKLOAD_EVIDENCE_DIR`. It creates one Job per target using the provisioning gate's immutable, payload-clean image. Each Job executes verified CUDA arithmetic and matrix multiplication, dynamically loads GLX/EGL, and creates a Vulkan instance with an NVIDIA physical device. It verifies the runtime digest and deletes each Job with a UID precondition, then proves both Job and pod absence. The evidence directory holds private execution receipts and a separate sanitized result summary. This does not replace representative Isaac training or the canonical Sim2Real ladder. The RTX override also sets toolkit `RUNTIME_CONFIG_SOURCE=file`. With newer containerd binaries, `containerd config dump` can migrate the configuration in memory. Using that output may produce a version-4 NVIDIA drop-in beside a version-2 root, preventing containerd startup. Reading the file preserves the host schema; see NVIDIA's [configuration-source guidance](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/getting-started.html#specifying-configuration-options-for-containerd). For an already broken node, apply the corrected toolkit configuration first. After verifying no application workloads are active, retain the root and drop-in files privately, move only the proven incompatible NVIDIA drop-in out of the imports directory, validate `containerd config dump`, and restart containerd. Verify the replacement toolkit produces a compatible fragment and the full health gate passes. Do not change version numbers blindly or discard unrelated host configuration. A node group can report `PROVISIONING` again when its workers lose readiness. Fleet treats it as zero incremental reserved demand only after a complete Compute inventory proves the exact group/cluster/project labels, requested template, running worker count, reservation bindings, and attached disks. Target and actual node counts must also match. This capacity proof does not mark the cluster healthy: every deployment readiness gate still runs. ## Spec (npa.fleet/v0.0.1) The version remains additive. A cluster without `backend` is the historical mk8s shape. Explicit targets use `backend: mk8s` plus `mk8s: {...}` or `backend: soperator` plus `soperator: {...}`; both may appear in one project or fleet. Fleet owns target identity and shared-network inventory, while the selected backend owns one target's plan, materialization, apply, native status, verification, and destroy. See `docs/cluster-backends.md` for one-entry and mixed examples and the fail-closed state rules. A `defaults` cluster profile is deep-merged under every cluster, so **identical** fleets are just project entries with no overrides. Projects may declare custom `clusters` (overrides and/or several clusters), and identical + custom may be freely **mixed**. Projects reference an existing `project_id` or a `name` that is created on demand as `project_prefix` + `name`. Project-scoped `object_storage` is independent of a cluster's shared filesystem. `storage_class: enhanced` maps to Nebius Enhanced Throughput, `size_gibibytes` becomes the bucket's exact binary capacity cap, and deploy requires provider read-back plus a write/read/delete probe. Leave `bucket_name` empty in public specs so Fleet derives a stable private name at runtime. Fleet retains buckets and durable artifacts on cluster destroy; explicit `npa storage bucket delete` remains the destructive cleanup boundary. Fleet uses its own project-scoped storage service account and access key, separate from LeRobot bootstrap identities, and never enables the editors fallback. Storage reconciliation currently requires at least one selected mk8s target in each project that declares storage; soperator-only selections fail before mutation. Before any project, bucket, subnet, or cluster mutation, deploy budgets all new bucket declarations against tenant `storage.bucket.count` and the matching `storage.bucket.size.` allowance (`enhanced-throughput` for Enhanced). The size allowance uses bytes, separately from filesystem quotas. Missing, unreadable, or insufficient evidence fails closed. Every selected existing bucket must match the exact project, name, region, active/unsuspended state, storage class, and cap before reuse. Verified existing buckets are not charged again as new quota demand; a new storage declaration is still checked when its cluster is unchanged. Read/write success without verified probe deletion cannot mark Fleet storage ready. A bucket cap is a maximum, not reserved storage. Available tenant capacity can change after preflight, including when other buckets grow. See the official [bucket contract](https://docs.nebius.com/terraform-provider/reference/resources/storage_v1_bucket) and [storage quotas](https://docs.nebius.com/object-storage/resources/quotas-limits). Run the read-only reservation and storage quota regression against a private planning spec even before its target projects exist: ```bash NPA_INTEGRATION_E2E=1 NPA_FLEET_QUOTA_VERIFY_SPEC= \ npa/.venv/bin/python -m pytest \ npa/tests/e2e/test_fleet_storage_quota_live_e2e.py -q ``` This proves capacity evidence only; project identity, actual bucket provisioning, GPU readiness, and representative workloads remain separate deployment gates. ```yaml apiVersion: npa.fleet/v0.0.1 name: fleet1-test tenant_id: "" # resolved from ~/.nebius + ~/.npa when empty region: us-central1 profile: "" # ~/.nebius profile to authenticate as; "" = active project_prefix: "fleet1-test-" defaults: # For RTX/Isaac rendering clusters, replace the next two driver lines with # gpu_workload_profile: rtx-rendering and an explicit reviewed digest: # gpu_graphics_smoke_image: '/graphics@sha256:<64-hex-digest>' gpu_driver_mode: auto managed_driver_preset: cuda13.0 allow_unsafe_nvswitch_operator: false gpu_health_stabilization_seconds: 120 gpu_health_timeout_minutes: 60 gpu_cuda_smoke: true cpu_nodes: { count: 1, platform: cpu-d3, preset: 48vcpu-192gb } gpu_nodes: count: 1 platform: gpu-rtx6000 preset: 1gpu-24vcpu-218gb # Optional runtime-only ID; renders STRICT and never falls back to PAYG. capacity_block_group: "" enable_filestore: true filestore_disk_size_gibibytes: 1024 filestore_mount_path: /mnt/data filestore_mount_tag: npa-shared-fs # Runtime/operator-supplied standards-based chart source. Keep private # registry endpoints out of committed specs and PR evidence. filesystem_csi_chart_repository: "" projects: - name: a # -> project fleet1-test-a (identical profile) object_storage: # separate from enable_filestore enabled: true storage_class: enhanced size_gibibytes: 1024 - name: b # -> project fleet1-test-b (identical profile) # - name: c # custom: overrides + a second cluster # clusters: # - name: train # gpu_nodes: { count: 2, platform: gpu-h200-sxm, preset: 8gpu-128vcpu-1600gb } # enable_gpu_cluster: true # infiniband_fabric: us-central1-a # - name: infer # inherits defaults # - project_id: project-existing123 # deploy into an existing project by id # clusters: [ {} ] ``` Example spec: `npa/examples/fleet/fleet1-test.yaml`. ## Targeting another tenant (`profile`) A Nebius service account belongs to exactly one tenant, so deploying into a second tenant means authenticating as *that* tenant's principal. Set the spec's `profile:` (or pass `--profile `, which wins) to name a `~/.nebius` profile; every `nebius` CLI call, the minted terraform `TF_VAR_iam_token`, and the generated kubeconfig's exec-credential args are pinned to it. The machine's active profile is never mutated, so concurrent fleets in different tenants stay independent. With a profile set, `tenant_id` resolves from **that** profile's `tenant-id` (never the active profile's) and a profile with no `tenant-id` is a hard error instead of a silent deploy into the wrong tenant. `destroy` falls back to the profile recorded in each cluster's env sidecar at deploy time, so a teardown always authenticates as the principal that created the cluster. Register a service-account profile non-interactively: ```bash nebius profile create --endpoint api.nebius.cloud \ --service-account-id --public-key-id \ --private-key-file-path ~/.nebius/.pem \ --parent-id --tenant-id --skip-auth nebius --profile iam get-access-token >/dev/null # verify ``` `nebius profile create` also *activates* the new profile; re-activate the previous one (`nebius profile activate `) if other tooling on the host depends on it. ## Procedure Parallel applies use a shared download cache only during initialization. NPA holds the cache lock through initialization and copying each target's selected providers into its own Terraform data directory. Later initialization can then rewrite cached downloads without changing binaries used by another apply or destroy. Dependency-lock checksum verification remains enabled. Older recipe sources can omit newer regions from their default-node table. Materialization preserves known-region defaults while allowing explicit platforms and presets in other regions. It supplies no guessed GPU or fabric; resolve those values from the provider inventory before deployment. 1. Keep committed files public-safe: never hardcode tenant/project/registry IDs or SSH keys. The spec resolves tenant/region from `~/.nebius/config.yaml` and `~/.npa/config.yaml` when its fields are empty; the SSH public key comes from `ssh_public_key` or `~/.ssh/id_ed25519.pub` / `id_rsa.pub`. 2. **Plan first** (no infra): `npa fleet plan --spec fleet.yaml` shows the projects (create vs existing) and per-cluster node config. 3. **enable_gpu_cluster is auto**: GPU clustering (InfiniBand fabric) is only valid on fabric-capable 8-GPU SXM presets. Single-GPU presets (e.g. RTX PRO 6000 `1gpu-24vcpu-218gb`) auto-set `enable_gpu_cluster=false`; set it `true` only with an 8-GPU preset **and** `infiniband_fabric`. Every GPU pool defaults to `gpu_driver_mode: auto`, which selects Nebius's managed driver image plus the provider device plugin; CPU-only clusters emit no GPU-driver input. `managed_driver_preset` defaults to the vendored recipe's supported `cuda13.0` and is configurable. `operator` remains an explicit escape hatch, but NVSwitch topologies reject it unless `allow_unsafe_nvswitch_operator: true` acknowledges the Network Operator/MOFED versus Fabric Manager host-device race. The same strategy resolver applies to `npa cluster up` and Fleet. 4. **Bind reserved GPU capacity explicitly when required.** Set `gpu_nodes.capacity_block_group` to a runtime-supplied Capacity Block Group ID. Fleet renders `gpu_nodes_reservation_policy = { policy = "STRICT", ... }`, so an unavailable or incompatible block fails instead of falling back to ordinary on-demand capacity. Never commit a live capacity block ID. Discover and verify reservations read-only with: ```bash nebius --profile

capacity capacity-block-group list \ --parent-id --all --format json nebius --profile

capacity capacity-interval list \ --parent-id --all --format json nebius --profile

capacity capacity-block-group list-resources \ --id --format json nebius --profile

capacity resource-advice list \ --parent-id --all --format json ``` Preflight requires the named block to be active, in the target tenant and region, and matched to the GPU platform and InfiniBand fabric. It checks the aggregate GPU requirement against remaining reserved capacity. Only after that validation does it exclude those GPUs from ordinary GPU quota; all node, boot-disk, GPU-cluster, Kubernetes, and storage quotas still apply. 5. **Preflight quotas at the tenant, before anything else.** Each cluster needs, in the target region: `compute.instance.count` (worker nodes only; the managed control plane is service-owned), `compute.instance.non-gpu.vcpu` for the CPU preset, `compute.instance.gpu.` for the GPU preset (on-demand GPU quota is frequently **0**), `compute.disk.count`/`compute.disk.size.network-ssd`, `compute.gpucluster.count` when `enable_gpu_cluster`, and `compute.filesystem.count` + `compute.filesystem.size.network-ssd` when `enable_filestore`. Each private worker consumes two `vpc.allocation.count` slots (its private address and pod alias range); the managed control-plane endpoint is service-owned. A create-on-demand project additionally needs one `vpc.network.count`, one `vpc.subnet.count`, two `vpc.pool.count`, one `vpc.routetable.count`, and one `vpc.route.count`. List them all at once with `nebius --profile

quotas quota-allowance list --parent-id --all --format json` (each item carries `metadata.name`, `spec.region`, `spec.limit`). `deploy` does this automatically (`--preflight`, on by default) and refuses to apply when a capacity block or tenant limit cannot cover the in-scope clusters; `--no-preflight` attempts it anyway. Before calculating creation-only VPC requirements, preflight lists projects once and reuses an existing immutable project ID when a name already exists. An unreadable project inventory fails closed rather than assuming the project exists. Both explicit `project_id` targets and projects resolved by name require a provider read verifying exact ID, tenant, region, active state, and absence of suspension or deletion. An explicit ID may use a local Fleet role whose name differs from the provider project name. With `--no-create-projects`, a missing target aborts the entire preflight before any project's storage or workers are created; never substitute a stale or deleting project from local state. Run the read-only live identity regression against an owner-private spec with explicit tenant, region, profile, and project IDs: ```bash NPA_INTEGRATION_E2E=1 NPA_FLEET_PROJECT_VERIFY_SPEC= \ npa/.venv/bin/python -m pytest \ npa/tests/e2e/test_fleet_project_identity_live_e2e.py -q ``` This checks valid identity and rejects mismatched tenant/region selections. It creates no resources and does not replace GPU or workload validation. The allowance read is paged to completion and selects records whose `metadata.parent_id` is the requested tenant. A project/unset allowance can never shadow a finite tenant allowance; duplicate finite evidence must agree exactly. Older unscoped records are a fallback only when no authoritative tenant record exists. Required finite allowances fail closed when their limit, unit, state, or consumption evidence cannot be interpreted. Exact usage wins; otherwise the API's fractional `status.usage_percentage` is used (values above 1 mean over-limit), and `USAGE_STATE_NOT_USED` is accepted as zero consumption. Disk/filesystem sizes require `byte`; vCPU and GPU quotas accept only their explicit compatible `count`/`vcpu` and `count`/`gpu` units. A selected unlimited allowance may omit status because no arithmetic is required. The five creation-topology allowances (`vpc.network.count`, `vpc.subnet.count`, `vpc.pool.count`, `vpc.routetable.count`, and `vpc.route.count`) are optional catalog entries: when absent from a completed, readable regional tenant catalog, preflight reports them as unadvertised and does not reject the region. If any is advertised with a finite limit, it is checked strictly. All compute, disk, mk8s, allocation, GPU-cluster, and filesystem allowances required by the selected shape remain mandatory. Project-level allowances only *subdivide* the tenant allowance, so a tenant limit of 0 cannot be worked around by creating a project quota: raising a tenant allowance is a `root-g00root` operation and a tenant-scoped service account gets `PermissionDenied ... resource ID: root-g00root`. A new tenant therefore needs its GPU/filesystem quotas raised by the Nebius account team before any GPU or shared-filesystem cluster can be applied. 6. **Deploy** (asks for confirmation): `npa fleet deploy --spec fleet.yaml`. It prints the projects/clusters it will create/update and prompts before acting; pass `--yes`/`-y` for non-interactive runs. Missing projects are created via the `nebius` CLI unless `--no-create-projects`. Deploy runs per cluster and continues past a failing target (`--fail-fast` to stop); a JSON summary lists deployed vs failed clusters with kube contexts. A successful kubeconfig write also registers the fleet target under `~/.npa/clusters/` so project-scoped workflow, controller, and `provision-if-absent` commands can consume it without a second manual cluster registration step. Fleet's unique local context may differ from the provider cluster name. Exact controller cleanup compares `--cluster-name` with the saved `provider_name`, while still verifying the immutable project/cluster IDs and the saved kubeconfig against the live provider. 7. **Consume the latest recipe**: `--k8s-training-ref main` clones `nebius-solutions-library` and uses its `k8s-training` (or `--k8s-training-dir` for a local checkout). Omit both to use the repo-vendored, tested copy. NPA applies its compatibility preparation after materializing every source, including the package-only pinned-ref fallback; currently this removes `kubectl debug --quiet` from the filesystem verifier because kubectl 1.36 otherwise hides both required success evidence and the debugger-pod name. For GPU clusters it also inspects the materialized recipe's variables and `gpu_settings` wiring. If the selected managed-driver mode/preset cannot be represented, deployment fails with an actionable compatibility error rather than silently reverting to the operator driver. MIG-enabled targets also require all seven pinned Operator/lifecycle variables in the resolved recipe; this check runs before quota, project, subnet, or Terraform mutation. 8. **Status / teardown**: `npa fleet status --spec fleet.yaml`; `npa fleet destroy --spec fleet.yaml` (prompts; `--yes`/`-y` or `--force` to skip). 9. **MIG readiness is part of deploy.** A MIG-enabled cluster is not marked deployed until two consecutive exact snapshots agree. `npa fleet verify-mig --spec fleet.yaml --output json` is the read-only diagnostic; add `--wait --reconcile` to perform the single ordered GFD/device-plugin stale-resource repair. It removes only the exact obsolete `nvidia.com/gpu=mig-not-ready:NoSchedule` taint from a successful replacement worker, preserving unrelated taints. It also reconciles a stale `OnDelete` driver pod template one worker at a time, but only after failing closed on every active application pod that requests an `nvidia.com/*` resource; operators must delete those workloads explicitly. Nonzero `nvidia.com/gpu` in either capacity or allocatable, cordoned/NotReady nodes, or stale Operator/DaemonSet generations are always failures. Readiness, driver replacement, and a mandatory representative `mig-1g.24gb` CUDA vectorAdd/MIG-identity smoke share `gpu_health_timeout_minutes`; timeout or smoke cleanup failure leaves the cluster in validation-failed state instead of reporting deployment success. ## Add / remove clusters and projects For concurrent workflows from one operator host, keep each project's SkyPilot state and API endpoint separate. An isolated HOME does not isolate SkyPilot's default listening port. Supply the operator-managed endpoint through `SKYPILOT_API_SERVER_ENDPOINT`, with a stable HOME, user identity, and exact project kubeconfig on its server. NPA inspects only the selected local port; unrelated servers cannot poison a healthy selected server's check. Legacy HOME, user-ID, or kubeconfig staleness remains automatically repairable when stopping the local daemon is safe. Explicitly conflicting runtime ownership or another daemon that SkyPilot's stop API could affect blocks repair before any server is stopped. Remote API endpoints use SkyPilot's API/controller checks instead of local procfs inspection. Preserve the same endpoint and state when monitoring or cancelling the run. The workflow submitter waits for SkyPilot's API launch result while detaching the actual workload. A rejected storage or image precheck therefore surfaces as a launch error instead of an accepted request with no observable job. For durable Nebius mounts, configure the exact project's `nebius` AWS storage profile in that isolated HOME and verify storage access before submission. Restrict Nebius to its storage capability when compute must stay on the existing reserved Kubernetes fleet. The fleet is spec-driven and idempotent, so growing or shrinking it is targeted: - **Add** one or many: put the new project/cluster in the spec and deploy just those — `npa fleet deploy --spec fleet.yaml --only-projects c,d` or `--only-clusters train,infer`. Existing clusters are reconciled in place and untouched clusters are left alone (the persisted summary is merged, not overwritten). - **Remove** one or many: `npa fleet destroy --spec fleet.yaml --only-clusters train` (or `--only-projects c`). Destroy tears down each **spec-declared** cluster that has local state, reclaims any project VPC network the fleet created after the last cluster state is gone, and drops local state only after authoritative Terraform teardown succeeds. An incomplete destroy reports `destroy-incomplete`, retains its exact state, and prints a scoped retry command. It does not enumerate clusters via the API, so a cluster created out-of-band is not reclaimed. Omitting `--only-*` tears down the whole fleet. Both `deploy` and `destroy` confirm before acting (bypass with `--yes`/`-y`; `destroy` also accepts `--force`). ## Gotchas - **Per-cluster isolation**: each `(project, cluster)` gets its own Terraform install dir + local state under `~/.npa/fleet///` and an env sidecar so `destroy` can rebuild the required `TF_VAR_*`. The sidecar's `status` starts as `provisioning`. GPU clusters move through `validating-gpu-health` and are promoted to `deployed` only after the requested Ready-node/GPU topology, absence of `NebiusGPUError`, exposed Fabric state, driver components, stable boot IDs, and per-node CUDA vectorAdd all pass for the configured stabilization interval. Health failures report `deployed-validation-failed`; credential failures report `deployed-credentials-failed`. Both retain Terraform/cloud state, kubeconfig, and local evidence for diagnosis and an idempotent retry. - **Idempotent preflight is live-verified**: a repeat deploy counts an existing target as zero incremental quota only when its saved tfvars still match and the exact provider project, cluster, and node groups are all running with the requested shapes. Reserved pools must still report non-preemptible nodes and the exact `STRICT` reservation binding; stale or incomplete evidence falls back to the ordinary conservative quota preflight. The same capacity proof applies during GPU validation or after a later application failure: a partial apply must have an exact saved cluster identity (or one matching local Terraform cluster resource), plus running provider groups. Split reserved pools are matched as separate groups. Explicit project IDs take precedence over public aliases, and an RTX Helm selector update is excluded from capacity demand. The v1 API represents preemption with an Empty marker (`{}` enables it; omission disables it), which is decoded at the API boundary before strict reservation matching. Unknown state, wrong identity, missing groups, changed capacity, or non-STRICT reservation evidence still fails closed. - **Region domain**: the recipe's `provider.tf` domain is patched to `api.nebius.cloud` for non-EU regions automatically (EU uses `api.eu.nebius.cloud`). If the upstream recipe drifts (renames `provider.tf`, moves the provider block, or changes the default domain), the patch becomes a no-op and the deploy logs a loud `WARNING` rather than silently talking to the wrong endpoint. - **Latest-recipe coupling** (`--k8s-training-ref`/`--k8s-training-dir`): the rendered `terraform.tfvars` targets the recipe's *current* variable surface -- the pinned `filesystem_csi.chart_version`, the `loki`/observability toggles, and the o11y/kuberay/gatekeeper `enable_*` flags. Pulling a newer recipe whose variables changed can require updating `fleet/tfvars.py`; validate with `npa fleet plan` + a `terraform plan` before a fleet-wide apply. - **Existing GPU pools do not change their boot image in place**: after moving an affected spec to `auto`/`managed-image`, perform a controlled rolling node-group update or recreation. A newer NPA binary alone cannot repair already-booted operator-driver nodes; preserve reservation capacity and obey workload disruption policy while replacing them. - **Filesystem quota**: `enable_filestore: true` creates one shared filesystem per cluster and consumes tenant `compute.filesystem.count` + `compute.filesystem.size.network-ssd` quota. Set `enable_filestore: false` (or raise quota) if the tenant is at its filesystem limit. - **Filesystem boot safety**: the filesystem is attached `READ_WRITE` to every CPU and GPU node-group template with `filestore_mount_tag`, and cloud-init persists the same tag at `filestore_mount_path` as virtiofs with `defaults,nofail`. Nebius warns that omitting `nofail` can prevent a node from booting after the filesystem is missing. The bundled CSI chart version is `0.1.6`, matching the current official filesystem-over-CSI guide. Set `filesystem_csi_chart_repository` at runtime when the selected recipe has no public chart default; `fleet plan` reports only whether CSI is enabled and never echoes that repository endpoint. - **Auto-created VPC on destroy**: when a target project has no subnet, deploy resolves one project network/subnet before parallel cluster applies and stores ownership in `.npa-fleet-network.json`. All clusters without explicit overrides share that authoritative subnet. `destroy` reclaims exactly that pair (subnet then network) after every project cluster state is gone. A *reused* pre-existing subnet is left untouched, and created *projects* are never deleted. - **Parallelism** (`--concurrency N` / `-j N`, default 1 = sequential): applies/ destroys N clusters at once. Each cluster has isolated terraform state, so there is no lock contention; the provider plugin cache is pre-warmed once (a single `init`) to avoid concurrent-init corruption, and each cluster writes restrictive per-cluster diagnostics to `/deploy.log` or the surviving `/.logs///destroy.log`. JSON mode uses the same logs even for sequential or one-target runs so stdout remains one JSON document. Wall-clock drops from `sum` to ~`max` of the applies. Project network resolution remains sequential and single-flight, eliminating duplicate network/subnet creation in a fresh shared project; explicit per-cluster `subnet_id` values are preserved. - **Stale IAM token**: a stale ambient `NEBIUS_IAM_TOKEN` shadows the profile exec-plugin; npa strips it for `nebius`/`terraform` calls unless `NPA_REUSE_IAM_TOKEN` is set (CI injecting a short-lived token). This is also why `--profile` must be threaded through rather than relying on the ambient token: the token, not the profile, decides the principal. The current vendored recipe uses the explicit profile for refreshable Nebius authentication and profile-bound Kubernetes credential exec plugins. NPA removes the minted token's provider override for compatible recipes, so a long apply or destroy does not keep using the token minted before workers began provisioning. Older recipes retain their minted-token behavior; use the vendored recipe when long operations outlive that token. - **Default StorageClass depends on `enable_filestore`**: the recipe installs the filesystem CSI (`csi-mounted-fs-path-sc`, `ReadWriteMany`) only when a shared filesystem is attached. Without it the only class is `compute-csi-default-sc` (provisioner `compute.csi.nebius.com`, `ReadWriteOnce` disk-over-CSI); a PVC naming `csi-mounted-fs-path-sc` then sits `Pending` with `ProvisioningFailed: storageclass ... not found` and its pod never schedules. - **A quota-starved node group looks like a hang, not an error**: mk8s *accepts* a node group whose instances it cannot create, then retries forever while compute rejects each one. Terraform only prints `Still creating...` until the timeout, and the real reason is visible solely in the node group's own events: `nebius --profile

mk8s node-group get --id --format json` shows `QuotaFailure` with `quota`, `limit`, and `requested`. This is exactly why the quota preflight exists — the node group's k8s-side state is `PROVISIONING`, not `FAILED`, so nothing else surfaces the wall. - **`AUTO` is not an explicit reservation guarantee**: Nebius may fall back to ordinary on-demand capacity. Fleet's `capacity_block_group` surface therefore always renders `STRICT` and validates that exact block before apply. - **terraform >= 1.12**: the recipe's modules use `ephemeral` blocks and a `>= 1.12` version constraint. Deploy and destroy assert machine-readable `terraform version -json` output before lifecycle work; set `NPA_TERRAFORM_BIN` if the system terraform is older. ## Verify ```bash npa fleet plan --help npa fleet deploy --help npa fleet verify-mig --help npa fleet verify-storage --help npa fleet verify-graphics --help npa/.venv/bin/python -m pytest npa/tests/unit/test_fleet_cli.py -q ```