Immutable. This exact content is served forever at /api/v1/blob/545302ebf9dde198.
--- name: teardown-and-cost description: Use to stop spend safely and audit orphaned resources — the mandatory cancel-before-destroy ordering, which npa commands touch cloud versus local state, and how to find leaked clusters, agent VMs, controllers, and buckets. --- # Teardown and cost hygiene There is no `npa cost` and no `npa quota` command. Spend control here is procedural: run things in an order that cannot orphan resources, and audit for residue afterwards. The single most expensive mistake is destroying infrastructure while managed jobs are still running, which leaves jobs billing against resources nothing is tracking any more. ## The ordering is not optional Cancel workloads, then remove the things that host them, outermost last: ```bash npa workbench workflow cancel <run-id> --project <alias> --json # every run npa agent destroy --project <alias> --name <name> --yes npa skypilot cleanup-controller --project <alias> --context <context> --yes npa cluster down --project <alias> --force npa storage bucket delete --project <alias> --yes --wait npa storage service-account delete --project <alias> --dry-run npa storage service-account delete --project <alias> --yes npa configure --forget-project <alias> npa cleanup --full --yes --project <alias> ``` Cancel is repeat-safe — a run that never launched is a successful no-op — so there is no reason to skip it. `cleanup-controller` refuses while managed jobs are in progress and retries that specific refusal after the queue drains; treat the refusal as correct, not as something to force. Deleting the project itself is separately opt-in and stays retained by default: ```bash npa destroy --project <alias> --all --json # plan only npa destroy --project <alias> --all --yes # execute, project retained npa destroy --project <alias> --all --delete-project --yes --json ``` `npa destroy --all` without `--delete-project --yes` never deletes the project. Full details of the ownership gating live in `skills/tools/nebius-infra/SKILL.md`; this skill is the operational ordering and the audit. ## What is cloud spend and what is only local clutter Confusing these two wastes time and money in opposite directions. | Command | Touches cloud spend | Notes | |---|---|---| | `npa cleanup` | No | Local caches and state only, even with `--yes` | | `npa cleanup --full --yes` | No | Also removes saved HF/Token Factory/NGC creds; read-only IAM verification | | `npa workbench workflow cancel` | Yes — stops it | Repeat-safe | | `npa skypilot cleanup-controller` | Yes | Shared; drains first | | `npa cluster down` | Yes | Terraform-managed cluster and its nodes | | `npa agent destroy` | Yes | Agent VM and its resources | | `npa storage bucket delete` | Yes | Contents and versions included | | `npa destroy --all --delete-project --yes` | Yes | Ownership-gated; all three flags required | `npa cleanup` never deletes cloud resources. It *reports* what teardown left behind and prints the ordered runbook, which makes it the right first command when you inherit a machine and do not know what is still running: ```bash npa cleanup --json # report only npa cleanup --list-receipts # durable audit trail, no secrets ``` ## Auditing for orphans Residue accumulates when a teardown was interrupted or run out of order. Check each class explicitly — nothing sweeps them for you: ```bash npa workbench workflow list --project <alias> --json # non-terminal runs npa skypilot status --project <alias> --context <ctx> # shared controller npa cluster list --project <alias> # clusters, incl. stale npa agent list # agent VMs in config npa storage bucket list --project <alias> # marks the configured one npa fleet status --spec <spec.yaml> # cross-project fleets ``` Cross-project fleets are the easiest thing to lose track of, because a fleet spec can create clusters in projects that never appear in your default alias. Audit fleets from the spec that created them. `npa cleanup` queries the SkyPilot managed-job queue as part of its report. Do not reach for `--skip-jobs` to make it quiet: skipping the queue with `--attest-no-active-jobs` is an explicit attestation that you verified terminal state some other way, not a shortcut. ## Receipts: how to finish a teardown you can no longer address by alias Teardown receipts live under `~/.npa/teardown-receipts/`, contain no secrets, and survive removal of the project config. If you already ran `configure --forget-project` and later discover residue, the receipt is the recovery identity: ```bash npa cleanup --list-receipts npa destroy --receipt <id> --all --delete-project --yes --json ``` Prune only aged terminal receipts, and only explicitly: ```bash npa cleanup --prune-receipts --receipt-retention-days 90 --yes ``` ## Choosing a cheap tier in the first place Most verification does not need a GPU, and the tiers are ordered by cost: 1. **Dry run / plan.** `npa workbench workflow plan-spec`, `submit --plan-only`, `golden-eval run <name>` (dry-run is the default), `provision-if-absent --dry-run`. No cloud resources. 2. **Zero-GPU hosted inference.** Token Factory (`skills/tools/token-factory`) answers real captioning/generation/reasoning questions with no cluster at all. 3. **Serverless single job.** `golden-eval run <name> --serverless` runs one container on one GPU and returns PASS/FAIL, without standing up a cluster. 4. **Cluster.** Only when the work genuinely needs a persistent multi-stage graph. Preemptible GPU nodes change the capacity pool but **not** hard instance, disk, or IP quotas, and a reclaim stops the node mid-run. They lower price, not requirements. ## Gotchas - **A PENDING job is still billing the cluster it is stuck on.** `ImagePullBackOff` and `Unschedulable` are retried forever and never self-terminate. Cancel them; see `skills/atomic/debug-failed-run/SKILL.md`. - **`npa cluster down` going quiet for minutes is expected.** It previews the PodDisruptionBudgets that hold up the node drain before it destroys anything. - **The controller is shared.** Never tear it down because *your* run finished; it may be hosting someone else's managed jobs. That is why `cleanup-controller` requires the exact project and saved context rather than an ambient kube-context. - **Full cleanup exiting 2 is a real result, not a flake.** It means storage IAM is present or unverified: cleanup was partial and something still exists. - **Never run a live GPU tier on a schedule.** Scheduled GPU runs are the classic way to discover a five-figure bill; keep GPU rotations manually dispatched. ## Verify ```bash npa/.venv/bin/python -m pytest npa/tests/guardrails/test_skills_index.py -q ```