ubuntu-lxd-gpu-server git:20260610.1c5c032A

Immutable. This exact content is served forever at /api/v1/blob/e2c71b60becc5dc5.

---
name: ubuntu-lxd-gpu-server
description: Install LXD on an Ubuntu server and pass all NVIDIA GPUs into LXD system containers via CDI — install snapd+LXD (snap), run `lxd init` with a ZFS or dir storage pool, generate a host CDI spec with nvidia-ctk, and grant every GPU to every instance through the default profile, then verify nvidia-smi inside a container. Use when asked to install or set up LXD/lxc on a GPU host, give LXD containers GPU access, do LXD NVIDIA GPU passthrough, share all GPUs across LXD instances, or when `nvidia.runtime=true` fails with "driver rpc error: timed out" (use CDI instead). Assumes the host NVIDIA driver + nvidia-container-toolkit are already installed (see ubuntu-nvidia-gpu-enablement).
---

# Ubuntu LXD GPU Server

Install LXD on an Ubuntu host and expose **all** NVIDIA GPUs to LXD system containers via **CDI**, granted
through the `default` profile so every instance inherits them. Assumes the host driver + `nvidia-container-toolkit`
(`nvidia-ctk`) are already in place — if not, run the **`ubuntu-nvidia-gpu-enablement`** skill first.

⚠️ **Use CDI, not `nvidia.runtime=true`.** LXD's legacy libnvidia-container hook hangs at container start with
`nvidia-container-cli: initialization error: driver rpc error: timed out` on recent kernels / Blackwell GPUs.
CDI uses the host's `nvidia-ctk` and a static spec — no driver RPC, no timeout. (Why: [REFERENCE.md](REFERENCE.md) §4.)

## Quick start

```bash
# 1. install LXD + wire all GPUs into the default profile. Storage: zfs:<pool>/lxd | dir | zfs-loop:50GiB
sudo LXD_STORAGE=zfs:rpool/lxd bash scripts/install-lxd.sh
# 2. verify a fresh container sees every GPU (launches a throwaway container, asserts the count, cleans up)
bash scripts/verify-gpu.sh
```

## Pre-flight

- Sudo user (SSH fine). `nvidia-smi -L` lists the GPUs on the **host**.
- `nvidia-ctk --version` works (host CDI toolkit). Missing → `ubuntu-nvidia-gpu-enablement` Step 5.
- Egress to snap + the image server (`images.lxd.canonical.com`).
- Storage decision: a ZFS pool (redundant root mirror, or a data pool) is ideal; otherwise `dir` works anywhere.
  Redundant pool → containers survive a disk loss; big stripe → more space. See REFERENCE §2.

## Steps (what `install-lxd.sh` does)

1. **snapd + LXD.** Minimal/debootstrap bases ship no snapd: `apt-get install -y snapd && snap wait system seed.loaded`,
   then `snap install lxd`. ⚠️ sudo's `secure_path` lacks `/snap/bin` **and** the `lxd` group needs a re-login —
   so this session, call `lxc`/`lxd` by absolute path (`sudo /snap/bin/lxc …`).
2. **`lxd init`** (preseed): one storage pool + `lxdbr0` NAT bridge. ZFS source `<pool>/lxd` puts rootfs on your
   chosen pool; `dir` is filesystem-agnostic. Full preseed + backends in REFERENCE §2.
3. **Generate the CDI spec** with the *host* toolkit (declares each GPU + an `all` device):
   ```bash
   sudo mkdir -p /etc/cdi && sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml
   ```
4. **Grant all GPUs to all instances** via the default profile:
   ```bash
   sudo /snap/bin/lxc profile device add default gpu0 gpu gputype=physical id=nvidia.com/gpu=all
   ```
   Per-instance instead: `lxc config device add <inst> gpu0 gpu gputype=physical id=nvidia.com/gpu=all`.
   A single GPU: `id=nvidia.com/gpu=0` or `id=nvidia.com/gpu=<UUID>` (REFERENCE §3).

## Verify

```bash
sudo /snap/bin/lxc launch ubuntu:24.04 g1
sudo /snap/bin/lxc exec g1 -- nvidia-smi -L      # must list every host GPU
sudo /snap/bin/lxc exec g1 -- nvidia-smi         # full table; libcuda is injected too (CUDA works)
sudo /snap/bin/lxc delete -f g1
```
`scripts/verify-gpu.sh` automates this and fails loudly if the container's GPU count ≠ the host's.

## Maintenance

⚠️ The CDI spec hardcodes the running driver's library paths. **Regenerate after every host driver upgrade**
(`sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml`) or GPU containers break. REFERENCE §5 has a
boot-time systemd unit that does this automatically.

Deep dives — storage backends & preseed, GPU selection, CDI-vs-runtime diagnosis, moving the pool between ZFS
pools, troubleshooting, uninstall — in [REFERENCE.md](REFERENCE.md).