proxmox · diff

git:20260313.e5b700b to git:20260313.02a8e7c

188 added, 236 removed. Audit A to A.

---

name: proxmox

description: >-

Proxmox VE administration: VM and LXC provisioning, storage backends, networking,

clustering, high availability, API automation, cloud-init templates, backups, and

PCIe passthrough. Invoke whenever task involves any interaction with Proxmox VE —

configuring hosts, managing guests, designing storage or networking, writing

automation scripts, planning clusters, troubleshooting, or reviewing PVE

configurations.

---

# Proxmox VE

- **Production infrastructure demands production discipline. Every Proxmox

- configuration must be secure by default, redundant where it matters, and

- automated where possible.**

+ **Production infrastructure demands production discipline. Every Proxmox configuration must be secure by default,

+ redundant where it matters, and automated where possible.**

---

## Route to Reference

- | Topic | Reference | Contents |

- |-------|-----------|----------|

- | VM and LXC management | [`${CLAUDE_SKILL_DIR}/references/vm-and-lxc.md`] | VM vs LXC comparison table, configuration options, template workflows, linked vs full clone trade-offs |

- | Storage backends | [`${CLAUDE_SKILL_DIR}/references/storage-backends.md`] | Backend capability matrix, ZFS tuning (ARC/L2ARC/SLOG/volblocksize), Ceph configuration, LVM-Thin monitoring, storage selection decision tree |

- | Networking | [`${CLAUDE_SKILL_DIR}/references/networking.md`] | Bridge configuration, VLAN layout, bonding modes, SDN zones (VXLAN/EVPN), MTU considerations, OVS vs Linux bridge, firewall lockout prevention |

- | Clustering and HA | [`${CLAUDE_SKILL_DIR}/references/clustering-and-ha.md`] | Corosync configuration, quorum math, QDevice setup, split-brain prevention, fencing methods, HA groups, migration, quorum loss recovery |

- | API and automation | [`${CLAUDE_SKILL_DIR}/references/api-automation.md`] | REST API architecture, pvesh/qm/pct reference, Terraform patterns, cloud-init customization (cicustom, network v2), hookscript lifecycle, CI/CD pipelines |

- | Backup strategies | [`${CLAUDE_SKILL_DIR}/references/backup-strategies.md`] | vzdump modes, PBS architecture, encryption key management, garbage collection safety, verification jobs, retention policies, off-site sync patterns |

+ | Topic | Reference | Contents |

+ | --------------------- | ------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- |

+ | VM and LXC management | [`${CLAUDE_SKILL_DIR}/references/vm-and-lxc.md`] | VM vs LXC comparison table, configuration options, template workflows, linked vs full clone trade-offs |

+ | Storage backends | [`${CLAUDE_SKILL_DIR}/references/storage-backends.md`] | Backend capability matrix, ZFS tuning (ARC/L2ARC/SLOG/volblocksize), Ceph configuration, LVM-Thin monitoring, storage selection decision tree |

+ | Networking | [`${CLAUDE_SKILL_DIR}/references/networking.md`] | Bridge configuration, VLAN layout, bonding modes, SDN zones (VXLAN/EVPN), MTU considerations, OVS vs Linux bridge, firewall lockout prevention |

+ | Clustering and HA | [`${CLAUDE_SKILL_DIR}/references/clustering-and-ha.md`] | Corosync configuration, quorum math, QDevice setup, split-brain prevention, fencing methods, HA groups, migration, quorum loss recovery |

+ | API and automation | [`${CLAUDE_SKILL_DIR}/references/api-automation.md`] | REST API architecture, pvesh/qm/pct reference, Terraform patterns, cloud-init customization (cicustom, network v2), hookscript lifecycle, CI/CD pipelines |

+ | Backup strategies | [`${CLAUDE_SKILL_DIR}/references/backup-strategies.md`] | vzdump modes, PBS architecture, encryption key management, garbage collection safety, verification jobs, retention policies, off-site sync patterns |

---

## Guest Management

<guest-management>

### VM vs LXC Decision

- **Default to LXC** for trusted Linux workloads — near-zero overhead, high density

- - **Use VMs** when the workload requires: non-Linux OS, full kernel isolation, PCIe

- passthrough, or live migration

- - **Use unprivileged containers** (default) — they map container UID 0 to a

- non-privileged host UID, preventing container escape attacks

- - Only use privileged containers when unprivileged mode is incompatible (specific

- device access, certain NFS mounts)

+ - **Use VMs** when the workload requires: non-Linux OS, full kernel isolation, PCIe passthrough, or live migration

+ - **Use unprivileged containers** (default) — they map container UID 0 to a non-privileged host UID, preventing

+ container escape attacks

+ - Only use privileged containers when unprivileged mode is incompatible (specific device access, certain NFS mounts)

### VM Configuration

- Use **VirtIO** drivers for disk and network — mandatory for production performance

- - Enable **QEMU Guest Agent** inside every VM for graceful shutdown, snapshot

- consistency, and IP address reporting

+ - Enable **QEMU Guest Agent** inside every VM for graceful shutdown, snapshot consistency, and IP address reporting

- Use **OVMF (UEFI) + Q35 machine type** for PCIe passthrough and Secure Boot

- - Set CPU type to `host` for maximum performance in homogeneous clusters; use

- `x86-64-v2-AES` or the lowest common CPU generation when live migration across

- different CPU generations is required

+ - Set CPU type to `host` for maximum performance in homogeneous clusters; use `x86-64-v2-AES` or the lowest common CPU

+ generation when live migration across different CPU generations is required

- Enable **memory ballooning** for dynamic RAM management

- Enable **NUMA** topology for VMs with many cores on multi-socket hosts

### Container Configuration

- Set **explicit memory limits** — containers without limits can exhaust host RAM

- Enable **nesting** (`features: nesting=1`) only when required (Docker inside LXC)

- For Docker workloads in LXC: unprivileged + `nesting=1` + `keyctl=1`

- - Use **bind mounts** to share host directories, not NFS/CIFS mounts inside the

- container

- - PVE 9.0 removed cgroup v1 entirely — containers requiring cgroup v1 must move

- to VMs

+ - Use **bind mounts** to share host directories, not NFS/CIFS mounts inside the container

+ - PVE 9.0 removed cgroup v1 entirely — containers requiring cgroup v1 must move to VMs

### Templates and Cloning

- Build templates with the guest agent and cloud-init pre-installed

- Convert to template: `qm template <vmid>` or backup as `.tar.gz` for containers

- Use **linked clones** for development/ephemeral workloads (fast, shared base disk)

- Use **full clones** for production (independent, no template dependency)

</guest-management>

---

## Storage

### Backend Selection

- | Need | Backend |

- |------|---------|

- | Local redundancy, data integrity, snapshots | **ZFS** |

- | Local snapshots/clones without ZFS overhead | **LVM-Thin** |

- | Shared storage for HA clusters (3+ nodes) | **Ceph** |

- | Simple shared storage (existing NAS/SAN) | **NFS** or **iSCSI** |

- | Deduplicated backups | **PBS** |

+ | Need | Backend |

+ | ------------------------------------------- | -------------------- |

+ | Local redundancy, data integrity, snapshots | **ZFS** |

+ | Local snapshots/clones without ZFS overhead | **LVM-Thin** |

+ | Shared storage for HA clusters (3+ nodes) | **Ceph** |

+ | Simple shared storage (existing NAS/SAN) | **NFS** or **iSCSI** |

+ | Deduplicated backups | **PBS** |

### ZFS Rules

- Use HBA in IT mode — never hardware RAID controllers with write-back cache

- - **ARC sizing:** cap explicitly in `/etc/modprobe.d/zfs.conf`; 25-30% of host

- RAM for mixed workloads. PVE 8.1+ defaults to 10% (max 16GB) for new installs.

- Rule of thumb: 2GB base + 1GB per TB of storage

- - Set `ashift=12` for 4K sector disks (most modern drives); incorrect ashift

- halves IOPS due to sector misalignment

- - **Volblocksize** (VMs on zvols): 16k for mirrors/RAID10; increase for wide

- RAIDZ to reduce write amplification. Can only be set at zvol creation

- - **Recordsize** (containers on datasets): tune per workload — 8k for Postgres,

- 16k for MariaDB, 1M for large sequential files

- - Enable `compression=lz4` — can actually increase I/O performance by writing

- less data to disk

- - **SLOG:** 8-32GB enterprise NVMe with power-loss protection; only holds ~5s

- of write data. Benefits sync-heavy workloads (databases, NFS)

- - **L2ARC:** only if ARC hit ratio is low and adding RAM is not possible; size

- 5-20x RAM; budget 1GB RAM per 50GB of L2ARC for metadata

+ - **ARC sizing:** cap explicitly in `/etc/modprobe.d/zfs.conf`; 25-30% of host RAM for mixed workloads. PVE 8.1+

+ defaults to 10% (max 16GB) for new installs. Rule of thumb: 2GB base + 1GB per TB of storage

+ - Set `ashift=12` for 4K sector disks (most modern drives); incorrect ashift halves IOPS due to sector misalignment

+ - **Volblocksize** (VMs on zvols): 16k for mirrors/RAID10; increase for wide RAIDZ to reduce write amplification. Can

+ only be set at zvol creation

+ - **Recordsize** (containers on datasets): tune per workload — 8k for Postgres, 16k for MariaDB, 1M for large sequential

+ files

+ - Enable `compression=lz4` — can actually increase I/O performance by writing less data to disk

+ - **SLOG:** 8-32GB enterprise NVMe with power-loss protection; only holds ~5s of write data. Benefits sync-heavy

+ workloads (databases, NFS)

+ - **L2ARC:** only if ARC hit ratio is low and adding RAM is not possible; size 5-20x RAM; budget 1GB RAM per 50GB of

+ L2ARC for metadata

- Schedule regular scrubs (weekly or monthly)

- Plan capacity upfront — ZFS pools cannot be shrunk

- Never create swap on a ZFS zvol — causes blocking I/O during backups

### Ceph Rules

- Minimum 3 nodes for proper quorum and data distribution

- - Dedicated 10GbE+ network for Ceph traffic (25GbE+ recommended for NVMe OSDs),

- separate from Corosync — Ceph rebalance traffic saturates links and causes

- Corosync instability

+ - Dedicated 10GbE+ network for Ceph traffic (25GbE+ recommended for NVMe OSDs), separate from Corosync — Ceph rebalance

+ traffic saturates links and causes Corosync instability

- Use SSD/NVMe for OSD WAL/DB

- Dedicated disks for OSDs — never share with the host OS

- PVE 9.0 defaults to Ceph Squid (v19.2)

### Storage Anti-Patterns

- - Running ZFS or Ceph on top of hardware RAID with write-back cache — defeats

- data integrity guarantees

- - Overprovisioning LVM-Thin without monitoring — a full thin pool causes I/O

- errors for all guests on that pool

+ - Running ZFS or Ceph on top of hardware RAID with write-back cache — defeats data integrity guarantees

+ - Overprovisioning LVM-Thin without monitoring — a full thin pool causes I/O errors for all guests on that pool

- Storing backups on the same physical disks as production data

- - Letting ZFS ARC consume unbounded RAM — causes VM crashes when the hypervisor

- and VMs compete for memory

+ - Letting ZFS ARC consume unbounded RAM — causes VM crashes when the hypervisor and VMs compete for memory

</storage>

---

## Networking

### Core Model

- Every guest connects to a **Linux bridge**. Use **VLAN-aware bridges** (single

- bridge with 802.1Q tagging) instead of per-VLAN bridges. The VLAN-aware checkbox

- must be explicitly enabled — it is off by default.

+ Every guest connects to a **Linux bridge**. Use **VLAN-aware bridges** (single bridge with 802.1Q tagging) instead of

+ per-VLAN bridges. The VLAN-aware checkbox must be explicitly enabled — it is off by default.

### VLAN Best Practices

- - Place the **management interface on a dedicated VLAN** — never share with

- guest traffic

- - Configure **trunk ports** on physical switches for the Proxmox host — frames

- with VLAN IDs not allowed by the switch are silently dropped

+ - Place the **management interface on a dedicated VLAN** — never share with guest traffic

+ - Configure **trunk ports** on physical switches for the Proxmox host — frames with VLAN IDs not allowed by the switch

+ are silently dropped

- Assign VLAN tags per guest in the network configuration

- Verify no double-tagging mismatch between Proxmox and switch native VLAN

### Traffic Separation

- | Traffic | Minimum bandwidth |

- |---------|------------------|

- | Management / Corosync | 1GbE (dedicated) |

+ | Traffic | Minimum bandwidth |

+ | --------------------- | ------------------------------------ |

+ | Management / Corosync | 1GbE (dedicated) |

| Ceph cluster + public | 10GbE (dedicated), 25GbE recommended |

- | Migration | 10GbE (recommended) |

- | Guest | Depends on workload |

+ | Migration | 10GbE (recommended) |

+ | Guest | Depends on workload |

- **Critical rule:** Never combine Corosync traffic with high-bandwidth Ceph or

- migration traffic on a single 1GbE link. Corosync is latency-sensitive — network

- contention causes cluster instability and false fencing.

+ **Critical rule:** Never combine Corosync traffic with high-bandwidth Ceph or migration traffic on a single 1GbE link.

+ Corosync is latency-sensitive — network contention causes cluster instability and false fencing.

### SDN

- Use SDN (VXLAN zones) for overlay networking across nodes without physical switch

- changes. Use EVPN for advanced multi-tenant setups with BGP routing.

+ Use SDN (VXLAN zones) for overlay networking across nodes without physical switch changes. Use EVPN for advanced

+ multi-tenant setups with BGP routing.

**SDN gotchas:**

- VXLAN adds 50-byte header — set VNet MTU to 1450 (or 1370 with IPSEC)

- SDN changes are **staged**, not live — click Apply at Datacenter level

- DHCP requires a gateway configured on the subnet

- Multiple EVPN exit nodes require disabling `rp_filter` in sysctl

- - **OVS vs Linux bridge:** OVS is automatically VLAN-aware and may resolve

- 10GbE throughput bottlenecks seen with native Linux bridge

+ - **OVS vs Linux bridge:** OVS is automatically VLAN-aware and may resolve 10GbE throughput bottlenecks seen with native

+ Linux bridge

</networking>

---

## Clustering and High Availability

### Cluster Rules

- Use a **dedicated network** for Corosync — latency under 5ms required

- - Configure **redundant Corosync links** (up to 8 supported via Kronosnet)

- on separate physical networks

- - For 2-node clusters, deploy a **QDevice** on a third machine for quorum —

- a 2-node cluster without QDevice is a split-brain generator

- - QDevice is discouraged for odd-numbered clusters — it becomes a single

- point of failure due to (N-1) vote allocation

- - If using LACP bonds for Corosync, set `bond-lacp-rate fast` on both node

- and switch — default slow rate has 90s failover, causing fencing after ~60s

- - **Avoid** `balance-rr`, `balance-xor`, `balance-tlb`, `balance-alb` bond

- modes for Corosync — they cause asymmetric connectivity and mass fencing

+ - Configure **redundant Corosync links** (up to 8 supported via Kronosnet) on separate physical networks

+ - For 2-node clusters, deploy a **QDevice** on a third machine for quorum — a 2-node cluster without QDevice is a

+ split-brain generator

+ - QDevice is discouraged for odd-numbered clusters — it becomes a single point of failure due to (N-1) vote allocation

+ - If using LACP bonds for Corosync, set `bond-lacp-rate fast` on both node and switch — default slow rate has 90s

+ failover, causing fencing after ~60s

+ - **Avoid** `balance-rr`, `balance-xor`, `balance-tlb`, `balance-alb` bond modes for Corosync — they cause asymmetric

+ connectivity and mass fencing

- Never join a node with existing VMs/containers to a cluster — start fresh

- - Update nodes **one at a time** — the LRM requests a service freeze from the

- CRM during updates; if both are updating, the watchdog fences the node

+ - Update nodes **one at a time** — the LRM requests a service freeze from the CRM during updates; if both are updating,

+ the watchdog fences the node

### HA Requirements

- **Shared or replicated storage** accessible from all HA nodes

- **Working fencing mechanism** — test before relying on HA

- **Minimum 3 quorum votes** (3 nodes, or 2 nodes + QDevice)

- Configure **HA groups** with node priorities for controlled failover

### Fencing

- Fencing guarantees a failed node is offline before its services restart elsewhere.

- **HA without fencing is a data corruption risk** — two nodes writing to shared

- storage simultaneously causes irrecoverable damage.

+ Fencing guarantees a failed node is offline before its services restart elsewhere. **HA without fencing is a data

+ corruption risk** — two nodes writing to shared storage simultaneously causes irrecoverable damage.

- Verify watchdog status: `ha-manager status`

- Test fencing by simulating node failure before going to production

- - Use hardware watchdog (`iTCO_wdt` via `/etc/default/pve-ha-manager`) when

- available, software watchdog (`softdog`) as fallback

+ - Use hardware watchdog (`iTCO_wdt` via `/etc/default/pve-ha-manager`) when available, software watchdog (`softdog`) as

+ fallback

- Configure `WATCHDOG_MODULE=iTCO_wdt` in `/etc/default/pve-ha-manager`

### Quorum Loss Recovery

- If the cluster loses quorum, `pmxcfs` becomes read-only — no VM operations

- are possible. Emergency recovery:

+ If the cluster loses quorum, `pmxcfs` becomes read-only — no VM operations are possible. Emergency recovery:

- - `pvecm expected 1` forces single-node quorum — use only to restore vital

- guests or fix the quorum issue itself

- - **Never** make cluster changes (add/remove nodes, storage, guests) while

- expected votes are overridden

+ - `pvecm expected 1` forces single-node quorum — use only to restore vital guests or fix the quorum issue itself

+ - **Never** make cluster changes (add/remove nodes, storage, guests) while expected votes are overridden

### Migration

- - **Live migration** (VMs only): requires shared/replicated storage, no PCIe

- passthrough devices, brief pause at cutover

- - **Offline migration** (VMs and containers): guest stops, data transfers, guest

- starts on target

+ - **Live migration** (VMs only): requires shared/replicated storage, no PCIe passthrough devices, brief pause at cutover

+ - **Offline migration** (VMs and containers): guest stops, data transfers, guest starts on target

- Use a dedicated high-bandwidth network for migration traffic

</clustering>

---

## API and Automation

### Authentication

- - Use **API tokens with privilege separation** for all automation — never root

- credentials

+ - Use **API tokens with privilege separation** for all automation — never root credentials

- Store token secrets in vault or environment variables — never in code

- Use ticket authentication only for interactive or short-lived tools

### CLI Tools

- | Tool | Purpose |

- |------|---------|

- | `pvesh` | Direct REST API access from CLI |

- | `qm` | VM lifecycle management |

- | `pct` | Container lifecycle management |

- | `ha-manager` | HA resource management |

- | `pvecm` | Cluster management |

- | `pvesm` | Storage management |

+ | Tool | Purpose |

+ | ------------ | ------------------------------- |

+ | `pvesh` | Direct REST API access from CLI |

+ | `qm` | VM lifecycle management |

+ | `pct` | Container lifecycle management |

+ | `ha-manager` | HA resource management |

+ | `pvecm` | Cluster management |

+ | `pvesm` | Storage management |

### Terraform

- Use **API tokens** with privilege separation — never root credentials

- - Use cloud-init templates as the base for Terraform-managed VMs — templates

- must have `qemu-guest-agent` installed or Terraform hangs on "still creating"

- - Store Terraform state remotely (GitLab HTTP backend, S3, Consul) — never

- local state for shared infrastructure

- - Use `lifecycle { ignore_changes }` for fields Proxmox modifies outside

- Terraform (e.g., disk size after manual resize)

+ - Use cloud-init templates as the base for Terraform-managed VMs — templates must have `qemu-guest-agent` installed or

+ Terraform hangs on "still creating"

+ - Store Terraform state remotely (GitLab HTTP backend, S3, Consul) — never local state for shared infrastructure

+ - Use `lifecycle { ignore_changes }` for fields Proxmox modifies outside Terraform (e.g., disk size after manual resize)

- CI/CD pipeline: validate -> plan (save artifact) -> apply (manual trigger)

- Never commit API secrets to git — use CI/CD variables (`TF_VAR_*`)

### Cloud-Init

- Prepare a base VM with `qemu-guest-agent` and `cloud-init` installed

- Add a Cloud-Init drive (IDE or SCSI CD-ROM)

- Configure network, SSH keys, and user data via the Cloud-Init panel or API

- Convert to template, deploy via linked clone

- Use **SSH key authentication** — cloud-init password storage is less secure

- - **cicustom** for advanced needs: reference custom YAML snippets for user,

- network, and meta data from a snippets-capable storage

+ - **cicustom** for advanced needs: reference custom YAML snippets for user, network, and meta data from a

+ snippets-capable storage

- Store cicustom snippets on shared storage (CephFS) in clusters for HA

- Windows templates: use Cloudbase-Init with `configdrive2` format + Sysprep

</automation>

---

## Backups

### Backup Rules

- - **Use PBS for production** — deduplication, incremental backups, verification,

- and encryption. Directory/NFS backups are acceptable only for lab environments.

+ - **Use PBS for production** — deduplication, incremental backups, verification, and encryption. Directory/NFS backups

+ are acceptable only for lab environments.

- Use **snapshot mode** for VMs (crash-consistent, no downtime)

- Use **zstd compression** — best balance of ratio and speed

- Run backups during **low-usage windows**, stagger across storage targets

- Follow the **3-2-1 rule**: 3 copies, 2 media types, 1 off-site

### Retention

Configure retention per backup job:

```

keep-daily=7,keep-weekly=4,keep-monthly=6,keep-yearly=1

```

- Retention options process in order — each option covers only its time period.

- Use higher retention than minimally required; you cannot recreate pruned backups.

+ Retention options process in order — each option covers only its time period. Use higher retention than minimally

+ required; you cannot recreate pruned backups.

### PBS Security

- Restrict PVE backup user/token to create-only access (no delete) on PBS

- Use **dedicated API tokens per host/cluster** scoped to specific namespaces

- Separate PBS admin credentials from PVE access

- - Enable backup verification jobs — re-verify all backups monthly (bit rot

- detection). Encrypted chunks can only verify CRC-32, not plaintext content

+ - Enable backup verification jobs — re-verify all backups monthly (bit rot detection). Encrypted chunks can only verify

+ CRC-32, not plaintext content

- Encrypt off-site backups with client-side encryption

- - **Store encryption keys separately** from the backed-up system — losing the

- key means losing access. Use an RSA master key for recovery. Store key copies

- in a password manager, USB drive, and QR-encoded paper backup

+ - **Store encryption keys separately** from the backed-up system — losing the key means losing access. Use an RSA master

+ key for recovery. Store key copies in a password manager, USB drive, and QR-encoded paper backup

- Never disable `gc-atime-safety-check` — risks deleting in-use chunks

- - The `protected` flag does not sync between PBS instances — re-protect

- snapshots manually on remote targets after sync

+ - The `protected` flag does not sync between PBS instances — re-protect snapshots manually on remote targets after sync

### Non-Negotiable

- **Test restores regularly.** A backup that cannot be restored is worthless.

- - **Monitor backup jobs.** Configure notification matchers for `vzdump` errors.

- A silently failing backup is worse than no backup.

- - **Document the restore procedure.** Where, what credentials, exact commands,

- expected recovery time.

+ - **Monitor backup jobs.** Configure notification matchers for `vzdump` errors. A silently failing backup is worse than

+ no backup.

+ - **Document the restore procedure.** Where, what credentials, exact commands, expected recovery time.

### Off-site Sync

- - Use **dedicated remote users** per sync job — shared users with

- `remove-vanished` enabled delete each other's snapshots

+ - Use **dedicated remote users** per sync job — shared users with `remove-vanished` enabled delete each other's

+ snapshots

- Verify removable datastores are mounted at scheduled sync times

</backups>

---

## PCIe Passthrough

### Requirements

- CPU: VT-d (Intel) or AMD-Vi enabled in BIOS/UEFI

- IOMMU enabled in kernel: `intel_iommu=on` or `amd_iommu=on` in boot parameters

- Interrupt remapping supported (verify with `dmesg | grep remapping`)

- Dedicated IOMMU group for the passthrough device

- VFIO modules loaded: `vfio`, `vfio_iommu_type1`, `vfio_pci`, `vfio_virqfd`

### Configuration

- - Use **OVMF (UEFI) firmware + Q35 machine type** — provides virtual PCIe bus;

- if the GPU lacks a UEFI-capable ROM, use SeaBIOS instead

- - Blacklist the host driver for the passthrough device (e.g., `nouveau`, `nvidia`,

- `radeon`) or bind via `vfio-pci` IDs in `/etc/modprobe.d/`

+ - Use **OVMF (UEFI) firmware + Q35 machine type** — provides virtual PCIe bus; if the GPU lacks a UEFI-capable ROM, use

+ SeaBIOS instead

+ - Blacklist the host driver for the passthrough device (e.g., `nouveau`, `nvidia`, `radeon`) or bind via `vfio-pci` IDs

+ in `/etc/modprobe.d/`

- Add device via `hostpci0: 0000:01:00,pcie=1` in VM configuration

- - Pass through **all device functions** — a GPU requires both video and audio

- functions; partial passthrough fails. GPUs with USB-C controllers need that

- controller bound to `vfio-pci` too, or the host freezes

+ - Pass through **all device functions** — a GPU requires both video and audio functions; partial passthrough fails. GPUs

+ with USB-C controllers need that controller bound to `vfio-pci` too, or the host freezes

- For GPU passthrough: add `x-vga=1` for primary GPU, configure `vga: none`

- - GPU output is **not visible** via NoVNC/SPICE — use a physical monitor,

- HDMI/DP dummy plug, or **Looking Glass** (shared-memory low-latency display)

- - VMs with passthrough devices **cannot be live-migrated** — use cluster-wide

- resource mappings (`/cluster/mapping/pci`) for HA with passthrough

+ - GPU output is **not visible** via NoVNC/SPICE — use a physical monitor, HDMI/DP dummy plug, or **Looking Glass**

+ (shared-memory low-latency display)

+ - VMs with passthrough devices **cannot be live-migrated** — use cluster-wide resource mappings (`/cluster/mapping/pci`)

+ for HA with passthrough

### GPU-Specific Issues

- - **NVIDIA Error 43 (Windows):** set CPU type to `host`, add

- `options kvm ignore_msrs=1` in `/etc/modprobe.d/kvm.conf`

- - **AMD reset bug** (Vega, Polaris, some Navi): GPU fails to reset after VM

- shutdown, preventing reuse without host reboot. Fix: install `vendor-reset`

- kernel module for vendor-specific reset quirks. RDNA2+ generally unaffected

- - **NVIDIA vGPU:** officially supported since vGPU Software 18 on PVE. Requires

- valid NVIDIA entitlement. Ampere+ GPUs need SR-IOV enabled first via

- `pve-nvidia-vgpu-helper`

+ - **NVIDIA Error 43 (Windows):** set CPU type to `host`, add `options kvm ignore_msrs=1` in `/etc/modprobe.d/kvm.conf`

+ - **AMD reset bug** (Vega, Polaris, some Navi): GPU fails to reset after VM shutdown, preventing reuse without host

+ reboot. Fix: install `vendor-reset` kernel module for vendor-specific reset quirks. RDNA2+ generally unaffected

+ - **NVIDIA vGPU:** officially supported since vGPU Software 18 on PVE. Requires valid NVIDIA entitlement. Ampere+ GPUs

+ need SR-IOV enabled first via `pve-nvidia-vgpu-helper`

### LXC Device Passthrough

- Since PVE 8.2, device passthrough for containers is configurable via the UI.

- Limited compared to VM passthrough — no full PCIe passthrough, but supports

- specific device access (GPU rendering, USB devices).

+ Since PVE 8.2, device passthrough for containers is configurable via the UI. Limited compared to VM passthrough — no

+ full PCIe passthrough, but supports specific device access (GPU rendering, USB devices).

### Troubleshooting

- `dmesg | grep -e DMAR -e IOMMU` — verify IOMMU is enabled

- `pvesh get /nodes/{node}/hardware/pci --pci-class-blacklist ""` — list groups

- - If IOMMU group contains multiple devices (common on B550/X570), use

- `pcie_acs_override=downstream,multifunction` as a workaround — not

- recommended for production but necessary on many consumer motherboards

+ - If IOMMU group contains multiple devices (common on B550/X570), use `pcie_acs_override=downstream,multifunction` as a

+ workaround — not recommended for production but necessary on many consumer motherboards

- IOMMU groups can change between kernel major versions — verify after updates

</passthrough>

---

## Security

### Certificates

- - PVE generates self-signed certificates by default — replace with ACME

- (Let's Encrypt) certificates for trusted HTTPS

- - Use **DNS-01 challenge** for nodes behind firewalls or for wildcard certs

- (via acme.sh-compatible DNS API plugins)

+ - PVE generates self-signed certificates by default — replace with ACME (Let's Encrypt) certificates for trusted HTTPS

+ - Use **DNS-01 challenge** for nodes behind firewalls or for wildcard certs (via acme.sh-compatible DNS API plugins)

- Use **HTTP-01 challenge** for internet-reachable nodes on port 80

- Trusted certificates are required for reliable WebAuthn (FIDO2) operations

### Firewall

- Proxmox VE includes a distributed firewall (iptables-based, nftables available

- since PVE 8.2 as tech preview) configurable at datacenter, host, and guest levels.

+ Proxmox VE includes a distributed firewall (iptables-based, nftables available since PVE 8.2 as tech preview)

+ configurable at datacenter, host, and guest levels.

- - Enable the firewall **selectively at each level** — datacenter first, then host,

- then per-VM/container interface. It is disabled by default at all levels.

- - Before enabling, create rules to allow management access from remote IPs (ports

- 8006, 22, 3128) — enabling without rules locks you out. **Always keep an SSH

- session open** before applying firewall changes

- - Create an **IPSet named `management`** with trusted admin IPs — Proxmox

- auto-generates required management access rules from this set

- - Use **security groups** for reusable rule sets (e.g., "webserver" group opening

- ports 80/443), applied to multiple VMs for consistency

+ - Enable the firewall **selectively at each level** — datacenter first, then host, then per-VM/container interface. It

+ is disabled by default at all levels.

+ - Before enabling, create rules to allow management access from remote IPs (ports 8006, 22, 3128) — enabling without

+ rules locks you out. **Always keep an SSH session open** before applying firewall changes

+ - Create an **IPSet named `management`** with trusted admin IPs — Proxmox auto-generates required management access

+ rules from this set

+ - Use **security groups** for reusable rule sets (e.g., "webserver" group opening ports 80/443), applied to multiple VMs

+ for consistency

- Use **`ipfilter-net*` IPSets** per VM interface to prevent IP spoofing

- Enable firewall **logging** for dropped/rejected packets — disabled by default

- - **nftables backend** (PVE 8.2+): enable via `nftables: 1` in host.fw; supports

- forwarded traffic rules at host and VNet levels

+ - **nftables backend** (PVE 8.2+): enable via `nftables: 1` in host.fw; supports forwarded traffic rules at host and

+ VNet levels

### User Management and RBAC

- Grant permissions to **groups**, not individual users — shorter, maintainable ACLs

- - Use **resource pools** to group related VMs/containers/storage; assign permissions

- at the pool level rather than per-resource

- - Use **realms** for authentication: PAM (local), LDAP, Active Directory,

- OpenID Connect (Keycloak, Authentik)

- - Define **roles** with specific privileges (PVEVMAdmin, PVEAuditor,

- PVEDatastoreUser) — assign at the narrowest path needed

- - Use **privilege-separated API tokens** — token permissions are the intersection

- of user permissions and token-specific ACLs

- - Enforce **two-factor authentication** (TOTP, YubiKey, WebAuthn) at the realm

- level for all interactive accounts

+ - Use **resource pools** to group related VMs/containers/storage; assign permissions at the pool level rather than

+ per-resource

+ - Use **realms** for authentication: PAM (local), LDAP, Active Directory, OpenID Connect (Keycloak, Authentik)

+ - Define **roles** with specific privileges (PVEVMAdmin, PVEAuditor, PVEDatastoreUser) — assign at the narrowest path

+ needed

+ - Use **privilege-separated API tokens** — token permissions are the intersection of user permissions and token-specific

+ ACLs

+ - Enforce **two-factor authentication** (TOTP, YubiKey, WebAuthn) at the realm level for all interactive accounts

### Hardening

- - Use the **Enterprise repository** for production — the no-subscription and test

- repositories are not as thoroughly validated

+ - Use the **Enterprise repository** for production — the no-subscription and test repositories are not as thoroughly

+ validated

- Keep Proxmox VE updated — apply security patches promptly

- Restrict management interface access (dedicated VLAN, firewall rules)

- Disable root SSH login — use a non-root account with sudo

- Use API tokens with privilege separation for automation

- Do not run additional services (Docker, web servers) directly on the Proxmox host

- - Enable **non-free-firmware** repository for CPU microcode security updates

- (enabled by default on PVE 9.0 new installs)

+ - Enable **non-free-firmware** repository for CPU microcode security updates (enabled by default on PVE 9.0 new

+ installs)

</security>

---

## Monitoring

### Metrics Export

- - Configure **external metric servers** (Datacenter > Metric Server) to export

- host, guest, and storage stats to **InfluxDB** or **Graphite**

- - Use **Grafana** for dashboards — community dashboard ID 10048 provides

- per-host, per-VM, and per-storage visualization

+ - Configure **external metric servers** (Datacenter > Metric Server) to export host, guest, and storage stats to

+ **InfluxDB** or **Graphite**

+ - Use **Grafana** for dashboards — community dashboard ID 10048 provides per-host, per-VM, and per-storage visualization

### Critical Alerts

Configure notification matchers (PVE 8.1+ notification system) for:

- **Backup failures:** `match-field exact:type=vzdump`, `match-severity error`

- **Fencing/HA events:** `match-field exact:type=fencing`

- **Replication failures:** `match-field exact:type=replication`

- **ZFS errors:** enable ZED email notifications in `/etc/zfs/zed.d/zed.rc`

- **Disk health:** `smartmontools` for S.M.A.R.T. monitoring

- **Ceph health:** monitor for `HEALTH_WARN` and `HEALTH_ERR` states

### Notification Targets

- Supported targets: Sendmail/Postfix, SMTP relay, Gotify push notifications,

- Webhooks (Discord, Slack, Mattermost). Configure at Datacenter > Notifications.

+ Supported targets: Sendmail/Postfix, SMTP relay, Gotify push notifications, Webhooks (Discord, Slack, Mattermost).

+ Configure at Datacenter > Notifications.

</monitoring>

---

## Anti-Patterns

- These anti-patterns are non-obvious traps that the positive rules above do not

- fully convey — common mistakes where the "right" approach is counterintuitive.

+ These anti-patterns are non-obvious traps that the positive rules above do not fully convey — common mistakes where the

+ "right" approach is counterintuitive.

- | Anti-Pattern | Risk | Fix |

- |--------------|------|-----|

- | Running Docker directly on the PVE host | Conflicts with PVE networking/storage, complicates updates | Run Docker inside a VM or LXC container |

- | Swap on ZFS zvol | Blocks server during backups, high I/O load | Partition a physical disk for swap |

- | Using `host` CPU type in mixed clusters | Live migration fails on different CPU generations | Use `x86-64-v2-AES` or lowest common model |

- | LACP bonds for Corosync with default rate | 90s failover exceeds fencing timeout (~60s) | Set `bond-lacp-rate fast` on node and switch |

- | Load-balancing bond modes for Corosync | Asymmetric connectivity causes mass fencing | Use `active-backup` or LACP with fast rate |

- | Using no-subscription repo in production | Less validated packages, potential instability | Use Enterprise repository for production |

- | Updating all cluster nodes simultaneously | Watchdog fences nodes during CRM/LRM freeze | Update one node at a time, verify each |

- | Unbounded ZFS ARC on VM hosts | ARC silently consumes RAM, VMs crash from OOM | Cap ARC explicitly in `/etc/modprobe.d/zfs.conf` |

- | VXLAN with default MTU 1500 | 50-byte encapsulation overhead causes fragmentation | Set VNet MTU to 1450 (1370 with IPSEC) |

- | Storing PBS encryption key on backed-up system | Key lost when system fails, backups irrecoverable | Store key in password manager + offline backup |

- | Shared remote user for multiple PBS sync jobs | Jobs delete each other's snapshots with `remove-vanished` | Dedicated remote user per sync job |

+ | Anti-Pattern | Risk | Fix |

+ | ---------------------------------------------- | ---------------------------------------------------------- | ------------------------------------------------ |

+ | Running Docker directly on the PVE host | Conflicts with PVE networking/storage, complicates updates | Run Docker inside a VM or LXC container |

+ | Swap on ZFS zvol | Blocks server during backups, high I/O load | Partition a physical disk for swap |

+ | Using `host` CPU type in mixed clusters | Live migration fails on different CPU generations | Use `x86-64-v2-AES` or lowest common model |

+ | LACP bonds for Corosync with default rate | 90s failover exceeds fencing timeout (~60s) | Set `bond-lacp-rate fast` on node and switch |

+ | Load-balancing bond modes for Corosync | Asymmetric connectivity causes mass fencing | Use `active-backup` or LACP with fast rate |

+ | Using no-subscription repo in production | Less validated packages, potential instability | Use Enterprise repository for production |

+ | Updating all cluster nodes simultaneously | Watchdog fences nodes during CRM/LRM freeze | Update one node at a time, verify each |

+ | Unbounded ZFS ARC on VM hosts | ARC silently consumes RAM, VMs crash from OOM | Cap ARC explicitly in `/etc/modprobe.d/zfs.conf` |

+ | VXLAN with default MTU 1500 | 50-byte encapsulation overhead causes fragmentation | Set VNet MTU to 1450 (1370 with IPSEC) |

+ | Storing PBS encryption key on backed-up system | Key lost when system fails, backups irrecoverable | Store key in password manager + offline backup |

+ | Shared remote user for multiple PBS sync jobs | Jobs delete each other's snapshots with `remove-vanished` | Dedicated remote user per sync job |

---

## Application

### When Configuring Proxmox

- Apply these conventions without narrating each rule

- If the environment has existing patterns, follow them — flag divergences once

- Security practices are defaults, not optional add-ons

- Test changes in a non-production environment when possible

- Document non-obvious configuration choices in comments or a wiki

### When Reviewing Configuration

- Verify storage backend matches workload requirements

- Check network separation (Corosync, Ceph, guest, management)

- Verify HA prerequisites: shared storage, fencing, quorum

- - Check backup coverage: all production guests backed up, retention configured,

- notifications enabled, restores tested

- - Verify security: firewall enabled, ACME certificates, RBAC configured, no

- root API usage, 2FA enforced

- - Verify monitoring: metric server configured, ZED enabled, notification

- matchers for backup/fencing/replication failures

+ - Check backup coverage: all production guests backed up, retention configured, notifications enabled, restores tested

+ - Verify security: firewall enabled, ACME certificates, RBAC configured, no root API usage, 2FA enforced

+ - Verify monitoring: metric server configured, ZED enabled, notification matchers for backup/fencing/replication

+ failures

### When Writing Automation

- Use `pvesh` or the REST API — never screen-scrape the web GUI

- Use API tokens with privilege separation

- Handle API errors and task status polling (long operations return task IDs)

- Use cloud-init templates for VM provisioning

- Idempotent scripts: running twice produces the same result

- - For automation-heavy workloads with high API volume, increase `MAX_WORKERS`

- in `/etc/default/pvedaemon` and `/etc/default/pveproxy`

+ - For automation-heavy workloads with high API volume, increase `MAX_WORKERS` in `/etc/default/pvedaemon` and

+ `/etc/default/pveproxy`

</application>

---

## Integration

- This skill provides Proxmox VE discipline alongside sibling skills in the

- infrastructure plugin:

+ This skill provides Proxmox VE discipline alongside sibling skills in the infrastructure plugin:

- - **`devops`** — foundational discipline (IaC principles, change management,

- observability) that applies to all Proxmox work

- - **`networking`** — general network infrastructure (VLANs, firewalls, DNS)

- beyond PVE-specific networking covered here

- - **`ansible`** — configuration management for automating Proxmox host setup

- and post-provisioning

+ - **`devops`** — foundational discipline (IaC principles, change management, observability) that applies to all Proxmox

+ work

+ - **`networking`** — general network infrastructure (VLANs, firewalls, DNS) beyond PVE-specific networking covered here

+ - **`ansible`** — configuration management for automating Proxmox host setup and post-provisioning

- **`containers`** — Docker/Podman management inside PVE guests (VMs or LXC)

---

## Critical Rules

- - **HA without fencing is a data corruption risk** — never enable HA without a tested

- fencing mechanism

+ - **HA without fencing is a data corruption risk** — never enable HA without a tested fencing mechanism

- **Cap ZFS ARC explicitly** — unbounded ARC causes VM OOM crashes

- **Dedicated Corosync network** — never share with Ceph or migration traffic on 1GbE

- **Test restores regularly** — a backup that cannot be restored is worthless

- **API tokens with privilege separation** for all automation — never root credentials

- **Use VirtIO drivers** for all production VM disks and network interfaces

- **Proxmox is infrastructure. Treat it with the same rigor as production code:

- version-controlled configuration, tested changes, monitored operations, documented

- decisions.**

+ **Proxmox is infrastructure. Treat it with the same rigor as production code: version-controlled configuration, tested

+ changes, monitored operations, documented decisions.**