git:20260314.9629b35 to git:20260314.4231fc0

---

name: proxmox

description: >-

configurations.

---

# Proxmox VE

**Production infrastructure demands production discipline. Every Proxmox configuration must be secure by default,

redundant where it matters, and automated where possible.**

## Route to Reference

- **Storage backends** — [`${CLAUDE_SKILL_DIR}/references/storage-backends.md`]: Backend capability matrix, ZFS tuning

(ARC/L2ARC/SLOG/volblocksize), Ceph configuration, LVM-Thin monitoring, storage selection decision tree

- **Networking** — [`${CLAUDE_SKILL_DIR}/references/networking.md`]: Bridge configuration, VLAN layout, bonding modes,

SDN zones (VXLAN/EVPN), MTU considerations, OVS vs Linux bridge, firewall lockout prevention

- **Clustering and HA** — [`${CLAUDE_SKILL_DIR}/references/clustering-and-ha.md`]: Corosync configuration, quorum math,

QDevice setup, split-brain prevention, fencing methods, HA groups, migration, quorum loss recovery

- **API and automation** — [`${CLAUDE_SKILL_DIR}/references/api-automation.md`]: REST API architecture, pvesh/qm/pct

reference, Terraform patterns, cloud-init customization (cicustom, network v2), hookscript lifecycle, CI/CD pipelines

- **Backup strategies** — [`${CLAUDE_SKILL_DIR}/references/backup-strategies.md`]: vzdump modes, PBS architecture,

encryption key management, garbage collection safety, verification jobs, retention policies, off-site sync patterns

## Guest Management

<guest-management>

- **Default to LXC** for trusted Linux workloads — near-zero overhead, high density

- **Use VMs** when the workload requires: non-Linux OS, full kernel isolation, PCIe passthrough, or live migration

- **Use unprivileged containers** (default) — they map container UID 0 to a non-privileged host UID, preventing

container escape attacks

- Only use privileged containers when unprivileged mode is incompatible (specific device access, certain NFS mounts)

### VM Configuration

- Use **VirtIO** drivers for disk and network — mandatory for production performance

- Enable **QEMU Guest Agent** inside every VM for graceful shutdown, snapshot consistency, and IP address reporting

- Use **OVMF (UEFI) + Q35 machine type** for PCIe passthrough and Secure Boot

- Set CPU type to `host` for maximum performance in homogeneous clusters; use `x86-64-v2-AES` or the lowest common CPU

generation when live migration across different CPU generations is required

- Enable **memory ballooning** for dynamic RAM management

- Enable **NUMA** topology for VMs with many cores on multi-socket hosts

### Container Configuration

- Set **explicit memory limits** — containers without limits can exhaust host RAM

- Enable **nesting** (`features: nesting=1`) only when required (Docker inside LXC)

- Use **bind mounts** to share host directories, not NFS/CIFS mounts inside the container

- PVE 9.0 removed cgroup v1 entirely — containers requiring cgroup v1 must move to VMs

### Templates and Cloning

- Build templates with the guest agent and cloud-init pre-installed

- Convert to template: `qm template <vmid>` or backup as `.tar.gz` for containers

- Use **linked clones** for development/ephemeral workloads (fast, shared base disk)

- Use **full clones** for production (independent, no template dependency)

</guest-management>

## Storage

<storage>

### Backend Selection

- **Local redundancy, data integrity, snapshots** → ZFS

- **Local snapshots/clones without ZFS overhead** → LVM-Thin

- **Shared storage for HA clusters (3+ nodes)** → Ceph

- **Simple shared storage (existing NAS/SAN)** → NFS or iSCSI

- **Deduplicated backups** → PBS

### ZFS Rules

- Use HBA in IT mode — never hardware RAID controllers with write-back cache

- **ARC sizing:** cap explicitly in `/etc/modprobe.d/zfs.conf`; 25-30% of host RAM for mixed workloads. PVE 8.1+

defaults to 10% (max 16GB) for new installs. Rule of thumb: 2GB base + 1GB per TB of storage

- Set `ashift=12` for 4K sector disks (most modern drives); incorrect ashift halves IOPS due to sector misalignment

- **Volblocksize** (VMs on zvols): 16k for mirrors/RAID10; increase for wide RAIDZ to reduce write amplification. Can

only be set at zvol creation

- **Recordsize** (containers on datasets): tune per workload — 8k for Postgres, 16k for MariaDB, 1M for large sequential

files

- Enable `compression=lz4` — can actually increase I/O performance by writing less data to disk

- **SLOG:** 8-32GB enterprise NVMe with power-loss protection; only holds ~5s of write data. Benefits sync-heavy

workloads (databases, NFS)

- **L2ARC:** only if ARC hit ratio is low and adding RAM is not possible; size 5-20x RAM; budget 1GB RAM per 50GB of

L2ARC for metadata

- Schedule regular scrubs (weekly or monthly)

- Plan capacity upfront — ZFS pools cannot be shrunk

- Never create swap on a ZFS zvol — causes blocking I/O during backups

### Ceph Rules

- Minimum 3 nodes for proper quorum and data distribution

- Dedicated 10GbE+ network for Ceph traffic (25GbE+ recommended for NVMe OSDs), separate from Corosync — Ceph rebalance

traffic saturates links and causes Corosync instability

- Use SSD/NVMe for OSD WAL/DB

- Dedicated disks for OSDs — never share with the host OS

- PVE 9.0 defaults to Ceph Squid (v19.2)

### Storage Anti-Patterns

- Running ZFS or Ceph on top of hardware RAID with write-back cache — defeats data integrity guarantees

- Overprovisioning LVM-Thin without monitoring — a full thin pool causes I/O errors for all guests on that pool

- Storing backups on the same physical disks as production data

- Letting ZFS ARC consume unbounded RAM — causes VM crashes when the hypervisor and VMs compete for memory

</storage>

## Networking

<networking>

### Core Model

Every guest connects to a **Linux bridge**. Use **VLAN-aware bridges** (single bridge with 802.1Q tagging) instead of

per-VLAN bridges. The VLAN-aware checkbox must be explicitly enabled — it is off by default.

### VLAN Best Practices

- Place the **management interface on a dedicated VLAN** — never share with guest traffic

- Configure **trunk ports** on physical switches for the Proxmox host — frames with VLAN IDs not allowed by the switch

are silently dropped

- Assign VLAN tags per guest in the network configuration

- Verify no double-tagging mismatch between Proxmox and switch native VLAN

### Traffic Separation

- **Management / Corosync** — 1GbE (dedicated)

- **Ceph cluster + public** — 10GbE (dedicated), 25GbE recommended

- **Migration** — 10GbE (recommended)

- **Guest** — depends on workload

**Critical rule:** Never combine Corosync traffic with high-bandwidth Ceph or migration traffic on a single 1GbE link.

Corosync is latency-sensitive — network contention causes cluster instability and false fencing.

### SDN

Use SDN (VXLAN zones) for overlay networking across nodes without physical switch changes. Use EVPN for advanced

**SDN gotchas:**

- VXLAN adds 50-byte header — set VNet MTU to 1450 (or 1370 with IPSEC)

- SDN changes are **staged**, not live — click Apply at Datacenter level

- DHCP requires a gateway configured on the subnet

- Multiple EVPN exit nodes require disabling `rp_filter` in sysctl

- **OVS vs Linux bridge:** OVS is automatically VLAN-aware and may resolve 10GbE throughput bottlenecks seen with native

Linux bridge

## Clustering and High Availability

<clustering>

### Cluster Rules

- Use a **dedicated network** for Corosync — latency under 5ms required

- Configure **redundant Corosync links** (up to 8 supported via Kronosnet) on separate physical networks

- For 2-node clusters, deploy a **QDevice** on a third machine for quorum — a 2-node cluster without QDevice is a

split-brain generator

- QDevice is discouraged for odd-numbered clusters — it becomes a single point of failure due to (N-1) vote allocation

- If using LACP bonds for Corosync, set `bond-lacp-rate fast` on both node and switch — default slow rate has 90s

failover, causing fencing after ~60s

- **Avoid** `balance-rr`, `balance-xor`, `balance-tlb`, `balance-alb` bond modes for Corosync — they cause asymmetric

connectivity and mass fencing

- Never join a node with existing VMs/containers to a cluster — start fresh

- Update nodes **one at a time** — the LRM requests a service freeze from the CRM during updates; if both are updating,

the watchdog fences the node

### HA Requirements

- **Shared or replicated storage** accessible from all HA nodes

- **Working fencing mechanism** — test before relying on HA

- **Minimum 3 quorum votes** (3 nodes, or 2 nodes + QDevice)

- Configure **HA groups** with node priorities for controlled failover

### Fencing

Fencing guarantees a failed node is offline before its services restart elsewhere. **HA without fencing is a data

corruption risk** — two nodes writing to shared storage simultaneously causes irrecoverable damage.

- Verify watchdog status: `ha-manager status`

- Test fencing by simulating node failure before going to production

- Use hardware watchdog (`iTCO_wdt` via `/etc/default/pve-ha-manager`) when available, software watchdog (`softdog`) as

fallback

- Configure `WATCHDOG_MODULE=iTCO_wdt` in `/etc/default/pve-ha-manager`

### Quorum Loss Recovery

If the cluster loses quorum, `pmxcfs` becomes read-only — no VM operations are possible. Emergency recovery:

- `pvecm expected 1` forces single-node quorum — use only to restore vital guests or fix the quorum issue itself

- **Never** make cluster changes (add/remove nodes, storage, guests) while expected votes are overridden

### Migration

- **Offline migration** (VMs and containers): guest stops, data transfers, guest starts on target

- Use a dedicated high-bandwidth network for migration traffic

</clustering>

## API and Automation

<automation>

### Authentication

- Use **API tokens with privilege separation** for all automation — never root credentials

- Store token secrets in vault or environment variables — never in code

- Use ticket authentication only for interactive or short-lived tools

### CLI Tools

- **`pvesh`** — direct REST API access from CLI

- **`qm`** — VM lifecycle management

- **`pct`** — container lifecycle management

- **`ha-manager`** — HA resource management

- **`pvecm`** — cluster management

- **`pvesm`** — storage management

### Terraform

- Use **API tokens** with privilege separation — never root credentials

- Use cloud-init templates as the base for Terraform-managed VMs — templates must have `qemu-guest-agent` installed or

Terraform hangs on "still creating"

- Store Terraform state remotely (GitLab HTTP backend, S3, Consul) — never local state for shared infrastructure

- Use `lifecycle { ignore_changes }` for fields Proxmox modifies outside Terraform (e.g., disk size after manual resize)

- CI/CD pipeline: validate -> plan (save artifact) -> apply (manual trigger)

- Never commit API secrets to git — use CI/CD variables (`TF_VAR_*`)

### Cloud-Init

- Prepare a base VM with `qemu-guest-agent` and `cloud-init` installed

- Add a Cloud-Init drive (IDE or SCSI CD-ROM)

- Configure network, SSH keys, and user data via the Cloud-Init panel or API

- Convert to template, deploy via linked clone

- Use **SSH key authentication** — cloud-init password storage is less secure

- **cicustom** for advanced needs: reference custom YAML snippets for user, network, and meta data from a

snippets-capable storage

- Store cicustom snippets on shared storage (CephFS) in clusters for HA

- Windows templates: use Cloudbase-Init with `configdrive2` format + Sysprep

</automation>

## Backups

<backups>

### Backup Rules

- Follow the **3-2-1 rule**: 3 copies, 2 media types, 1 off-site

### PBS Security

- Restrict PVE backup user/token to create-only access (no delete) on PBS

- Separate PBS admin credentials from PVE access

### Non-Negotiable

- **Test restores regularly.** A backup that cannot be restored is worthless.

</backups>

## PCIe Passthrough

<passthrough>

### Requirements

- CPU: VT-d (Intel) or AMD-Vi enabled in BIOS/UEFI

- VFIO modules loaded: `vfio`, `vfio_iommu_type1`, `vfio_pci`, `vfio_virqfd`

### Configuration

### GPU-Specific Issues

- **NVIDIA Error 43 (Windows):** set CPU type to `host`, add `options kvm ignore_msrs=1` in `/etc/modprobe.d/kvm.conf`

- **AMD reset bug** (Vega, Polaris, some Navi): GPU fails to reset after VM shutdown, preventing reuse without host

reboot. Fix: install `vendor-reset` kernel module for vendor-specific reset quirks. RDNA2+ generally unaffected

- **NVIDIA vGPU:** officially supported since vGPU Software 18 on PVE. Requires valid NVIDIA entitlement. Ampere+ GPUs

### LXC Device Passthrough

Since PVE 8.2, device passthrough for containers is configurable via the UI. Limited compared to VM passthrough — no

full PCIe passthrough, but supports specific device access (GPU rendering, USB devices).

### Troubleshooting

- `dmesg | grep -e DMAR -e IOMMU` — verify IOMMU is enabled

- `pvesh get /nodes/{node}/hardware/pci --pci-class-blacklist ""` — list groups

- IOMMU groups can change between kernel major versions — verify after updates

</passthrough>

## Security

<security>

### Certificates

### Firewall

- Use **`ipfilter-net*` IPSets** per VM interface to prevent IP spoofing

### User Management and RBAC

### Hardening

- Restrict management interface access (dedicated VLAN, firewall rules)

</security>

## Monitoring

<monitoring>

</monitoring>

## Anti-Patterns

## Application

<application>

### When Configuring Proxmox

- Security practices are defaults, not optional add-ons

### When Reviewing Configuration

### When Writing Automation

</application>

## Integration

This skill provides Proxmox VE discipline alongside sibling skills in the infrastructure plugin:

- **`devops`** — foundational discipline (IaC principles, change management, observability) that applies to all Proxmox

work

- **`networking`** — general network infrastructure (VLANs, firewalls, DNS) beyond PVE-specific networking covered here

- **`ansible`** — configuration management for automating Proxmox host setup and post-provisioning

- **`containers`** — Docker/Podman management inside PVE guests (VMs or LXC)

## Critical Rules

- **HA without fencing is a data corruption risk** — never enable HA without a tested fencing mechanism

- **Cap ZFS ARC explicitly** — unbounded ARC causes VM OOM crashes

- **Dedicated Corosync network** — never share with Ceph or migration traffic on 1GbE

- **Test restores regularly** — a backup that cannot be restored is worthless

- **API tokens with privilege separation** for all automation — never root credentials

- **Use VirtIO drivers** for all production VM disks and network interfaces

**Proxmox is infrastructure. Treat it with the same rigor as production code: version-controlled configuration, tested

changes, monitored operations, documented decisions.**