vllm-nvidia-hardware git:20261018.fa3e7b2A

Immutable. This exact content never changes and is served at /api/v1/blob/898985ae4d117509.

---
name: vllm-nvidia-hardware
description: |-
  NVIDIA AI-hardware + vLLM-platform reference covering Hopper (H100/H200), Blackwell (B100/B200/B300) and Blackwell Ultra, Grace-Blackwell superchips and NVL72 racks (GB200, GB300), Vera Rubin (R100/R300) with VR200 NVL144 and Kyber NVL576, Dell PowerEdge XE family and IR5000/IR7000/IR9048 racks. Per-SKU HBM, FP4/FP8/FP16 TFLOPs, NVLink5, TDP, rack power/cooling (135 kW GB300, 180-220 kW NVL144, 600 kW Kyber), DLC vs RDHx, 800 VDC HVDC. Memory-wall roofline, HBM3E→HBM4 supply 2026. vLLM attention-backend × SM matrix, FP4/FP8 paths, KV connectors, Blackwell gotchas (SM103 TRTLLM hang, 270 vs 279 GB B300 bin split).
when_to_use: |-
  Trigger on NVIDIA-AI hardware sizing, procurement, facility, or vLLM-deployment questions. Triggers — HBM capacity/bandwidth, TDP, NVLink, GB300 NVL72 power/cooling, Vera Rubin timing, Dell PowerEdge XE, IR7000 racks, ORv3 power shelves, DLC flow rate, warm-water 45 °C, 800 VDC HVDC, rack weight, floor load, HBM supply 2026, FlashAttention/FlashInfer/FlashMLA/CUTLASS MLA/TRTLLM per SM, NVFP4, MXFP4, NixlConnector, MooncakeConnector, LMCache, CUDA 13 arch list, SM100/SM103/SM120 gotchas, memory-wall roofline, prefill-vs-decode. Narrow phrasings — "HBM on our XE9780", "wait for Rubin", "which GPU for {model}", "B200 vs H200 choice". Also implicit — "sizing for {model}", "audit hardware", "rack power budget", "can this GPU run {model}", "GPU recommendation", "deploy-memo hardware", "spec-study hardware".
---

# vLLM on NVIDIA hardware — Hopper through Rubin

Target audience: operators who run vLLM on NVIDIA datacenter GPUs, sizing from single H100 nodes up to GB300 NVL72 racks, and evaluating Vera Rubin for 2026–2027 purchases.

This skill is a **reference**, not a walkthrough — most of the content is SKU tables, facility prerequisites, and platform compatibility matrices. The SKILL.md body holds the quick-answer shortcuts; the `references/` directory has the full tables. Read the reference file that matches the question.

## The one thing to know before anything else

LLM inference has two phases with radically different bottlenecks:

- **Prefill** is compute-bound (GEMMs, AI ≫ ridge point) — more FLOPs help.
- **Decode** is memory-bandwidth-bound (AI ≈ 1, 100× below the ridge) — more HBM bandwidth helps, more FLOPs don't.

Every hardware decision — FP4 vs FP8, B300's higher FLOPs with the same 8 TB/s, NVL72's domain collapse, Rubin's HBM4 jump to ~20 TB/s — is about relieving the memory wall on decode while keeping prefill healthy. Read `references/fundamentals.md` for the roofline math and the HBM roadmap context that makes the rest of the tables meaningful.

## Quick-answer router

**Hardware specs** ("what's the HBM on X?", "TDP of Y?")
- NVIDIA GPU SKUs (Hopper, Blackwell, Blackwell Ultra) → `references/gpu-specs.md`
- Vera Rubin roadmap (R100, Rubin Ultra, NVL144, Kyber NVL576) → `references/rubin-roadmap.md`
- Dell PowerEdge XE servers → `references/dell-xe.md`
- GB300 NVL72 vendor landscape + facility prereqs → `references/nvl72-procurement.md`

**Memory-wall reasoning** ("why does FP4 help?", "why NVL72?")
- Fundamentals (roofline, BW/FLOPs gap, HBM roadmap, supply, energy) → `references/fundamentals.md`
- Mitigations × platform matrix (what works on Hopper vs Blackwell) → `references/mitigations.md`

**vLLM on NVIDIA** ("what backend on SM100?", "FlashMLA on Blackwell?")
- Attention backend × SM matrix, quant, KV connectors, known gotchas, release-note highlights → `references/vllm-platform-matrix.md`
- GEMM backends (DeepGEMM vs CUTLASS, FP8 alignment, E8M0 scaling, JIT cache) → `references/gemm-backends.md`

## The five most common operator mistakes this skill exists to prevent

1. **Conflating the two B300 bins.** NVIDIA ships B300 in two bins: **DGX / GB300 Superchip = 279 GB HBM3E / 8 TB/s / 1,400 W** (launch coverage says 288 GB; size against the datasheet's 279), and **HGX B300 NVL8 SXM6 (OEM baseboards) = 270 GB / 7.7 TB/s / 1,100 W**. Dell PowerEdge XE9780/XE9780L/XE9780LAP/XE9785/XE9785L all carry the **HGX NVL8 bin** (270 GB / 1,100 W). Getting this wrong puts the 8-GPU TDP budget off by 2.4 kW and the per-GPU HBM off by 9 GB. `references/gpu-specs.md` §2 has both rows.

2. **Assuming B300 is HBM4.** It isn't. B300 uses HBM3E (12-Hi stacks, 279 GB at the top bin). HBM4 first ships in **Vera Rubin R100** (H2 2026) with ~20 TB/s per GPU. This matters for roadmap planning.

3. **Assuming GB300 NVL72 can use rear-door heat exchangers.** It can't — at 135 kW nominal / 155 kW peak per rack, direct liquid cooling is mandatory on GPUs + CPUs + NVSwitch + CX-8 NICs. Facility work (row-level CDU, 3φ 480V feeds, 1500 kg/m² floor) routinely takes 6–9 months. See `references/nvl72-procurement.md`.

4. **Confusing Dell's "XE9780 (air)" with "XE9780L (liquid)".** They are separate SKUs in the same spec sheet. Also note that **XE9680L in Dell's 2026 portfolio is a 4U B200 liquid-cooled node**, not a liquid-cooled variant of the 6U XE9680 H100/H200 chassis. `references/dell-xe.md` has the name disambiguation.

5. **Running TRTLLM attention on GB300 with FlashInfer 0.6.7 / 0.6.7.post1.** SM103 (GB300) **hung** with FlashInfer 0.6.7 (regression vs 0.6.6) — **fixed 2026-04-07** via flashinfer-ai/flashinfer#2956 (closes #2939), first shipped in **0.6.7.post2** (0.6.7.post1 still hangs). If pinned to 0.6.7 or 0.6.7.post1, disable TRTLLM on SM103 (the FlashInfer default backend is actually faster there). `references/vllm-platform-matrix.md` has this and other Blackwell readiness notes.

## Operator cheat sheet

### Running vLLM on Hopper (H100/H200)
- Default to FP8 weights + FP8 KV cache on H200 for long context.
- Use FlashMLA for DeepSeek models; FlashAttention-3 elsewhere.
- Disagg prefill across nodes needs XDR IB + GPUDirect-RDMA — use CPU/NVMe offload (LMCache) first.
- **No FP4. No NVL72. No 270+ GB single-GPU.**

### Running vLLM on Blackwell (B200/B300)
- Switch to FP4 (NVFP4 via ModelOpt checkpoints) — 1.8–2× decode throughput on the same HBM budget.
- NVL72 collapses disagg cost — keep prefill/decode inside one NVLink5 domain; use `NixlConnector`.
- Pin vLLM **≥ v0.19** for first-class B300/GB300 (SM 10.3) support, and
  **≥ v0.25.1 if serving NVFP4**: v0.25.0 and earlier can emit garbage
  (repeated `!!!!!`) from the fused allreduce+RMSNorm+quant path on NVFP4
  models with Gemma/Qwen-style RMSNorm. Silent wrong output, no crash — see
  `references/vllm-platform-matrix.md` §6.
- **≥ v0.32.0 for NVFP4/MXFP4 MoE with expert parallelism on the CUTLASS MoE
  backend**: earlier releases ignored `expert_map`, routing tokens to the wrong
  local experts — degenerate output (~0% GSM8K), no error (#59042).
- 270 GB (HGX bin) or 279 GB (DGX bin) HBM per B300 often removes the *need* for KV offload on 70B-scale models. LMCache still earns its keep for 1M+ context and heavy prefix reuse.
- **Upgrading to v0.27.0 is an environment change, not a version bump:** `requirements/cuda.txt` pins `torch==2.13.0` / `torchvision==0.28.0` (#48155, flagged breaking upstream; Triton 3.7.1 rides along with torch rather than being pinned separately), and the image build sets `NCCL_VERSION=2.30.7` — the NCCL floor is what unlocks DeepEPv2 for Wide-EP (#45321). Rebuild the container; don't pip-upgrade in place.
- **v0.32.0 floors:** Python ≥ 3.11 (#60402), `transformers>=5.16.1,<5.20.0` +
  `tokenizers>=0.23.2`, GCC ≥ 13 for CUDA source builds (CMake fails early,
  #60259). Unchanged since v0.27.0: `torch==2.13.0`, image `CUDA_VERSION=13.0.3`,
  `NCCL_VERSION=2.30.7`. FlashInfer is 0.7.0.post1 and `flashinfer-cubin` is no
  longer a pip dependency: after every pip install/upgrade run `vllm download-kernels`
  (#58765), or startup JIT-compiles for minutes on Blackwell and a version-mismatched
  kernel set makes FlashInfer refuse to import. The `vllm/vllm-openai` image does it at build.

### Buying NVIDIA hardware in 2026
- **GB300 NVL72 is safe for 2026 capacity:** size the row for 135–155 kW, DLC @ 25 °C, 3φ 480 V. Vendors: NVIDIA DGX B300, Dell IR7000 + XE9780L / XE8712 sleds, Supermicro SRS-GB300-NVL72, Lenovo 7DJVCTO2WW, HPE.
- **Greenfield rows should be spec'd for Rubin NVL144:** 180–220 kW, 45 °C W45 warm water, 800 VDC HVDC, new MGX rack. Retrofitting later is expensive.
- **Rubin timing, as of 2026-10-18:** first deliveries are in production use — CoreWeave announced Vera Rubin NVL72 availability on 2026-09-30 with a first customer running production workloads (cluster stood up early September), and Supermicro announced it is shipping racks on 2026-09-23. NVIDIA (results, 2026-08-26) says racks are "running at partners including CoreWeave, Google Cloud, Microsoft Azure, Oracle Cloud Infrastructure and Nebius", but gives **no general-availability or customer-delivery date**, and only CoreWeave has announced customer availability. Early allocation is not broad supply: the signal to plan a purchase against is still an OEM quoting firm order dates. **vLLM Rubin (`sm_107`) support is preview-only** — native SM107 needs **CUDA ≥ 13.4**; release images stay CUDA 13.0.3 with no SM107 cubins (Rubin runs SM100-family `10.0f` code); prebuilt Rubin images exist only as `vllm/vllm-openai:cu134-nightly` (#55953, PyTorch nightlies); tracking issue #49735 is still open. Detail in `references/rubin-roadmap.md`.
- **Expect HBM allocation and PSU rectifier shelf lead times to dominate schedule risk.** SK hynix / Micron / Samsung HBM sold out through CY26.

### Key numbers to memorize
| Metric | Value |
|---|---|
| GB300 NVL72 rack power | 135 kW nominal / 155 kW peak |
| GB300 NVL72 aggregate HBM | 20.1 TB HBM3E (72 × 279 GB; NVIDIA: "> 20 TB") |
| GB300 NVL72 aggregate HBM BW | ~576 TB/s |
| GB300 NVL72 NVLink domain | 72 GPUs flat, 130 TB/s |
| B300 (HGX NVL8, Dell fleet) | 270 GB / 1,100 W |
| B300 (DGX / GB300 Superchip) | 279 GB / 1,400 W |
| H200 SXM5 | 141 GB HBM3E / 4.8 TB/s / 700 W |
| Rubin R100 (H2 2026) | ~288 GB HBM4 / ~20 TB/s |
| Rubin NVL144 rack power | 180–220 kW (800 VDC HVDC) |
| Rubin Ultra Kyber NVL576 rack | 600 kW |

## Paired reference

If this skill helps with *why* a hardware choice matters for KV-cache sizing, the companion skill `vllm-caching` covers *how* to configure tiered KV (`--kv-offloading-size`, LMCache, NixlConnector, MooncakeConnector). Both skills should trigger together on combined "we're buying GB300 and need to size LMCache" style questions.

## Before stating a version, pin or status

- **Read dependency pins from `requirements/{cuda,common,rocm}.txt` at the release
  tag, never from release-note prose** — the notes' "Dependencies" section mixes CI
  pins (`requirements/test/*`) with runtime pins.
- **Read the bug-fix sections of the current patch release, not only features** —
  silent-corruption fixes (wrong text, no crash) land there.
- **Anchor shipping claims to dated, checkable artifacts** — a named customer in
  production, an OEM quoting firm order dates — never to "in production" statements,
  which describe fab output, not delivery.
- **Re-probe every "not supported yet" before repeating it** — negatives about a
  moving upstream decay fastest.
- Prefer NVIDIA first-party datasheets, then OEM datasheets (Dell/Lenovo/Supermicro),
  SemiAnalysis, then news; where sources disagree, report the range with each citation.
  Sources and the verification stamp: `references/sources.md`.