ai-hardware-selection · git:20260601.8eda6f7 · 2026-06-01 · sha256 c3fcd980a7496cd6
ai-hardware-selection git:20260601.8eda6f7A
Immutable. This exact content never changes and is served at /api/v1/blob/c3fcd980a7496cd6.
No tool to install first
claude plugin marketplace add https://markdownregistry.com/mp/a/art_bc35c2qyjioshbph/c3fcd980a7496cd6c0f55f45313b24402821f5f7877b0d76c22b09642d93bd19.json
Then /plugin install ai-hardware-selection@markdownregistry-bc35c2qyjioshbph in Claude Code. It installs this exact version, the one graded A here, pinned to the commit it was graded at, and leaves it there.
--- name: ai-hardware-selection description: | Selecting accelerators for AI workloads: GPU vs TPU vs NPU vs FPGA vs CPU, and the metrics that actually decide it — memory capacity & bandwidth, TOPS/ FLOPS, interconnect, and cost/Watt. Architect-level hardware-fit reasoning. USE WHEN: choosing AI hardware/accelerators, "which GPU", "TPU vs GPU", "NPU", "FPGA", "HBM/memory bandwidth", "TOPS", "cost per token", VRAM sizing for a model, training vs inference hardware, accelerator interconnect. DO NOT USE FOR: serving software topology (use `inference-serving-topology`); on-device runtimes (use `edge-inference`); generic CPU perf (use systems/hardware-aware-design). allowed-tools: Read, Grep, Glob --- # AI Hardware Selection ## The metric that usually decides: memory, then bandwidth For LLM **inference**, the binding constraint is typically **VRAM/HBM capacity** (weights + KV-cache must fit) and **memory bandwidth** (decode is memory-bound) — not raw FLOPS. Size first: `weights ≈ params × bytes/param` (e.g. 70B × 2B(FP16) ≈ 140GB → multi-GPU or quantize). Add KV-cache (grows with context × batch). Only then look at TOPS. ## Accelerator families | Type | Strength | Use | |---|---|---| | **GPU** (NVIDIA H/B-series, AMD MI) | Flexible, huge ecosystem, HBM | Training + inference, the default | | **TPU** | Matmul-dense, pod-scale interconnect | Large-scale training/inference on GCP | | **NPU** | Perf/Watt at low power | Edge / mobile / AI-PC inference | | **FPGA** | Custom low-latency dataflow | Niche ultra-low-latency / fixed pipelines | | **CPU** | Available, fine for small/batch | Small models, embeddings, light load | ## Other levers - **Interconnect** (NVLink, InfiniBand): decisive for multi-GPU training and tensor parallelism — bandwidth between accelerators bounds scaling. - **Precision support**: FP8/INT4 support multiplies effective throughput/capacity. - **Cost/Watt & TCO**: cloud per-hour vs owned; power/cooling; utilization. The honest metric is **cost per token (or per request) at target latency**. - **Training vs inference**: training needs FLOPS + interconnect + memory; inference needs memory capacity/bandwidth + latency. ## When to recommend what - Default / flexibility / training → NVIDIA GPUs (size by model VRAM). - Edge/mobile/low-power inference → NPU. - Hyperscale training on GCP → TPU pods. - Fixed ultra-low-latency pipeline → FPGA (only if justified).