inference-serving-topology · git:20260601.8eda6f7 · 2026-06-01 · sha256 65cb5f572e4f19cf
inference-serving-topology git:20260601.8eda6f7A
Immutable. This exact content never changes and is served at /api/v1/blob/65cb5f572e4f19cf.
No tool to install first
claude plugin marketplace add https://markdownregistry.com/mp/a/art_l6i7c2smarhtjupg/65cb5f572e4f19cf3e2ac38f00c55375e353aa96b55e21639ec7ba98a934a99b.json
Then /plugin install inference-serving-topology@markdownregistry-l6i7c2smarhtjupg in Claude Code. It installs this exact version, the one graded A here, pinned to the commit it was graded at, and leaves it there.
--- name: inference-serving-topology description: | LLM/model inference serving architecture: the engine → serving → orchestration layering (vLLM/SGLang/TensorRT-LLM, Triton, KServe/Ray Serve), KV-cache & continuous batching, prefill-decode disaggregation, and scaling. Architect-level topology, not model training. USE WHEN: designing model/LLM serving infra, "vLLM", "SGLang", "TensorRT-LLM", "Triton", "KServe", "Ray Serve", "continuous batching", "KV cache", "prefill decode", "TTFT", multi-GPU/multi-model serving, inference autoscaling. DO NOT USE FOR: on-device (use `edge-inference`); provider routing (use `model-gateway-routing`); RAG app logic (use rag/rag-frameworks skills). allowed-tools: Read, Grep, Glob --- # Inference Serving Topology ## The three layers (name which you're designing) 1. **Engine** — executes the model on accelerators: **vLLM**, **SGLang**, **TensorRT-LLM**. Owns paged **KV-cache**, **continuous (in-flight) batching**, quantization. This is where throughput/latency is won. 2. **Serving** — request routing, batching policy, API contract, metrics, rate limiting: **Triton** (production shell around an engine), **KServe**, **LiteLLM/Envoy AI Gateway**. 3. **Orchestration** — scaling, health, placement: **Kubernetes + KEDA**, **Ray Serve**, llm-d, GKE Inference Gateway. A common 2026 pairing: **vLLM as the token engine + Triton as the production shell**; **Ray Serve** when you need multi-GPU/multi-node distributed strategies. ## Levers that decide the topology - **KV-cache** is the memory bottleneck for LLMs → paged KV (vLLM), cache reuse, quantized KV. Drives max batch / context. - **Continuous batching** (vs static) is mandatory for throughput. - **Prefill–decode disaggregation**: split the compute-bound prefill from the memory-bound decode onto different pools → better utilization at scale. - **Parallelism**: tensor / pipeline / expert (MoE) / data-parallel attention — chosen by model size vs GPU memory. - **Targets**: state TTFT (time-to-first-token, low hundreds of ms) and inter-token latency (tens of ms) goals — they drive batching/parallelism. ## Scale ladder Single GPU + vLLM → multi-GPU one node → Ray Serve/KServe multi-node + KEDA autoscaling → disaggregated prefill/decode + multi-region. Don't jump tiers without a load/latency reason. ## When to recommend what - One model, moderate load → single-node vLLM (+ Triton for ops). - Many models / platform team → Triton + KServe on k8s. - Large model / high concurrency / SLO-driven → Ray Serve, disaggregation, tensor/expert parallelism.