model-gateway-routing · git:20260601.8eda6f7 · 2026-06-01 · sha256 7f570a10280b228c
model-gateway-routing git:20260601.8eda6f7A
Immutable. This exact content never changes and is served at /api/v1/blob/7f570a10280b228c.
No tool to install first
claude plugin marketplace add https://markdownregistry.com/mp/a/art_rviwkvhxfxxcutf2/7f570a10280b228c5c66654899ff483756ae04dca605733e20794af9bd58a5eb.json
Then /plugin install model-gateway-routing@markdownregistry-rviwkvhxfxxcutf2 in Claude Code. It installs this exact version, the one graded A here, pinned to the commit it was graded at, and leaves it there.
--- name: model-gateway-routing description: | Model gateway / LLM router architecture: a control point in front of multiple models/providers for routing (cost/quality/latency), fallback, rate limiting, caching, observability, and governance. Architect-level, multi-provider. USE WHEN: designing an LLM gateway/router, "model router", "LLM gateway", "multi-provider", "fallback", "cost routing", "LiteLLM", "Envoy AI Gateway", semantic cache, central key/quota/observability for LLM calls. DO NOT USE FOR: single-engine serving (use `inference-serving-topology`); edge/cascade (use `hybrid-edge-cloud`); agent orchestration (use `agentic-architecture`). allowed-tools: Read, Grep, Glob --- # Model Gateway / Routing A gateway is a single control point between apps and many models/providers. It turns "which model?" and cross-cutting concerns into infrastructure. ## What it centralizes (the reasons to build/buy one) - **Routing**: pick a model per request by **cost / quality / latency / context length / capability**, or A/B and canary new models. - **Fallback & resilience**: retry/failover across providers on error or rate limit; circuit-break a failing provider. - **Cost control**: per-team/app budgets, quotas, and **cost attribution**; route cheap queries to cheap models. - **Caching**: exact + **semantic cache** to skip duplicate/near-duplicate calls. - **Security/governance**: central API-key custody, PII redaction, audit logs, policy (which teams may call which models). - **Observability**: latency/tokens/cost/error metrics in one place. Implementations: **LiteLLM**, **Envoy AI Gateway**, cloud AI gateways, or custom. ## Design decisions - **Routing policy**: static (rules) vs learned/heuristic (route by predicted difficulty). Keep it explainable; mind added hop latency. - **Sync vs streaming**: must pass through token streaming with low overhead. - **Statelessness**: keep the gateway stateless + horizontally scalable; push state (cache, budgets) to fast stores. - **Failure semantics**: define what happens when all providers fail. ## When to recommend - Multiple models/providers, multiple teams, real cost/governance needs → yes. - Single model, single team, prototype → a gateway is premature; call the model directly and add the gateway when the second model/provider/team appears.