llm-cost-latency-benchmark · diff
git:20260911.25436b5 to git:20261005.2bc0998
431 added, 12 removed. Audit A to A.
---
name: llm-cost-latency-benchmark
- description: "Measure an AI workflow's observed token cost and end-to-end latency across representative cases with reproducible configuration."
+ description: "Benchmark AI workflow cost and latency with TTFT/TPOT/ITL/e2e percentiles, cache-aware token billing, goodput under SLOs, and a quality floor. Use before claiming a model, prompt, or infra change is cheaper or faster."
category: development
risk: safe
source: self
source_type: self
date_added: "2026-09-11"
- tags: ["ai-workflows", "evaluation", "llm-cost-latency-benchmark"]
+ tags: ["ai-workflows", "evaluation", "latency", "ttft", "tpot", "cost", "tokens", "goodput", "prompt-caching", "slo"]
tools: ["claude", "cursor", "gemini", "codex"]
---
- # Cost and Latency Benchmark
+ # LLM Cost & Latency Benchmark AI Skill Guide (Claude)
- ## Scope
+ ## Overview & Engine Architecture
- Define the request population, concurrency, warm-up policy, cache state, budget, and required quality floor. Read current provider pricing from an authoritative source and record its date; do not hard-code remembered prices as current.
+ Cost and latency are **not one number**. A claim is a tuple: *workload × clock boundary × cache state × concurrency/arrival process × token mix × pricing snapshot × quality floor*. Mixing those variables produces vendor-slide fiction. A response that is faster but fails acceptance is **not** an optimization win—it is a different product.
+ Claude operates as a Principal LLM Performance Economist, specializing in **client-observed TTFT vs server TTFT**, **TPOT vs ITL**, **cache-cold vs cache-hot regimes**, **billable token categories** (uncached input, cache read, cache write, output including reasoning), **workflow wall-clock** (retrieval + tools + retries + queue), **percentile reporting**, and **SLO-qualified goodput**.
+
+ ```
+ ┌─────────────────────────────────────────────────────────────┐
+ │ Cost × Latency Benchmark Stack │
+ │ │
+ │ Workload contract │
+ │ ├── Case population (prod-shaped) + quality floor │
+ │ ├── Input/output length distributions (or fixed I/O) │
+ │ ├── Concurrency + arrival process (closed vs open loop) │
+ │ └── Cache policy: cold | warm | production-mix │
+ │ │
+ │ Timing planes (never conflate) │
+ │ ├── Client TTFT: send → first nonempty streamed token │
+ │ ├── Prefill / queue / network (diagnostic breakdown) │
+ │ ├── TPOT / ITL after first token │
+ │ └── Workflow e2e: user intent → final accepted artifact │
+ │ │
+ │ Cost planes │
+ │ ├── Provider usage: input, cached_read, cache_write, out │
+ │ ├── Non-token: tools, search, containers, retries, fails │
+ │ └── Pricing manifest: URL + retrieved_at + model tier │
+ │ │
+ │ Decision │
+ │ ├── Percentiles + sample size + goodput@SLO │
+ │ └── Compare only configs that meet the quality floor │
+ └─────────────────────────────────────────────────────────────┘
+ ```
+
+ ---
+
+ ## When to use / when not to
+
+ **Use when**
+
+ - Choosing models, regions, streaming settings, prompt-cache layouts, or concurrency limits.
+ - Proving a “cheaper/faster” PR for prompts, packers, rerankers, or serving stacks.
+ - Setting SLOs (interactive chat, voice, batch) and capacity planning.
+ - Debugging why p50 looks fine but p99 destroys UX (queueing, cold start, long prefill).
+
+ **Do not use when**
+
+ - Quality is undefined—run `@prompt-regression-gate` / `@ai-evaluation-dataset` first and set a **quality floor**.
+ - Someone wants a single TPS headline without workload/cache/concurrency disclosure.
+ - Extrapolating 10 synthetic short prompts to production guarantees.
+ - Hard-coding remembered $/MTok as “current” without a dated pricing source.
+
+ ---
+
+ ## Deep metric dictionary (formulas you must name)
+
+ Let request `i` be sent at client time `s_i`, first **nonempty usable** streamed token arrive at `f_i`, last token at `e_i`. Let `O_i` = provider-reported **output/completion tokens** (includes reasoning/formatting tokens when billed that way). Let `I_i` = total input/prompt tokens.
+
+ | Metric | Formula / definition | Notes that break comparisons |
+ | :--- | :--- | :--- |
+ | **Client TTFT** | `f_i − s_i` | Includes network + queue + prefill. State client region. |
+ | **Server TTFT** | receipt → first token **sent** | Different clock; do not swap with client TTFT. |
+ | **E2E request latency** | `e_i − s_i` | Model-call only unless labeled **workflow e2e**. |
+ | **TPOT** | `(e_i − f_i) / (O_i − 1)` | Undefined if `O_i < 2`. Excludes first token by convention (NVIDIA/vLLM-style). |
+ | **ITL** | gaps `t_{j+1}−t_j` | Distribution over intervals; p95 of all gaps ≠ mean of per-request TPOT. |
+ | **Decode TPS (per user)** | `(O_i − 1) / (e_i − f_i)` = `1/TPOT` | Not full-request rate. |
+ | **Full-request TPS** | `O_i / (e_i − s_i)` | Includes TTFT; lower; different claim. |
+ | **Aggregate TPS** | `sum(O) / T_run` | Fleet/GPU scope must be named. |
+ | **Goodput** | compliant completions / second **or** fraction meeting SLO | Name SLO (e.g. TTFT p99≤X ∧ TPOT p99≤Y); count errors as noncompliant. |
+
+ **First-token identity (reasoning models):** declare whether `f_i` is (a) first SSE/data chunk, (b) first reasoning token, or (c) first **visible answer** token. Voice/chat UX usually cares about (c). Chunk ≠ token (common harness bug called out on r/LocalLLaMA tooling).
+
+ **OpenTelemetry alignment (when instrumenting):** prefer `gen_ai.client.operation.time_to_first_chunk` / `gen_ai.usage.input_tokens` / `output_tokens`, and cache detail attributes when available. Report **billed** units when providers distinguish billed vs consumed.
+
+ ---
+
+ ## Cost accounting (think in billable categories)
+
+ ### Token classes (do not collapse)
+
+ Providers increasingly price **separately**:
+
+ 1. **Uncached input** — ordinary prompt tokens
+ 2. **Cache read (`cached_tokens`)** — discounted reuse of a prefix
+ 3. **Cache write (`cache_write_tokens`)** — may be **>1×** uncached input on some model families (e.g. OpenAI docs: writes at 1.25×, reads at 0.1× on newer families—**verify live pricing**)
+ 4. **Output / completion** — includes **reasoning tokens** and often invisible formatting/tool-channel tokens; billed as output even when not shown to the user
+
+ **Arithmetic sketch** (replace rates from a dated pricing pull):
+
+ ```text
+ cost ≈
+ u_in * rate_uncached_in
+ + c_read * rate_cached_in
+ + c_write * rate_cache_write
+ + out * rate_out
+ + Σ tool_calls * tool_rate
+ + Σ non_token_fees
+ ```
+
+ Rules:
+
+ - Prefer **provider `usage`** over local `len/4` guesses (tools, images, schemas break heuristics).
+ - If usage is missing → mark **`unknown`**, never invent.
+ - **Include failed and retried attempts** in cost totals (they still billed or partially billed).
+ - Stream with `stream_options.include_usage` (or provider equivalent) so finals are recorded.
+ - Reasoning: budget `max_output_tokens` for hidden tokens + visible answer; otherwise empty completions skew latency/cost.
+
+ ### Pricing manifest (mandatory artifact)
+
+ ```json
+ {
+ "retrieved_at": "2026-10-05T18:00:00Z",
+ "source_url": "https://…/pricing",
+ "model": "…",
+ "currency": "USD",
+ "rates_per_1m": {
+ "input_uncached": null,
+ "input_cached_read": null,
+ "input_cache_write": null,
+ "output": null
+ },
+ "notes": "Paste exact rows; do not rely on memory."
+ }
+ ```
+
+ ### Unit economics to report
+
+ | KPI | Definition |
+ | :--- | :--- |
+ | **$/accepted task** | Total $ for all attempts / tasks that passed quality floor |
+ | **$/1k input tokens (effective)** | Blended after cache hit mix |
+ | **Cache hit rate** | `cached_read / input` (verify via usage fields, not hope) |
+ | **Retry tax** | $ and latency of retries / successes |
+ | **Cost at iso-quality** | Only among configs meeting the floor |
+
+ Cheapest raw tokens with 2× retries and quality fails is usually **more expensive per accepted task**.
+
+ ---
+
+ ## Workload contract (comparability gate)
+
+ Before trusting any A vs B claim, both sides must match—or differences must be labeled as the **intentional** experimental factor:
+
+ 1. **Model + snapshot/version** + tokenizer
+ 2. **Prompt/tool schema hashes**
+ 3. **Input/output length policy** (fixed I/O vs production distribution)
+ 4. **Streaming on/off** (TTFT/ITL meaningless without streaming)
+ 5. **Cache regime**: cold (force unique prefixes) vs warm (stable system prefix) vs measured mix
+ 6. **Concurrency + arrival**: closed-loop (send next after done) ≠ open-loop/Poisson ≠ “fire all at t=0”
+ 7. **Client vantage** (region/network)
+ 8. **Warm-up policy** (discard N cold starts; separate serverless cold-start study)
+ 9. **Quality floor** + same eval dataset hash
+ 10. **Pricing date** for cost claims
+
+ If any required field is missing → verdict **cannot-normalize**, not “approximately faster.”
+
+ ### Cache is a different experiment
+
+ Prefix/KV caching can dominate TTFT. Practitioners repeatedly show:
+
+ - Accidental cache hits from reused prompts make TTFT look unrealistically low.
+ - Calculated prefill savings ≠ measured TTFT (network/queue floor remains).
+ - Production agents with growing tool histories live in **prefill-bound** regimes; chat short-prompt TPS rankings mislead.
+
+ **Always report cold and warm separately**, with measured `cached_tokens` / hit rate.
+
+ ### Concurrency and tails
+
+ - p50 TTFT at concurrency=1 ≈ network + single prefill—**almost useless** for capacity.
+ - Test near expected **p95 concurrent** load.
+ - Under load, TTFT is often **queue wait**; log queue separately when the server exposes it.
+ - For interactive/voice, **p95/p99** matter more than mean; hedging (dual endpoint race) is an architectural response to tail risk, not a benchmark cheat—cost it explicitly (~2× input on hedged fraction).
+
+ **Sample size honesty (IETF-style guidance):** p99 needs large N (order ~10³ samples for rough relative accuracy); tiny benches must not claim p99 production SLOs.
+
+ ---
+
+ ## Workflow vs model-call latency
+
+ For product decisions, measure **workflow wall-clock**:
+
+ ```text
+ user_submit → retrieve/tools/guards → model (TTFT…E2E) → validate/repair → accept
+ ```
+
+ Attribute spans:
+
+ | Span | Typical owners |
+ | :--- | :--- |
+ | Retrieval / rerank | `@rag-retrieval-audit` |
+ | Tool round-trips | agent graph |
+ | Validation retries | `@llm-json-contract-check` |
+ | Model TTFT/TPOT | provider/serving |
+ | Human-perceived | UX (streaming indicators reset abandonment clocks) |
+
+ Optimizing only decode TPS while RAG prefill and tools dominate e2e is cargo-cult performance work.
+
+ ---
+
## Procedure
- Measure wall-clock latency around the complete workflow, including retrieval, tools, retries, and queueing. Separate time to first output from time to completion when streaming. Use provider-reported usage where available.
+ ### 1. Preflight
- ## Checks
+ - Define population (prod log sample vs synthetic fixed I/O).
+ - Set quality floor (pass rate / P0 gate).
+ - Set SLOs (e.g. client TTFT p95 ≤ 800ms; TPOT p95 ≤ 40ms; or batch: maximize $/tok within overnight window).
+ - Budget ($) and max duration.
+ - Fetch **live** pricing → pricing manifest.
+ - Choose cache regime(s) and concurrency sweep plan.
- Track input, cached input, output, and other separately billed usage using the provider's documented units. Include failed and retried requests in totals. Mark unavailable usage as unknown instead of estimating it silently.
+ ### 2. Instrument each attempt
- ## Failure Handling
+ Record per request (sanitized):
- Report counts and latency percentiles with the measurement method and sample size. Separate cold and warm runs. Compare only configurations that meet the agreed quality floor and do not extrapolate small synthetic tests as production guarantees.
+ ```json
+ {
+ "case_id": "c-17",
+ "attempt": 1,
+ "ok_quality": true,
+ "error_type": null,
+ "s_client": 0.0,
+ "f_first_usable_token": 0.42,
+ "e_last_token": 1.88,
+ "ttft_ms": 420,
+ "tpot_ms": 28.5,
+ "usage": {
+ "input": 3500,
+ "cached_read": 3200,
+ "cache_write": 0,
+ "output": 180,
+ "reasoning_output_subset": 40
+ },
+ "cost_usd": null,
+ "cache_regime": "warm",
+ "concurrency_bin": 16,
+ "model": "…",
+ "pricing_retrieved_at": "…"
+ }
+ ```
- ## Deliverable
+ ### 3. Analyze
- Deliver raw sanitized measurements, pricing references, cost arithmetic, and configuration. A faster response that fails acceptance checks is not an optimization win.
+ - Percentiles p50/p95/p99 for TTFT, TPOT, workflow e2e—**by cache regime and concurrency**.
+ - Cost totals and **$/accepted**.
+ - Goodput @ declared SLO.
+ - Error/retry rates.
+ - Reject cross-regime averages that hide cold starts.
+ ### 4. Decide
+
+ Only compare configs that clear the quality floor. Prefer:
+
+ 1. Higher goodput at equal or lower $/accepted, or
+ 2. Lower latency percentiles **without** quality loss, or
+ 3. Explicit Pareto: “+12% cost for −40% p99 TTFT” with product sign-off.
+
+ Deliver raw measurements + config + pricing refs + arithmetic. Label extrapolation limits.
+
+ ---
+
+ ## Production example: percentile + cost rollup
+
+ ```python
+ """Cost/latency rollup with TTFT, TPOT, and dated pricing."""
+ from __future__ import annotations
+
+ import json
+ import math
+ from dataclasses import dataclass
+ from pathlib import Path
+ from typing import Any, Sequence
+
+
+ def percentile(sorted_vals: Sequence[float], p: float) -> float:
+ """Nearest-rank percentile; p in [0, 100]. Empty → nan."""
+ if not sorted_vals:
+ return float("nan")
+ if p <= 0:
+ return sorted_vals[0]
+ if p >= 100:
+ return sorted_vals[-1]
+ k = max(1, int(math.ceil(p / 100.0 * len(sorted_vals))))
+ return sorted_vals[k - 1]
+
+
+ @dataclass(frozen=True)
+ class Rates:
+ """USD per 1M tokens. None means unknown—do not invent."""
+ input_uncached: float | None
+ input_cached_read: float | None
+ input_cache_write: float | None
+ output: float | None
+
+
+ def cost_usd(usage: dict[str, int], rates: Rates) -> float | None:
+ need = {
+ "input_uncached": rates.input_uncached,
+ "input_cached_read": rates.input_cached_read,
+ "input_cache_write": rates.input_cache_write,
+ "output": rates.output,
+ }
+ if any(v is None for v in need.values()):
+ return None
+ inp = usage.get("input", 0)
+ cread = usage.get("cached_read", 0)
+ cwrite = usage.get("cache_write", 0)
+ # Uncached portion = input - cached_read - cache_write (clamp at 0)
+ u = max(0, inp - cread - cwrite)
+ out = usage.get("output", 0)
+ return (
+ u * rates.input_uncached
+ + cread * rates.input_cached_read
+ + cwrite * rates.input_cache_write
+ + out * rates.output
+ ) / 1_000_000.0
+
+
+ def tpot_ms(ttft_ms: float, e2e_ms: float, output_tokens: int) -> float:
+ if output_tokens < 2:
+ return float("nan")
+ return (e2e_ms - ttft_ms) / (output_tokens - 1)
+
+
+ def summarize(rows: list[dict[str, Any]], rates: Rates, slo_ttft_ms: float) -> dict[str, Any]:
+ ttfts = sorted(r["ttft_ms"] for r in rows if r.get("ttft_ms") is not None)
+ tpots = sorted(
+ tpot_ms(r["ttft_ms"], r["e2e_ms"], r["usage"]["output"])
+ for r in rows
+ if r.get("ttft_ms") is not None and r.get("e2e_ms") is not None
+ )
+ tpots = [x for x in tpots if x == x]
+
+ costs = []
+ unknown_cost = 0
+ for r in rows:
+ c = cost_usd(r["usage"], rates)
+ if c is None:
+ unknown_cost += 1
+ else:
+ costs.append(c)
+
+ accepted = [r for r in rows if r.get("ok_quality")]
+ good = [r for r in accepted if r["ttft_ms"] <= slo_ttft_ms]
+
+ return {
+ "n": len(rows),
+ "ttft_p50": percentile(ttfts, 50),
+ "ttft_p95": percentile(ttfts, 95),
+ "ttft_p99": percentile(ttfts, 99),
+ "tpot_p50": percentile(tpots, 50),
+ "tpot_p95": percentile(tpots, 95),
+ "cost_total_usd": sum(costs) if costs else None,
+ "cost_unknown_rows": unknown_cost,
+ "cost_per_accepted_usd": (sum(costs) / len(accepted)) if costs and accepted else None,
+ "goodput_fraction_ttft_slo": (len(good) / len(rows)) if rows else float("nan"),
+ "quality_pass_rate": (len(accepted) / len(rows)) if rows else float("nan"),
+ }
+
+
+ # Load JSONL attempts, attach pricing from manifest, print summarize().
+ ```
+
+ ### Minimal experiment matrix (start here)
+
+ | Cell | Purpose |
+ | :--- | :--- |
+ | Fixed 0.5k/0.5k I/O, c=1, cache-cold | Pure model responsiveness |
+ | Same, cache-warm | Prefix-cache benefit |
+ | Prod-length distribution, c≈p95 load | Capacity reality |
+ | Full workflow e2e | Product truth |
+ | + quality floor filter | Economic truth |
+
+ ---
+
+ ## Technical troubleshooting matrix
+
+ | Signature | Deep cause | What to do |
+ | :--- | :--- | :--- |
+ | Amazing TTFT in bench, bad in prod | Accidental prefix cache / short prompts | Force unique prefixes; match prod lengths; report hit rate |
+ | p50 fine, p99 awful | Queueing, throttling, noisy neighbors | Correlate spikes with RPS vs wall clock; raise replicas or shed load; consider hedge |
+ | High TPS, unhappy users | Optimizing aggregate decode, ignoring TTFT SLO | Track goodput; plot latency vs offered load |
+ | “Cheaper model” costs more | More output/reasoning/retries | Optimize **$/accepted** and output tokens, not $/MTok sticker |
+ | Empty/fast failures | Reasoning burned `max_tokens` | Raise output budget; classify as errors not wins |
+ | Serverless look slow | Cold start (pull/CUDA/weights) | Separate cold-start study; warm pool for interactive |
+ | TTFT ≈ queue | Saturation | Capacity problem, not “slow model” |
+ | Cross-provider bakeoff chaos | Different clocks/tokenizers/cache | Comparability gate; cannot-normalize if mismatched |
+ | Client harness overhead | Sync client dominating | Async load gen; note NVIDIA warnings on client overhead |
+ | Cost unknown | Streaming without usage | Enable usage on stream; else mark unknown |
+
+ ---
+
+ ## Best practices
+
+ 1. **Quality floor first**—then latency—then cost among survivors.
+ 2. Report **percentiles**, never mean-only for SLOs.
+ 3. Separate **cold/warm** cache and **model-only vs workflow** e2e.
+ 4. Pin and date **pricing**; re-pull when claiming savings.
+ 5. Include **retries, failures, tool fees** in cost.
+ 6. Name every TPS/TTFT definition in the report title.
+ 7. For p99 SLOs, collect enough samples or refuse the claim.
+ 8. Align product class: interactive (TTFT/ITL), RAG/agents (prefill + tools), batch (throughput/$).
+ 9. Prefer provider usage + OTel GenAI attributes for long-term dashboards.
+ 10. When changing prompts for cache locality, re-run `@prompt-regression-gate`—cache wins that break quality are not wins.
+
+ ---
+
+ ## Limitations
+
+ - Third-party latency leaderboards rarely match your prompt length, tools, and cache mix.
+ - This skill does not replace provider SLAs or capacity postmortems.
+ - Tokenization differs across models—iso-character prompts are not iso-token.
+ - Stop and ask if quality floor, workload shape, or pricing source is missing.
+
+ ---
+
+ ## Related skills
+
+ - `@prompt-regression-gate` — quality floor / paired go-no-go
+ - `@ai-evaluation-dataset` — representative case population
+ - `@rag-retrieval-audit` — retrieval latency vs generation
+ - `@llm-json-contract-check` — repair retries inflate cost/latency
+ - `@llm-cost-latency-benchmark` — this skill
+ - `@vllm` / provider skills — serving knobs (batching, prefix cache)
+
+ ---
+
+ ## Agent Operational Directive
+
+ > **MANDATORY**: Treat every cost/latency claim as a workload+clock+cache+concurrency+pricing+quality tuple. Measure client TTFT and TPOT/ITL with named formulas. Bill all token classes and failures honestly; date the pricing source. Never call a quality regression an optimization. Prefer goodput and $/accepted over vanity TPS. If comparability fields are missing, say **cannot-normalize**.
+
+ ---
+
+ ## Source anchors (research)
+
+ - Anyscale / vLLM metric definitions (TTFT, TPOT, ITL, goodput)
+ - NVIDIA / Artificial Analysis / GPUSmith: clock boundaries, TPS ambiguity, cache disclosure
+ - OpenAI: prompt caching billable categories; reasoning tokens billed as output; dated pricing page
+ - OpenTelemetry GenAI: `time_to_first_chunk`, token usage (+ cache attributes), billed vs consumed
+ - IETF draft LLM benchmarking methodology: percentiles, sample sizes, SLO capacity search
+ - r/LLMDevs & r/LocalLLaMA: p99 tails, hedging, prefill-bound agents, cold starts, false TTFT from first chunk / cache reuse