git:20260922.5e8f509 to git:20261019.b13b857

+ NVIDIA AIPerf — vendor-neutral generative-AI inference benchmarking (genai-perf successor). `aiperf profile`: concurrency / request-rate / fixed-schedule trace replay / user-centric / multi-run confidence, 19 endpoint types (chat, completions, embeddings, rankings, responses, messages, audio transcription, image-gen/edit, video-gen, NIM, HF-TGI, raw, template), 13 custom dataset formats (single/multi-turn; mooncake, bailian, burst_gpt, baseten, weka, tracelab traces; random_pool, dag_jsonl, raw_payload, inputs_json, sagemaker_data_capture) plus SPEED-Bench, 40+ public datasets, goodput SLOs, GPU + Prometheus telemetry, plot/analyze/synthesize/kube subcommands, plugins, reasoning TTFT/TTFO split.

+ Use for defensible latency / throughput / goodput numbers against any OpenAI-compatible server (vLLM, SGLang, TensorRT-LLM, NVIDIA Dynamo, NIM, Triton, HF TGI, Ollama), and for extending AIPerf with plugins. AIPerf is the `genai-perf` successor; its numbers differ for reasoning models (pitfall 2). For a vLLM-only target driven by the in-tree `vllm bench` toolchain, use the `vllm-benchmarking` skill; use AIPerf when the target is non-vLLM or the workload is trace-driven, multi-modal, multi-server, or needs goodput.

+ ## Version-gated capabilities — check `aiperf --version` before using

+ | Need | Use | Needs |

+ |---|---|---|

+ | Sweep or search instead of a shell loop | List flags (`--concurrency 10,50,100`), `--search-recipe <name>`, YAML `sweep:` / Bayesian search; multi-tier SLO search `--search-sla-tier` | recipes/BO v0.10.0; tiers v0.11.0 |

+ | Several warmup/profiling phases, or adaptive scaling, in one run | Phases and scale variables in the YAML config | v0.12.0 |

+ | Accuracy alongside perf | `--accuracy-benchmark` (see `references/cli.md`) | AIME v0.9.0; HellaSwag/BBH/AIME24-25 v0.10.0; MATH-500/GPQA/LCB/GSM8K v0.11.0 |

+ | Export to MLflow / W&B / OTel | `--mlflow-*`, `--wandb-*`, `--otel-url` + pip extra (`references/cli.md` → External exporters) | MLflow/OTel v0.9.0; W&B v0.11.0 |

+ | AMD GPU telemetry | `amdsmi` collector | v0.9.0 |

+ | Latency without the network path | `network_adjusted_*` metrics | v0.11.0 |

+ | Anthropic Messages API | `--endpoint-type messages` (`/v1/messages`); wire format: `messages-api` skill | v0.12.0 |

+ | Per-request spec-decode acceptance | vLLM adapter; pair with vLLM `--per-request-spec-decode-metrics` (`vllm-speculative-decoding`) | v0.12.0 |

+ | Agentic / DAG replay | `exgentic*`, `baseten_trace`, `weka_*`, `dag_jsonl`, `tracelab` (v0.13.0) | v0.12.0 |

+ | Inspect a run | `aiperf analyze swim-lane` / `turn-messages`; `aiperf chat` for a hand check | v0.12.0 |

+ | Run inside Kubernetes | `aiperf kube …` (beta) | v0.13.0 |

+ | Correct ITL when the first chunk bundles tokens | `--per-chunk-usage` (with `--use-server-token-count`; vLLM / TRT-LLM only) | v0.13.0 |

+ | Reset prefix cache / profile the server per run | `--reset-kv-cache` (`/reset_prefix_cache`), `--server-profiler` (`/start_profile`, `/stop_profile`) | v0.13.0 |

+ | Production system prompt, `vllm bench`/sglang-style random lengths, cache-bust markers, ASR | `--system-prompt-file`, `--random-range-ratio` + `--random-corpus-style`, `--cache-bust`, `--endpoint-type audio_transcription` | v0.13.0 |

+ Behaviour changes that make results non-comparable across versions:

+ - **v0.11.0 warmup fix** — warmup no longer leaks prefix-cache reuse into profiling. Baselines from ≤v0.10.x with warmup on are not comparable; re-run them.

+ - **v0.13.0 sends `X-Session-Affinity`** (the session's `X-Correlation-ID` value) on every request. A router that honours it pins a session's turns to one replica, so multi-turn results against such a router differ from v0.12.0. Disable: `AIPERF_HTTP_X_SESSION_AFFINITY_FROM_CORRELATION_ID=false`. Router-specific affinity is opt-in: `AIPERF_HTTP_X_DYNAMO_SESSION_ID_FROM_CORRELATION_ID=true` (Dynamo `--router-session-affinity-ttl-secs`), `AIPERF_HTTP_X_SMG_ROUTING_KEY_FROM_CORRELATION_ID=true` (SGLang Model Gateway manual routing).

+ - **Stable on PyPI:** v0.13.0 (2026-09-24), `requires-python >=3.11,<3.14`. `pip install aiperf`. A nightly wheel is published alongside as `aiperf-nightly` (v0.10.0+).

+ - **Repo `main`** at https://github.com/ai-dynamo/aiperf: post-v0.13.0.

+ | "Scale a real trace for KV-cache stress" | `aiperf profile ... --custom-dataset-type mooncake_trace --synthesis-*` | Prefix length / root count / speedup / ISL-OSL caps are `profile` flags, applied in memory at load. `aiperf synthesize` does not take a trace. |

+ | "Generate a synthetic agentic-coding dataset" | `aiperf synthesize agentic-code [--num-sessions N --output DIR]` | The only `synthesize` target. |

+ | "Sanity-check an endpoint by hand before a run" | `aiperf chat --model M --url U [--quick "hi"]` | Per-reply TTFT / TPS / latency. Not a benchmark. |

+ | "Why did this run behave like that?" | `aiperf analyze swim-lane --run-dirs <dir>` / `analyze turn-messages` | Per-session timeline PNG (+ `--html` viewer); per-turn input-message HTML viewer (needs `--export-level raw`). |

+ | "Start from a YAML config instead of flags" | `aiperf config init --list` → `config init -t <name> -o run.yaml` → `config validate --config-file run.yaml` → `aiperf profile -f run.yaml` | Bundled templates (e.g. `minimal`, `goodput_slo`). |

+ | "Run the benchmark inside Kubernetes" | `aiperf kube init` (starter config) → `kube profile` / `kube sweep` → `kube results` | v0.13.0+, beta. `kube preflight` / `kube debug` when it fails. `aiperf service` and `aiperf proxy` are pod internals — do not call them directly. |

+ | "Validate an export file" | `aiperf validate --target <format> --input <file>` | Sanity check that one export file is well-formed. |

+ aiperf profile \

+ --model Qwen/Qwen3-0.6B --url http://endpoint:8000 \

+ --endpoint-type chat --streaming \

+ --tokenizer Qwen/Qwen3-0.6B \

+ --concurrency 10,50,100,200,500 --request-count 1000 \

+ --isl 1000 --osl 500 \

+ --num-dataset-entries 1000 --random-seed 42 \

+ --artifact-dir artifacts/pareto

+ aiperf plot --paths artifacts/pareto --dashboard # finds every point's run dir recursively

+ A comma-separated `--concurrency` (also `--request-rate`, `--isl`, `--osl`; `--sweep-type grid|zip`) runs the sweep natively and gives each point its own seed (`base + index`) by default — do not write a shell loop with one fixed seed. To answer "highest throughput that still meets the SLA" directly, use a recipe instead of a grid: `--search-recipe max-throughput-ttft-sla --ttft-sla-ms 200` (others: `max-goodput-under-slo`, `concurrency-ramp`, `pareto-sweep`; `aiperf plugins search_recipe` lists them).

+ Same seed list, same hardware, same tokenizer for config A and config B. **Synthetic prompts are a 100-entry pool by default (`--num-dataset-entries`), reused across requests, and a given `--random-seed` regenerates the identical pool.** Against a prefix-caching server (vLLM APC, SGLang radix) a 1000-request run with the default pool, or a second run with the same seed, is served from cache — TTFT and throughput become cache measurements. So: size `--num-dataset-entries` ≥ requests + warmup, keep a distinct seed per sweep point (the native sweep does; `--parameter-sweep-same-seed` turns it off), and restart the server (or `--reset-kv-cache`, v0.13.0+) before switching config. Multi-run confidence (`--num-profile-runs 3 --confidence-level 0.95`) yields CI bands when each run is short enough that re-running 3× is feasible; its trials share one seed by default, so on a prefix-caching server add `--vary-seed-per-trial` or trials 2..N measure a warm cache.

+

+ 5. **`--isl-block-size` and the Mooncake/server block-size relationship — read this carefully.** Mooncake traces encode `hash_ids` at **512-token granularity by design** — the trace was generated with that block size and the IDs only make sense at 512. Do **not** lower `--isl-block-size` to "match the server" (e.g. 64 for SGLang `--page-size 64`); doing so makes aiperf reconstruct prompts ~8× shorter than the trace intends and corrupts the ISL distribution. What matters is whether **512 is a multiple of the server's block size** — if yes (e.g. SGLang 64 → 512/64=8, vLLM 16 → 512/16=32), each Mooncake block aligns with N server pages and prefix-cache reuse works fine. If the server block size doesn't divide 512, that's the broken case. Run `aiperf analyze-trace --input-file trace.jsonl --block-size 512` to confirm what the trace expects. The rule generalises: `--isl-block-size` = the block size the trace was **recorded** at, never the server's (loader defaults: 512 `mooncake_trace`, 16 `bailian_trace`; no effect on `baseten_trace`). Results from a run with a lowered block size measured a different, shorter workload — discard them and re-run; never keep them as a baseline.

+ 8. **`--export-level raw` on a long run.** Writes every request/response payload to disk. Fine for 1k requests, catastrophic for 1M. Default is `records` (per-request metrics, no payloads); only escalate when actually debugging. On v0.13.0 a raw export that takes more than ~20 s ends in exit 1 with `RequiredServiceReaped` although every request completed — the heartbeat watchdog reaps services during export; set `AIPERF_SERVICE_HEARTBEAT_MISSED_THRESHOLD=24`. Exit 0 does not prove the raw file is complete: a record processor force-killed at shutdown truncates it. Check `wc -l profile_export_raw.jsonl` against the request count before recomputing metrics from it.

+ 9. **Async video generation polling.** `endpoint-type video_generation` polls `/v1/videos/{job_id}` until the job completes. Tune `AIPERF_HTTP_VIDEO_POLL_INTERVAL` (default 0.1 s) — faster polling burns server resources, slower polling inflates request latency. With `--download-video-content`, request latency includes the bytes download.

+ 10. **`--goodput` is per-request, not per-percentile.** The flag asks "is this individual request below the threshold?" and reports `good_request_count / benchmark_duration` as `goodput`. There is no built-in `p99_time_to_first_token:400` syntax. To express "p99 TTFT must be ≤ 400 ms", set `--goodput "time_to_first_token:400"` and check `goodput / request_throughput ≥ 0.99` post-run — that ratio is the fraction of requests meeting all SLOs, which for a single threshold is exactly the "X% under the budget" measure operators usually want. Combine multiple thresholds and the same ratio still works as joint compliance.

+ ## Done = every row checked before a number leaves the run

+ | Check | Evidence in `profile_export_aiperf.json` / the run |

+ |---|---|

+ | All requests ran | `request_count` equals the requested count (or sessions × turns); `error_request_count` is 0 or each error is explained. Exit 1 with only `RequiredServiceReaped` on v0.13.0 is pitfall 8, not a failed run |

+ | The workload is the intended one | mean `input_sequence_length` / `output_sequence_length` match `--isl`/`--osl`, or the trace's `analyze-trace` figures (pitfall 5) |

+ | Cache state is known | pool ≥ requests, distinct seed per point and trial, server cache reset or restarted between configs (Methodology B) |

+ | Reasoning split works (reasoning models) | `reasoning_token_count` > 0, TTFO ≠ TTFT (pitfall 2) |

+ | Run is reproducible | `--tokenizer` explicit and `--tokenizer-revision` pinned; `aiperf --version`, server image and flags recorded beside the artifact dir |

+ | SLO verdict is per-percentile where the SLO says so | goodput ratio or exported p95/p99 values, not the raw `goodput` number (pitfall 10) |

+

+ --num-dataset-entries 2000 --vary-seed-per-trial \