git:20260910.55adacd to git:20260911.9f92063
12 added, 40 removed. Audit A to A.
---
name: vllm-ascend-performance-regression
description: Plan, record, and analyze controlled baseline-versus-candidate vLLM Ascend serving performance experiments with isolated sessions, identical non-code configuration, alternating A/B order, warmup exclusion, variance and outlier reporting, and metric-specific regression thresholds. Use for throughput, TTFT, TPOT, ITL, acceptance-rate, startup-time, or HBM regression checks. Do not use for correctness, single-state measurement, HBM component attribution, or profiling root-cause analysis.
---
- # vLLM Ascend Performance Regression
+ # vllm-ascend-performance-regression
- Wrap `vllm-ascend-benchmark` with a controlled two-state experiment.
+ Run or analyze a controlled baseline-versus-candidate serving experiment and apply metric-specific regression thresholds.
- ## Workflow
+ Choose the workload and metrics that reflect the user-visible change. Set threshold and direction per metric. Keep non-code conditions comparable and inspect variance, outliers and failed requests before attributing a delta to code.
- 1. Create independent baseline and candidate worktrees and bind the actual sources through coordinator task context.
- 2. Use the same machine allocation policy, NPU count, model and weight hash, environment, topology, Serving arguments, Benchmark arguments, dataset, request rate, and concurrency.
- 3. Put all non-code conditions in the experiment `shared` object. `plan` requires `machine`, `npu_devices`, `model`, `environment`, `topology` (with `tp` and `dp`), `serve_args`, `bench_args`, `dataset`, `max_concurrency`, and `request_rate`; a `shared` object without them cannot produce a parity certificate.
- 4. Run `scripts/performance_regression.py plan`; set `parent_run_id` in the config when the experiment is evidence for a change-validation plan.
- 5. Follow `schedule.json` exactly. For each state, let coordinator prepare its bound sources, start or confirm its service, then call `vllm-ascend-benchmark`.
- 6. Normalize each raw Benchmark result with `normalize`, then call `record`.
- 7. Run `analyze` only after the schedule is complete.
- 8. If the result is failed or inconclusive and operator timing is needed, recommend profiling collection; do not collect heavy profiles automatically.
+ ## Agent entry
- For three measurements the alternating sequence is:
+ Run from the repository root using the platform's Python launcher. The workspace
+ selects its installed platform environment automatically.
```text
- baseline warmup
- candidate warmup
- baseline 1
- candidate 1
- candidate 2
- baseline 2
- baseline 3
- candidate 3
+ python .agents/skills/vllm-ascend-performance-regression/scripts/performance_regression.py --config experiment.json
```
- ## Entry point
-
- `scripts/performance_regression.py` provides:
-
- - `plan`: validate that `shared` declares every parity condition, generate the alternating schedule, write a `parity-check.json` that names what it did and did not verify, and create Run Manifest v1;
- - `normalize`: convert one single-run or aggregated Benchmark result into the measurement contract;
- - `record`: accept the next normalized measurement only when its state, phase, ordinal, and inlined `shared` match the schedule;
- - `analyze`: consume an observational comparability certificate built from every measure-phase observation, then exclude warmups, report mean, sample deviation, coefficient of variation, outliers, relative change, and threshold verdict. `passed` requires the certificate.
+ The business config names baseline.sources and candidate.sources (actual vllm and vllm-ascend worktrees), benchmark options, runs, warmups and thresholds. The collector binds each source, waits for its managed service, warms each launch, alternates A/B order, records runtime observations, and releases owned executions. --results accepts existing measurement files for report-only use. Missing runtime evidence yields an inconclusive report.
- Read:
+ For a single-state throughput measurement use benchmark. For root-cause timing attribution use profiling collection and analysis.
- - [Behavior contract](references/behavior.md) for config, schedule, measurement, statistics, and status semantics.
- - [Command recipes](references/command-recipes.md) for the full lifecycle.
- - [Acceptance](references/acceptance.md) before claiming a regression or pass.
+ Read the relevant detail only when needed:
- ## Rules
+ - [behavior](references/behavior.md)
- - Never compare measurements whose `shared` objects differ.
- - Never run all baseline measurements before all candidate measurements.
- - Do not include warmups in statistics.
- - Preserve raw values even when configured to exclude detected outliers from the decision set.
- - Return `inconclusive` when required metrics are missing, too few decision values remain, or observed variation exceeds `max_cv`.
- - Use metric direction explicitly: higher is better or lower is better.
- - Keep experiment state under `.vaws-local/performance-regression/`.
+ - [Experiment input example](references/inputs.md)