git:20260910.55adacd to git:20260911.9f92063
14 added, 59 removed. Audit A to A.
---
name: vllm-ascend-correctness-validation
description: Plan, execute, normalize, and compare vLLM Ascend inference correctness across baseline and candidate code states, eager and graph modes, offline generate or chat, online chat completions, and AISBench task metrics. Use for accuracy validation, token-output comparison, graph-versus-eager checks, deterministic regression testing, or failure classification. Do not use to root-cause an already reproduced graph-only or isolated-operator failure, or for throughput benchmarking, HBM attribution, or profiling-only analysis.
---
- # vLLM Ascend Correctness Validation
-
- Produce traceable correctness evidence instead of treating a successful request as proof of correctness.
-
- ## Workflow
-
- 1. Define cases and the smallest affected validation matrix.
- 2. Create independent baseline and candidate code states or eager and graph states.
- 3. Bind each actual source state with `vaws_session(sources=...)`; coordinator prepares it for the managed run.
- 4. For offline cases, run `scripts/remote_correctness_harness.py` inside the remote NPU container. The harness prioritizes the materialized `vllm/` and `vllm-ascend/` source roots and propagates them through `PYTHONPATH`, so launching from `/vllm-workspace` or spawning a `python -m` child cannot resolve the outer repository directory as a false `vllm` namespace package.
- 5. For online cases, use `vllm-ascend-serving`, then run the same harness in `online-chat` mode.
- 6. Use `scripts/correctness_run.py init` to create the run directory and Run Manifest v1. Pass `--parent-run-id` for PR evidence, and declare any intentional non-code difference between the two states with `--allowed-difference KEY` (for example `engine_args.enforce_eager` for eager versus graph).
- 7. Use `scripts/correctness_run.py compare` to normalize the evidence into a classification and report. It first checks that both results carry an `execution` block and differ only in declared keys, then consumes an observational comparability certificate. An undeclared difference, a declaration/observation mismatch, or a `not-comparable` certificate aborts the comparison instead of being reported as a code regression. Two empty answers are not agreement. Null or blank identity values are unknown, not observations.
- 8. Route failures:
- - eager pass and graph fail: `vllm-ascend-graph-debug`;
- - multi-rank hang or inconsistent rank metadata: `vllm-ascend-distributed-debug` when available;
- - performance-only change: `vllm-ascend-performance-regression` when available;
- - task metric execution: use the bundled AISBench adapter.
-
- Do not run the full execution-mode, parallelism, and feature Cartesian product. Select cases from the code impact and `.agents/knowledge/`; record omitted combinations as risks.
-
- ## Determinism
-
- For exact token or text comparison:
-
- - set `temperature=0`;
- - set an explicit seed;
- - keep prompts, messages, chat template, max tokens, model weights, tokenizer, parallel topology, and feature flags identical;
- - repeat each case at least twice when nondeterminism is suspected;
- - never compare baseline and candidate results produced from different case files.
-
- Use task-level metrics when exact output is not an appropriate acceptance condition. Use numeric tolerances only for explicitly captured logits, hidden states, KV samples, or other numeric evidence.
-
- ## Result classes
+ # vllm-ascend-correctness-validation
- The comparator emits exactly one primary class per case:
+ Compare inference outputs across code or execution configurations with explicit comparability and numerical criteria.
- - `exact_match`
- - `token_divergence`
- - `numerical_difference_within_tolerance`
- - `numerical_regression`
- - `task_metric_regression`
- - `flaky_or_nondeterministic`
- - `infrastructure_failure`
- - `unsupported_combination`
+ Select deterministic prompts or token IDs, sampling, model and topology that exercise the change. Token equality and dataset task metrics answer different questions. Declare only the intended varying dimensions with --allowed-difference.
- Do not relabel infrastructure failures as correctness regressions. Do not treat unsupported or untested combinations as passing.
+ ## Agent entry
- ## Structured entry points
+ Run from the repository root using the platform's Python launcher. The workspace
+ selects its installed platform environment automatically.
- - `scripts/correctness_run.py`: initialize a run, compare normalized baseline and candidate outputs, write `comparison.json`, `report.md`, `reproduction.sh`, and update Run Manifest v1.
- - `scripts/remote_correctness_harness.py`: execute offline generate, offline chat, or online chat cases and write the normalized result contract, including the `execution` identity (`engine_args`, model, service, case digest) that decided the run.
- - `scripts/aisbench_adapter.py`: prepare an AISBench accuracy command/config and normalize task metrics into the correctness result contract; `normalize` requires `--execution` to declare the benchmarked service.
+ ```text
+ python .agents/skills/vllm-ascend-correctness-validation/scripts/correctness_run.py --cases cases.json --baseline baseline.json --candidate candidate.json
+ ```
- Read as needed:
+ The remote_correctness_harness.py payload captures offline runtime observations from the managed execution. Online/AISBench results use the server execution reference through aisbench_adapter.py. The comparison derives metadata from actual outputs, emits its certificate and report, and reports missing identity as inconclusive.
- - [Behavior contract](references/behavior.md) for case, result, comparison, and artifact schemas.
- - [Command recipes](references/command-recipes.md) for local control-plane and remote harness examples.
- - [AISBench adapter](references/aisbench.md) before preparing or importing AISBench results.
- - [Acceptance](references/acceptance.md) before marking a run passed.
+ Route an eager-passes/graph-fails reproduction to graph-debug, a rank-dependent failure to distributed-debug, and a reduced operator failure to operator-debug.
- ## Boundaries
+ Read the relevant detail only when needed:
- - Keep runtime state under `.vaws-local/correctness/`.
- - Keep progress on stderr and the final machine-readable payload on stdout.
- - Never put passwords, tokens, API keys, or credentials in configs or manifests.
- - Run torch and torch_npu code only in the remote Ascend container.
- - Treat the normalized output file, not mixed runtime stdout, as the comparison source.
+ - [behavior](references/behavior.md)
+ - [aisbench](references/aisbench.md)