---
name: ascend-profiling-collection
description: Collect one Ascend torch-profiler case end-to-end on a workspace-managed remote NPU container. Starts a profiled vLLM service, brackets a workload with /start_profile and /stop_profile, runs analyse() (db export by default), verifies the per-rank ascend_pytorch_profiler_*.db landed, and writes a manifest the analysis skill can consume. Use for requests like "采集 profiling", "torch profiler 跑一个 case", "采一份 profile 出来", "采 profiling 给我分析". Do not use for pure performance benchmarking, HBM/memory profiling, or for analysing already-collected profiling data (that is the analysis skill's job).
---

# Ascend Profiling Collection

Collect one torch-profiler case on a workspace-managed remote Ascend NPU container.

Remote substrate rule: use remote-dev companion tools (`remote_*` MCP tools,
launched via `uv run remote-dev` or MCP) for ad hoc remote
read/edit/bash/search/patch work around profile setup and output inspection.
Use this skill for the domain collection workflow and keep its scripts as the
compatibility backend for managed VAWS sessions.

This skill is **only** about collection: start a profiled service, bracket a workload with `/start_profile` and `/stop_profile`, run `torch_npu.profiler.profiler.analyse(...)`, verify the device-side data actually landed, and write a manifest. By default analyse() runs with `export_type=Constant.Db`, so each rank produces only `ASCEND_PROFILER_OUTPUT/ascend_pytorch_profiler_*.db` — the analysis skill rebuilds the kernel_details event stream directly from the db, and the multi-GB text exports (`trace_view.json`, CSVs) are skipped. Interpreting the data is a separate concern owned by the analysis skill.

## Use this skill when

- the user asks to collect / capture an Ascend torch-profiler trace for a specific config
- another skill (the analysis skill) needs a fresh profiling root with verified outputs
- the user wants to reproduce an existing root with a new model / mode / TP / DP

## Do not use this skill when

- the task is performance benchmarking only — use `vllm-ascend-benchmark`
- the task is HBM / memory analysis — use `ascend-memory-profiling`
- the task is analysing an already-collected profiling root (no need to re-collect)
- the machine is not yet ready in inventory — use `machine-management`

## Boundary with other skills

| Skill | Owns | This skill uses it for |
| --- | --- | --- |
| `vllm-ascend-serving` | Service lifecycle, `--profiler-config` passthrough | `serve_start.py` / `serve_stop.py` only; serving is **agnostic** to the profiler window |
| `remote-code-parity` | Local-to-container code sync | Implicit — invoked by `serve_start.py` |
| `vllm-ascend-benchmark` | `vllm bench serve` performance numbers | Not used; benchmark skill must not learn the profiler control plane |
| `ascend-memory-profiling` | HBM attribution via msprof | Independent; not invoked |

`/start_profile` and `/stop_profile` exist *because of* profiling, so the control-plane client lives here, not in `vllm-ascend-serving`.

## Critical rules

- The serving skill must remain profiling-agnostic. Never push profiler-window control or `analyse()` invocation into it.
- `--profiler-config` is the only profiling-related thing the serving skill knows about, and only because vLLM accepts it as an opaque blob.
- Run profiling **inside the remote container**. Never copy raw `*_ascend_pt` directories back to the local Mac.
- Hard-fail in three cases (all detailed in "Failure policy"):
  1. any rank's expected analyse output is missing after `analyse()` — the per-rank db in the default `--analyse-export db` mode, `kernel_details.csv` in text/both mode
  2. number of `*_ascend_pt` directories does not match `tp * (dp or 1)`
  3. workload was not real — follow-up request failed or benchmark wave fell below `--benchmark-success-threshold`
- Progress on `stderr` as `__VAWS_PROGRESS__=<json>`. Final manifest on `stdout` as one JSON object.
- Collection uses `--context-file` / `VAWS_CONTEXT_FILE` and `--execution-id` / `--service`. Do not guess a session from cwd.
- Local state lives under `.vaws-local/ascend-profiling-collection/runs/` for collection manifests; serving/parity state lives under the session namespace.

## Public entry point

```bash
python3 .agents/skills/ascend-profiling-collection/scripts/collect_torch_profile_case.py \
  [--context-file <path>] [--execution-id <id>] [--service vllm] \
  --model <remote-weight-path> \
  --served-model-name <name> \
  --tp <N> \
  --tag <stable-id> \
  --mode {enforce_eager|full_decode_only|piecewise_graph} \
  --request-kind {text|vl} \
  --benchmark-output-tokens <N> \
  [--dp <N>] \
  [--enable-expert-parallel] \
  [--speculative-tokens <N>] [--speculative-method <name>] \
  [--gpu-memory-utilization <f>] \
  [--max-model-len <N>] [--max-num-seqs <N>] [--max-num-batched-tokens <N>] \
  [--api-server-count <N>] \
  [--prompt-tokens <N>] [--followup-output-tokens <N>] \
  [--benchmark-total-requests <N>] [--benchmark-concurrency <N>] \
  [--benchmark-success-threshold <f>] \
  [--request-timeout <s>] [--profile-control-timeout <s>] [--health-timeout <s>] \
  [--torch-profiler-dir <relpath>] [--torch-profiler-with-stack] \
  [--analyse-export {db|text|both}] \
  [--archive-dir <remote-path>] \
  [--image-path <local-path>] [--image-height <px>] \
  [--skip-parity]
```

### Required parameters and why

Task identity is `--context-file` / `VAWS_CONTEXT_FILE`. A live service is `--execution-id` or `--service`.

The script intentionally has no Qwen-specific defaults. The agent must always pass:

| Required arg | Why |
| --- | --- |
| `--model` / `--served-model-name` | Different cases need different models, no safe default |
| `--tp` | Hardware shape; never assume it |
| `--mode` | The profile is meaningless without recording which graph mode produced it |
| `--request-kind` | Determines payload assembly (text vs VL) |
| `--benchmark-output-tokens` | Decode length is the dominant knob for what the trace looks like |
| `--tag` | Stable identifier folded into the run-dir name and manifest |

`--speculative-tokens 0` (the default) means "do not pass `--speculative-config` at all". Set to a positive integer to enable MTP/Eagle.

## Auxiliary entry points

The agent can call these directly if it already has a service running and only wants to flip the profiler window or re-run `analyse()` on an existing root.

### Flip the profiler window

```bash
# Start a profile window on a service that the serving skill already launched
python3 .agents/skills/ascend-profiling-collection/scripts/profile_control.py \
  [--execution-id <id> | --service vllm] --action start_profile [--timeout 900]

# Close it
python3 .agents/skills/ascend-profiling-collection/scripts/profile_control.py \
  [--execution-id <id> | --service vllm] --action stop_profile [--timeout 900]
```

The script reads the service port from the coordinator execution named by `--execution-id` or `--service`. A service must be running.

### Re-run `analyse()` on an existing root

```bash
python3 .agents/skills/ascend-profiling-collection/scripts/run_remote_analyse.py \
  [--execution-id <id> | --host <ip>] --profile-root <remote-path> \
  [--expected-ranks <N>] [--analyse-timeout <s>] [--analyse-parallelism <N>] \
  [--analyse-export {db|text|both}]
```

Discovers every `*_ascend_pt` under `--profile-root` and runs `torch_npu.profiler.profiler.analyse()` on each **concurrently on the container** (one SSH call, `xargs -P`; effective parallelism `min(rank_count, --analyse-parallelism)`, default 8 — per-rank analyse is CPU-bound and the containers have hundreds of cores, so TP16 no longer analyses 16 ranks serially). Each rank's stdout/stderr is captured in `<dir>/analyse_parallel.log`; per-rank exit codes are aggregated and any non-zero rank fails the run. `--analyse-timeout` is the overall wall-clock bound for the parallel phase (default 1800s, *not* multiplied by rank count; a remote `timeout(1)` wrapper kills stuck ranks).

`--analyse-export` selects the `export_type` passed to `analyse()`:

| Mode | analyse() call | Verified per rank |
| --- | --- | --- |
| `db` (default) | `analyse(dir, export_type=Constant.Db)` — skips all text exports | newest `ASCEND_PROFILER_OUTPUT/ascend_pytorch_profiler_*.db` exists and is non-empty |
| `text` | `analyse(dir, export_type=Constant.Text)` | `kernel_details.csv` + `trace_view.json` exist (historical contract) |
| `both` | `analyse(dir, export_type=[Constant.Text, Constant.Db])` | same as `text` |

The `Constant` import (`from torch_npu.profiler.analysis.prof_common_func._constant import Constant`) runs inside the generated per-rank payload so it always resolves against the container's torch_npu. On old CANN without db export support, analyse() raises (`is_support_export_db()`); that surfaces as a rank failure with the torch_npu error in `analyse_parallel.log` — use `--analyse-export text` on such hosts. Exits non-zero if any rank is incomplete.

Always pass `--expected-ranks` (typically `tp * (dp or 1)`) when running this against a fresh capture: without it a partial collection where some ranks never produced a directory looks "clean" because every directory that *did* land was complete. The orchestrator passes this automatically.

## Archiving to shared storage (`--archive-dir`)

Profiling outputs are far too large to pull back to the local Mac (a single dsv3.1 analysis once dragged back 2.2GB). When `--archive-dir <remote-path>` points at the shared-storage filesystem mounted on every managed host/container (e.g. `/mnt/weight/<user>/profiling/archives`), the orchestrator archives each rank's analyse outputs **on the container** (cp -r over the existing ssh channel, ranks serially) right after analyse+verify passes with all ranks ok:

```
<archive-dir>/<tag>_<started_at-compact-ts>/<rank-dir-basename>/
    ASCEND_PROFILER_OUTPUT/      # copied whole (db mode: carries the per-rank db; no csv to cherry-pick)
    profiler_info_*.json         # best-effort
    profiler_metadata.json       # best-effort
```

Because the rank-dir basename is kept as the subdirectory name, the archive root `<archive-dir>/<tag>_<ts>/` is itself a valid profiling root full of `*_ascend_pt` directories — feed it straight to the analysis skill's `--remote-profile-root` from **any** machine that mounts the same shared storage, no re-collection needed.

**Failure semantics**: archiving never overturns an already-ok collection. A copy failure records `archive_error` in the manifest (with the affected rank's `outputs.archived_path` left null and `archived: false`) plus a stderr warning; the run still exits 0 with `status: ok`.

Manifest additions (see the schema table below): top-level `archive_dir` (the actual archive root, null when not requested), `archived` (true only when every rank copied), `archive_error` (present only on failure), and per-rank `remote_profile_dirs[].outputs.archived_path`.

## Workflow

1. **Resolve the native task** (`--context-file`) and the live service (`--execution-id` / `--service`).
2. **Build serving args** — encode `--profiler-config` (always written) and the chosen graph mode.
3. **Start service** by shelling out to `serve_start.py`. Parity sync is automatic via the serving skill.
4. **Open SSH tunnel** to the service port so workload requests can be assembled locally (multimodal payloads need local image encoding).
5. **POST `/start_profile`** with the long control-plane timeout.
6. **Send benchmark wave** (concurrent chat-completions) followed by **one follow-up tail request**. The follow-up is intentionally short to capture a clean steady-state step.
7. **POST `/stop_profile`**.
8. **Stop service** by shelling out to `serve_stop.py`.
9. **Discover and analyse** every `*_ascend_pt` under `<runtime_dir>/<torch_profiler_dir>` via `run_remote_analyse.py` (parallel across ranks, see above).
10. **Verify outputs** per rank (db mode: non-empty per-rank db; text/both: `kernel_details.csv` + `trace_view.json`); classify each as `ok | partial | missing_kernel_details`.
11. **Archive (optional)** — with `--archive-dir`, copy each rank's `ASCEND_PROFILER_OUTPUT/` + profiler metadata to shared storage (see "Archiving to shared storage"); runs only when every rank verified ok, failures degrade to `archive_error` without flipping the run status.
12. **Write manifest** to `.vaws-local/ascend-profiling-collection/runs/<timestamp>_<tag>/manifest.json`.

## Failure policy

Accuracy beats coverage. The script exits non-zero (status `failed`) when **any** of:

- the service did not become `ready`
- `/start_profile` or `/stop_profile` returned non-2xx
- **workload was not real**: `workload_status.status != "ok"`, i.e. the
  follow-up request failed or the benchmark wave's success rate was below
  `--benchmark-success-threshold` (default 0.8). Without real traffic during
  the profile window the trace records nothing useful.
- **rank count mismatch**: number of `*_ascend_pt` directories `!= tp * (dp or 1)`
  → `analysis_status == "rank_count_mismatch"`. Some rank never dumped its
  profiler data; even if every directory that *did* land is complete, the
  topology is broken and downstream cross-rank analysis would be wrong.
- any rank's expected analyse output is missing after `analyse()` →
  `analysis_status == "missing_kernel_details"`. In the default
  `--analyse-export db` mode this means the rank's
  `ascend_pytorch_profiler_*.db` was not produced or is empty; in text/both
  mode it means `kernel_details.csv` is missing. The manifest's
  `expected_output_kind` field (`db` / `csv`) records which artifact the
  enum refers to; the enum set itself is unchanged so downstream gates keep
  working.

The last condition is the canonical "device-side data did not land" failure
documented in `references/behavior.md` ("Output verification"). Treat all of the above as
**re-collect required**, not as something the analysis skill can recover from.

If the orchestrator fails after `serve_start`, it always tries to stop the service (graceful, then `--force`) so no orphan vLLM process is left behind.

## Workspace knowledge hooks

Both hooks are local, advisory-only views over the workspace knowledge store (`.agents/knowledge/`, via `.agents/lib/vaws_knowledge_service.py` and the installed `vaws-knowledge` package); a missing/invalid knowledge dir degrades them to explicit empty arrays with a stderr progress note — collection is never blocked:

- **Preflight advisories**: before `serve_start`, the orchestrator queries `model-capabilities` / `parallelism-compatibility` / `known-failure-signatures` with `<served-model-name> tp<N> <mode>` (limit 3 per kind). Hits land in the manifest's `knowledge_advisories` array (`{entry_id, kind, summary, score}`) and are printed to stderr progress. Use them as hints (e.g. a model's verified TP/mode or a known container pitfall), not as gates.
- **Failure-gate enrichment**: when any hard-fail gate trips (service not ready, workload not real, `rank_count_mismatch`, `missing_kernel_details`), the observed error text is queried against `known-failure-signatures` (limit 3). Matches — including each entry's `resolution` — are written to `manifest.error.knowledge_matches`, and `manifest.error` becomes an object `{message, knowledge_matches}` instead of a bare string. The same object is the stdout failure JSON. No match → explicit empty array.

**沉淀提示**: when a collection failure turns into a confirmed new diagnosis (symptom + root cause + verified fix), capture it with `.agents/scripts/knowledge_capture.py` as a local candidate (`.vaws-local/knowledge/candidate/`, untracked); promotion into `.agents/knowledge/` happens only through reviewed project changes.

## Manifest schema

The manifest is the input contract for the analysis skill. Important fields:

| Field | Meaning |
| --- | --- |
| `schema_version` | Bumped when fields are renamed or removed |
| `tag`, `started_at`, `completed_at` / `failed_at` | Run identity |
| `machine`, `model`, `served_model_name`, `tp`, `dp` | Hardware / model identity |
| `mode`, `speculative_tokens`, `speculative_method`, `enable_expert_parallel`, `api_server_count` | What was profiled |
| `request_kind`, `prompt_tokens`, `benchmark_output_tokens`, `followup_output_tokens`, `benchmark_total_requests`, `benchmark_concurrency`, `benchmark_success_threshold` | What workload produced the trace and what success bar it had to clear |
| `expected_ranks` | `tp * (dp or 1)` — what `analyse()` was told to enforce |
| `torch_profiler_with_stack`, `torch_profiler_dir` | Profiler depth and on-disk location |
| `analyse_export` | `db` (default) / `text` / `both` — the `export_type` analyse() was run with |
| `serve_args` | Exact `serve_start.py` argv (audit trail) |
| `service_result`, `start_profile`, `stop_profile`, `stop_result` | Sub-call outputs |
| `benchmark_results`, `followup_result` | Per-request status / latency / response body |
| `workload_status` | `{status, bench_total, bench_ok, bench_success_rate, bench_threshold, followup_ok}` — workload hard gate |
| `remote_profile_root` | Path the analysis skill passes to its `analyze.py` |
| `remote_profile_dirs` | Per-rank `{path, outputs, analysis_status}`; `outputs` carries `export_type` plus `db_path` (db mode; csv fields are null) or `kernel_details_csv` / `trace_view_json` (text/both mode), plus `archived_path` (archive-side path of this rank's outputs, null when `--archive-dir` was not given or this rank's copy failed) |
| `archive_dir` | Actual archive root `<archive-dir>/<tag>_<compact-started_at>` when `--archive-dir` was given, else null |
| `archived` | `true` only when `--archive-dir` was given and every rank's copy succeeded |
| `archive_error` | Present only when an archive copy failed: per-rank error summary. Never flips an already-ok collection to failed |
| `rank_count` | Number of `*_ascend_pt` directories actually found |
| `analysis_status` | `ok` / `partial` / `rank_count_mismatch` / `missing_kernel_details` — analysis hard gate |
| `expected_output_kind` | `db` / `csv` — which artifact `missing_kernel_details` refers to under this run's `analyse_export` |
| `analyse_wall_s` | Wall-clock seconds of the parallel per-rank analyse phase |
| `analyse_parallelism` | Effective analyse parallelism used (`min(rank_count, --analyse-parallelism)`, default cap 8) |
| `status` | `ok` / `failed`. `ok` requires both `analysis_status == "ok"` and `workload_status.status == "ok"` |
| `error` | Set when `status == failed`: an object `{message, knowledge_matches}` — `message` lists which gate(s) tripped, `knowledge_matches` carries the top-3 known-failure-signature entries (with `resolution`) matching the error text, or an explicit empty array |
| `knowledge_advisories` | Preflight knowledge hits for `<served-model-name> tp<N> <mode>` across model-capabilities / parallelism-compatibility / known-failure-signatures (`{entry_id, kind, summary, score}`); advisory only, `[]` when nothing matched |

## Reference files

- `references/command-recipes.md` — common `collect_torch_profile_case.py` invocations
- `references/behavior.md` — control-plane timing, mode mapping, image encoding caveats
- `references/acceptance.md` — manual checks before marking a collection good