git:20260910.55adacd to git:20260911.9f92063

- # Ascend Profiling Collection

-

- Collect one torch-profiler case on a workspace-managed remote Ascend NPU container.

-

- Remote substrate rule: use remote-dev companion tools (`remote_*` MCP tools,

- launched via `uv run remote-dev` or MCP) for ad hoc remote

- read/edit/bash/search/patch work around profile setup and output inspection.

- Use this skill for the domain collection workflow and keep its scripts as the

- compatibility backend for managed VAWS sessions.

-

- This skill is **only** about collection: start a profiled service, bracket a workload with `/start_profile` and `/stop_profile`, run `torch_npu.profiler.profiler.analyse(...)`, verify the device-side data actually landed, and write a manifest. By default analyse() runs with `export_type=Constant.Db`, so each rank produces only `ASCEND_PROFILER_OUTPUT/ascend_pytorch_profiler_*.db` — the analysis skill rebuilds the kernel_details event stream directly from the db, and the multi-GB text exports (`trace_view.json`, CSVs) are skipped. Interpreting the data is a separate concern owned by the analysis skill.

-

- ## Use this skill when

-

- - the user asks to collect / capture an Ascend torch-profiler trace for a specific config

- - another skill (the analysis skill) needs a fresh profiling root with verified outputs

- - the user wants to reproduce an existing root with a new model / mode / TP / DP

-

- ## Do not use this skill when

-

- - the task is performance benchmarking only — use `vllm-ascend-benchmark`

- - the task is HBM / memory analysis — use `ascend-memory-profiling`

- - the task is analysing an already-collected profiling root (no need to re-collect)

- - the request is machine provisioning — use coordinator provision

-

- ## Boundary with other skills

-

- | Skill | Owns | This skill uses it for |

- | --- | --- | --- |

- | `vllm-ascend-serving` | Service lifecycle, `--profiler-config` passthrough | `serve_start.py` / `serve_stop.py` only; serving is **agnostic** to the profiler window |

- | `vaws-coordinator` | Managed sources and environment preparation | Owned by the single service run |

- | `vllm-ascend-benchmark` | `vllm bench serve` performance numbers | Not used; benchmark skill must not learn the profiler control plane |

- | `ascend-memory-profiling` | HBM attribution via msprof | Independent; not invoked |

-

- `/start_profile` and `/stop_profile` exist *because of* profiling, so the control-plane client lives here, not in `vllm-ascend-serving`.

-

- ## Critical rules

-

- - The serving skill must remain profiling-agnostic. Never push profiler-window control or `analyse()` invocation into it.

- - `--profiler-config` is the only profiling-related thing the serving skill knows about, and only because vLLM accepts it as an opaque blob.

- - Run profiling **inside the remote container**. Never copy raw `*_ascend_pt` directories back to the local Mac.

- - Hard-fail in three cases (all detailed in "Failure policy"):

- 1. any rank's expected analyse output is missing after `analyse()` — the per-rank db in the default `--analyse-export db` mode, `kernel_details.csv` in text/both mode

- 2. number of `*_ascend_pt` directories does not match `tp * (dp or 1)`

- 3. workload was not real — follow-up request failed or benchmark wave fell below `--benchmark-success-threshold`

- - Progress on `stderr` as `__VAWS_PROGRESS__=<json>`. Final manifest on `stdout` as one JSON object.

- - Collection uses `--context-file` / `VAWS_CONTEXT_FILE` and `--execution-id` / `--service`. Do not guess a session from cwd.

- - Local state lives under `.vaws-local/ascend-profiling-collection/runs/` for collection manifests; serving/parity state lives under the session namespace.

-

- ## Public entry point

-

- ```bash

- python3 .agents/skills/ascend-profiling-collection/scripts/collect_torch_profile_case.py \

- [--context-file <path>] [--execution-id <id>] [--service vllm] \

- --model <remote-weight-path> \

- --served-model-name <name> \

- --tp <N> \

- --tag <stable-id> \

- --mode {enforce_eager|full_decode_only|piecewise_graph} \

- --request-kind {text|vl} \

- --benchmark-output-tokens <N> \

- [--dp <N>] \

- [--enable-expert-parallel] \

- [--speculative-tokens <N>] [--speculative-method <name>] \

- [--gpu-memory-utilization <f>] \

- [--max-model-len <N>] [--max-num-seqs <N>] [--max-num-batched-tokens <N>] \

- [--api-server-count <N>] \

- [--prompt-tokens <N>] [--followup-output-tokens <N>] \

- [--benchmark-total-requests <N>] [--benchmark-concurrency <N>] \

- [--benchmark-success-threshold <f>] \

- [--request-timeout <s>] [--profile-control-timeout <s>] [--health-timeout <s>] \

- [--torch-profiler-dir <relpath>] [--torch-profiler-with-stack] \

- [--analyse-export {db|text|both}] \

- [--archive-dir <remote-path>] \

- [--image-path <local-path>] [--image-height <px>] \

- [--skip-parity]

- ```

-

- ### Required parameters and why

-

- Task identity is `--context-file` / `VAWS_CONTEXT_FILE`. A live service is `--execution-id` or `--service`.

-

- The script intentionally has no Qwen-specific defaults. The agent must always pass:

-

- | Required arg | Why |

- | --- | --- |

- | `--model` / `--served-model-name` | Different cases need different models, no safe default |

- | `--tp` | Hardware shape; never assume it |

- | `--mode` | The profile is meaningless without recording which graph mode produced it |

- | `--request-kind` | Determines payload assembly (text vs VL) |

- | `--benchmark-output-tokens` | Decode length is the dominant knob for what the trace looks like |

- | `--tag` | Stable identifier folded into the run-dir name and manifest |

-

- `--speculative-tokens 0` (the default) means "do not pass `--speculative-config` at all". Set to a positive integer to enable MTP/Eagle.

-

- ## Auxiliary entry points

-

- The agent can call these directly if it already has a service running and only wants to flip the profiler window or re-run `analyse()` on an existing root.

-

- ### Flip the profiler window

-

- ```bash

- # Start a profile window on a service that the serving skill already launched

- python3 .agents/skills/ascend-profiling-collection/scripts/profile_control.py \

- [--execution-id <id> | --service vllm] --action start_profile [--timeout 900]

-

- # Close it

- python3 .agents/skills/ascend-profiling-collection/scripts/profile_control.py \

- [--execution-id <id> | --service vllm] --action stop_profile [--timeout 900]

- ```

-

- The script reads the service port from the coordinator execution named by `--execution-id` or `--service`. A service must be running.

-

- ### Re-run `analyse()` on an existing root

-

- ```bash

- python3 .agents/skills/ascend-profiling-collection/scripts/run_remote_analyse.py \

- [--execution-id <id> | --host <ip>] --profile-root <remote-path> \

- [--expected-ranks <N>] [--analyse-timeout <s>] [--analyse-parallelism <N>] \

- [--analyse-export {db|text|both}]

- ```

-

- Discovers every `*_ascend_pt` under `--profile-root` and runs `torch_npu.profiler.profiler.analyse()` on each **concurrently on the container** (one SSH call, `xargs -P`; effective parallelism `min(rank_count, --analyse-parallelism)`, default 8 — per-rank analyse is CPU-bound and the containers have hundreds of cores, so TP16 no longer analyses 16 ranks serially). Each rank's stdout/stderr is captured in `<dir>/analyse_parallel.log`; per-rank exit codes are aggregated and any non-zero rank fails the run. `--analyse-timeout` is the overall wall-clock bound for the parallel phase (default 1800s, *not* multiplied by rank count; a remote `timeout(1)` wrapper kills stuck ranks).

-

- `--analyse-export` selects the `export_type` passed to `analyse()`:

-

- | Mode | analyse() call | Verified per rank |

- | --- | --- | --- |

- | `db` (default) | `analyse(dir, export_type=Constant.Db)` — skips all text exports | newest `ASCEND_PROFILER_OUTPUT/ascend_pytorch_profiler_*.db` exists and is non-empty |

- | `text` | `analyse(dir, export_type=Constant.Text)` | `kernel_details.csv` + `trace_view.json` exist (historical contract) |

- | `both` | `analyse(dir, export_type=[Constant.Text, Constant.Db])` | same as `text` |

- The `Constant` import (`from torch_npu.profiler.analysis.prof_common_func._constant import Constant`) runs inside the generated per-rank payload so it always resolves against the container's torch_npu. On old CANN without db export support, analyse() raises (`is_support_export_db()`); that surfaces as a rank failure with the torch_npu error in `analyse_parallel.log` — use `--analyse-export text` on such hosts. Exits non-zero if any rank is incomplete.

- Always pass `--expected-ranks` (typically `tp * (dp or 1)`) when running this against a fresh capture: without it a partial collection where some ranks never produced a directory looks "clean" because every directory that *did* land was complete. The orchestrator passes this automatically.

- ## Archiving to shared storage (`--archive-dir`)

- Profiling outputs are far too large to pull back to the local Mac (a single dsv3.1 analysis once dragged back 2.2GB). When `--archive-dir <remote-path>` points at the shared-storage filesystem mounted on every managed host/container (e.g. `/mnt/weight/<user>/profiling/archives`), the orchestrator archives each rank's analyse outputs **on the container** (cp -r over the existing ssh channel, ranks serially) right after analyse+verify passes with all ranks ok:

- ```

- <archive-dir>/<tag>_<started_at-compact-ts>/<rank-dir-basename>/

- ASCEND_PROFILER_OUTPUT/ # copied whole (db mode: carries the per-rank db; no csv to cherry-pick)

- profiler_info_*.json # best-effort

- profiler_metadata.json # best-effort

- Because the rank-dir basename is kept as the subdirectory name, the archive root `<archive-dir>/<tag>_<ts>/` is itself a valid profiling root full of `*_ascend_pt` directories — feed it straight to the analysis skill's `--remote-profile-root` from **any** machine that mounts the same shared storage, no re-collection needed.

-

- **Failure semantics**: archiving never overturns an already-ok collection. A copy failure records `archive_error` in the manifest (with the affected rank's `outputs.archived_path` left null and `archived: false`) plus a stderr warning; the run still exits 0 with `status: ok`.

-

- Manifest additions (see the schema table below): top-level `archive_dir` (the actual archive root, null when not requested), `archived` (true only when every rank copied), `archive_error` (present only on failure), and per-rank `remote_profile_dirs[].outputs.archived_path`.

-

- ## Workflow

-

- 1. **Resolve the native task** (`--context-file`) and the live service (`--execution-id` / `--service`).

- 2. **Build serving args** — encode `--profiler-config` (always written) and the chosen graph mode.

- 3. **Start service** by shelling out to `serve_start.py`. Parity sync is automatic via the serving skill.

- 4. **Open SSH tunnel** to the service port so workload requests can be assembled locally (multimodal payloads need local image encoding).

- 5. **POST `/start_profile`** with the long control-plane timeout.

- 6. **Send benchmark wave** (concurrent chat-completions) followed by **one follow-up tail request**. The follow-up is intentionally short to capture a clean steady-state step.

- 7. **POST `/stop_profile`**.

- 8. **Stop service** by shelling out to `serve_stop.py`.

- 9. **Discover and analyse** every `*_ascend_pt` under `<runtime_dir>/<torch_profiler_dir>` via `run_remote_analyse.py` (parallel across ranks, see above).

- 10. **Verify outputs** per rank (db mode: non-empty per-rank db; text/both: `kernel_details.csv` + `trace_view.json`); classify each as `ok | partial | missing_kernel_details`.

- 11. **Archive (optional)** — with `--archive-dir`, copy each rank's `ASCEND_PROFILER_OUTPUT/` + profiler metadata to shared storage (see "Archiving to shared storage"); runs only when every rank verified ok, failures degrade to `archive_error` without flipping the run status.

- 12. **Write manifest** to `.vaws-local/ascend-profiling-collection/runs/<timestamp>_<tag>/manifest.json`.

-

- ## Failure policy

-

- Accuracy beats coverage. The script exits non-zero (status `failed`) when **any** of:

-

- - the service did not become `ready`

- - `/start_profile` or `/stop_profile` returned non-2xx

- - **workload was not real**: `workload_status.status != "ok"`, i.e. the

- follow-up request failed or the benchmark wave's success rate was below

- `--benchmark-success-threshold` (default 0.8). Without real traffic during

- the profile window the trace records nothing useful.

- - **rank count mismatch**: number of `*_ascend_pt` directories `!= tp * (dp or 1)`

- → `analysis_status == "rank_count_mismatch"`. Some rank never dumped its

- profiler data; even if every directory that *did* land is complete, the

- topology is broken and downstream cross-rank analysis would be wrong.

- - any rank's expected analyse output is missing after `analyse()` →

- `analysis_status == "missing_kernel_details"`. In the default

- `--analyse-export db` mode this means the rank's

- `ascend_pytorch_profiler_*.db` was not produced or is empty; in text/both

- mode it means `kernel_details.csv` is missing. The manifest's

- `expected_output_kind` field (`db` / `csv`) records which artifact the

- enum refers to; the enum set itself is unchanged so downstream gates keep

- working.

-

- The last condition is the canonical "device-side data did not land" failure

- documented in `references/behavior.md` ("Output verification"). Treat all of the above as

- **re-collect required**, not as something the analysis skill can recover from.

-

- If the orchestrator fails after `serve_start`, it always tries to stop the service (graceful, then `--force`) so no orphan vLLM process is left behind.

-

- ## Workspace knowledge hooks

-

- Both hooks are local, advisory-only views over the workspace knowledge store (`.agents/knowledge/`, via `.agents/lib/vaws_knowledge_service.py` and the installed `vaws-knowledge` package); a missing/invalid knowledge dir degrades them to explicit empty arrays with a stderr progress note — collection is never blocked:

-

- - **Preflight advisories**: before `serve_start`, the orchestrator queries `model-capabilities` / `parallelism-compatibility` / `known-failure-signatures` with `<served-model-name> tp<N> <mode>` (limit 3 per kind). Hits land in the manifest's `knowledge_advisories` array (`{entry_id, kind, summary, score}`) and are printed to stderr progress. Use them as hints (e.g. a model's verified TP/mode or a known container pitfall), not as gates.

- - **Failure-gate enrichment**: when any hard-fail gate trips (service not ready, workload not real, `rank_count_mismatch`, `missing_kernel_details`), the observed error text is queried against `known-failure-signatures` (limit 3). Matches — including each entry's `resolution` — are written to `manifest.error.knowledge_matches`, and `manifest.error` becomes an object `{message, knowledge_matches}` instead of a bare string. The same object is the stdout failure JSON. No match → explicit empty array.

-

- **沉淀提示**: when a collection failure turns into a confirmed new diagnosis (symptom + root cause + verified fix), capture it with `.agents/scripts/knowledge_capture.py` as a local candidate (`.vaws-local/knowledge/candidate/`, untracked); promotion into `.agents/knowledge/` happens only through reviewed project changes.

-

- ## Manifest schema

-

- The manifest is the input contract for the analysis skill. Important fields:

- | Field | Meaning |

- | --- | --- |

- | `schema_version` | Bumped when fields are renamed or removed |

- | `tag`, `started_at`, `completed_at` / `failed_at` | Run identity |

- | `machine`, `model`, `served_model_name`, `tp`, `dp` | Hardware / model identity |

- | `mode`, `speculative_tokens`, `speculative_method`, `enable_expert_parallel`, `api_server_count` | What was profiled |

- | `request_kind`, `prompt_tokens`, `benchmark_output_tokens`, `followup_output_tokens`, `benchmark_total_requests`, `benchmark_concurrency`, `benchmark_success_threshold` | What workload produced the trace and what success bar it had to clear |

- | `expected_ranks` | `tp * (dp or 1)` — what `analyse()` was told to enforce |

- | `torch_profiler_with_stack`, `torch_profiler_dir` | Profiler depth and on-disk location |

- | `analyse_export` | `db` (default) / `text` / `both` — the `export_type` analyse() was run with |

- | `serve_args` | Exact `serve_start.py` argv (audit trail) |

- | `service_result`, `start_profile`, `stop_profile`, `stop_result` | Sub-call outputs |

- | `benchmark_results`, `followup_result` | Per-request status / latency / response body |

- | `workload_status` | `{status, bench_total, bench_ok, bench_success_rate, bench_threshold, followup_ok}` — workload hard gate |

- | `remote_profile_root` | Path the analysis skill passes to its `analyze.py` |

- | `remote_profile_dirs` | Per-rank `{path, outputs, analysis_status}`; `outputs` carries `export_type` plus `db_path` (db mode; csv fields are null) or `kernel_details_csv` / `trace_view_json` (text/both mode), plus `archived_path` (archive-side path of this rank's outputs, null when `--archive-dir` was not given or this rank's copy failed) |

- | `archive_dir` | Actual archive root `<archive-dir>/<tag>_<compact-started_at>` when `--archive-dir` was given, else null |

- | `archived` | `true` only when `--archive-dir` was given and every rank's copy succeeded |

- | `archive_error` | Present only when an archive copy failed: per-rank error summary. Never flips an already-ok collection to failed |

- | `rank_count` | Number of `*_ascend_pt` directories actually found |

- | `analysis_status` | `ok` / `partial` / `rank_count_mismatch` / `missing_kernel_details` — analysis hard gate |

- | `expected_output_kind` | `db` / `csv` — which artifact `missing_kernel_details` refers to under this run's `analyse_export` |

- | `analyse_wall_s` | Wall-clock seconds of the parallel per-rank analyse phase |

- | `analyse_parallelism` | Effective analyse parallelism used (`min(rank_count, --analyse-parallelism)`, default cap 8) |

- | `status` | `ok` / `failed`. `ok` requires both `analysis_status == "ok"` and `workload_status.status == "ok"` |

- | `error` | Set when `status == failed`: an object `{message, knowledge_matches}` — `message` lists which gate(s) tripped, `knowledge_matches` carries the top-3 known-failure-signature entries (with `resolution`) matching the error text, or an explicit empty array |

- | `knowledge_advisories` | Preflight knowledge hits for `<served-model-name> tp<N> <mode>` across model-capabilities / parallelism-compatibility / known-failure-signatures (`{entry_id, kind, summary, score}`); advisory only, `[]` when nothing matched |

- ## Reference files

- - `references/command-recipes.md` — common `collect_torch_profile_case.py` invocations

- - `references/behavior.md` — control-plane timing, mode mapping, image encoding caveats

- - `references/acceptance.md` — manual checks before marking a collection good