ascend-memory-profiling · git:20260910.55adacd · 2026-09-10 · sha256 8e0c65b04c0515f6
ascend-memory-profiling git:20260910.55adacdA
Immutable. This exact content is served forever at /api/v1/blob/8e0c65b04c0515f6.
---
name: ascend-memory-profiling
description: Profile and attribute HBM memory usage on Ascend NPU for vLLM serving scenarios. Breaks down memory into fixed overhead, model weights, KV cache, HCCL buffers, activations, and runtime, with traceable evidence chains. Use for requests like "分析显存占用", "显存 profiling", "HBM 用了多少", "内存各部分拆分". Do not use for performance profiling (kernel timing, throughput), offline inference, or non-Ascend hardware.
---
# Ascend Memory Profiling
Collect and analyze HBM memory usage on Ascend NPU devices running vLLM serving workloads. Produces a structured breakdown of memory by component, with every value traceable to its data source.
Remote substrate rule: use remote-dev companion tools (`remote_*` MCP tools,
launched via `uv run remote-dev` or MCP) for ad hoc remote
read/edit/bash/search/patch work around memory profiling setup and output
inspection. Use this skill for the domain HBM workflow and keep its scripts as
the compatibility backend for managed VAWS sessions.
## Use this skill when
- the user asks to profile or analyze GPU/NPU memory (显存) usage
- the user wants to understand what consumes HBM in a vLLM serving scenario
- the user asks "权重/KV cache/HCCL/激活各占多少显存"
- the user wants to verify memory allocation against theoretical expectations
- the user asks to compare memory usage across different configurations
## Do not use this skill when
- the task is performance profiling (kernel timing, bubble analysis, step/layer/operator breakdown, cross-rank diagnosis) → use `ascend-profiling-analysis` (consumes an `ascend-profiling-collection` manifest or a remote profile root)
- the task is starting/stopping a service without memory analysis → use `vllm-ascend-serving`
- the task involves non-Ascend hardware
- the task is offline (non-serving) inference only
## Data source priority
| Priority | Source | Role | Trustworthiness |
|----------|--------|------|-----------------|
| P0 | `msprof --application` wrapping | Full component breakdown (APP, HCCL, RUNTIME, SLOG) | Highest -- sees memory torch cannot manage |
| P1 | `npu-smi info` | Static baseline + phased delta | High -- hardware-level |
| P2 | vLLM startup logs | Weights, KV cache, num_gpu_blocks | Medium-high -- application-reported |
| P3 | `safetensors` file headers | Tensor shapes, dtypes, byte sizes (byte-accurate); component classification and shard strategy are rule-based inference | High for byte sizes; medium for per-device attribution |
| P4 | Model `config.json` | Theoretical weight calculation (fallback) | Reference only |
## Memory components
| Component | Source | How derived |
|-----------|--------|-------------|
| Fixed overhead (driver) | npu-smi Phase 0 | HBM_Used with no user process |
| Model weights | safetensors + vLLM logs | `weight_inspector.py` 解析文件头 → 按 TP/EP/DP 分片 → 与 vLLM `DeviceMemoryProfiler` 交叉验证 |
| KV cache | vLLM logs | "Available KV cache memory: X GiB" |
| ACL Graph 编译缓冲 | vLLM logs | "Graph capturing finished in X secs, took Y GiB" |
| HCCL buffers | msprof | npu_module_mem.csv Component=HCCL (per-device when PROF→device mapping available, otherwise process-level) |
| CANN Runtime | msprof | npu_module_mem.csv Component=RUNTIME (same scoping as HCCL) |
| Activations | npu-smi delta | HBM during inference minus HBM at idle |
| 未归因残差 | Residual | 所有已知组件加总后的余量 (有 msprof 时通常 < 200 MB) |
### Residual handling
All memory attribution is based on measured data — **no estimation or guessing** is performed.
- **With msprof**: HCCL, RUNTIME, SLOG are precise msprof measurements. Any remaining residual after all components is small (typically < 200 MB) and reported as "未归因残差".
- **Without msprof**: The residual is reported as "未归因 (缺少 msprof 数据)" with an explicit note that msprof collection is needed for a complete breakdown. No attempt is made to split the residual into sub-components.
## Workflow
**Always use the `vllm-ascend-serving` skill for service lifecycle management.** This profiling skill only collects and analyzes data — it attaches to a running service. **msprof wrapping is mandatory** for a complete, traceable memory breakdown.
Memory profiling uses `--context-file` / `VAWS_CONTEXT_FILE` and `--execution-id` / `--service`. It does not guess a session from cwd. Attach mode reads coordinator service facts, not a workspace session ledger.
Prefer `--attach` (service managed by `vllm-ascend-serving`). Standalone mode starts its own service as a fallback; it leases its service port through the session lease mechanism and releases the lease on exit, so it never bypasses session port leases or shared state isolation.
### Step 0: Check msprof availability
Before starting the service, verify that msprof is available on the remote machine:
```bash
python3 -c "
import sys; sys.path.insert(0, '.agents/skills/ascend-memory-profiling/scripts')
from _common import check_msprof_available, resolve_execution_target
target = resolve_execution_target(context_file=None, execution_id='<id>', service='vllm')
ep = target['endpoint']
print(check_msprof_available(ep))
"
```
If this fails, stop and fix the remote environment before proceeding.
### Step 1: Prepare the service with msprof (via `vllm-ascend-serving`)
Upload the msprof wrapper, then start the service with `--wrap-script`:
```bash
# Upload msprof wrapper to remote
python3 -c "
import sys; sys.path.insert(0, '.agents/skills/ascend-memory-profiling/scripts')
from _common import resolve_execution_target, upload_msprof_wrapper
target = resolve_execution_target(execution_id='<id>', service='vllm')
ep = target['endpoint']
print(upload_msprof_wrapper(ep, mem_freq=50))
"
# Start service with msprof wrapping (inside a session worktree the session is
# coordinator-owned service; pass --execution-id if attaching)
python3 .agents/skills/vllm-ascend-serving/scripts/serve_start.py \
--model <path> --tp <N> --service vllm \
--wrap-script /tmp/_vaws_msprof_wrap.sh \
[-- --speculative-config '...' --compilation-config '...' ...]
```
Note: `upload_msprof_wrapper` internally calls `check_msprof_available` as a safety net and writes a unique wrapper path under `/tmp` for each call.
For baseline npu-smi data, collect `npu-smi info` **before** starting the service.
### Step 2: Collect memory data (attach mode)
```bash
# Inside a session worktree (auto-resolved session)
python3 .agents/skills/ascend-memory-profiling/scripts/mem_collect.py \
--attach \
[--baseline-from <previous-run-dir>] \
[--tag <experiment-name>]
# Explicit session target
python3 .agents/skills/ascend-memory-profiling/scripts/mem_collect.py \
--execution-id <id> --attach
```
Use `--session-file <session.json>` when the session file path is the stable handle.
What happens:
- Reads `.vaws-local/sessions/<session-id>/serving.json` to discover port, PID, model path, tp/dp, extra args; the manifest records this path as `serving_state_ref`
- Auto-extracts `--speculative-config`, `--compilation-config`, etc. from the serving state
- Collects npu-smi snapshot, vLLM logs (from serving's runtime dir), weight manifest
- Sends inference request to collect activation delta
- **Does NOT** start or stop the service
### Step 3: Stop service (via `vllm-ascend-serving`)
```bash
python3 .agents/skills/vllm-ascend-serving/scripts/serve_stop.py \
--service vllm
```
Use the same target form that was used for `serve_start.py`.
### Step 4: Collect msprof data
After stop, run `mem_collect --attach` again **with `--resume-run`** pointing to the run directory from Step 2. This merges the msprof CSV data into the same run, producing a single manifest with both live npu-smi data and post-stop msprof CSVs:
```bash
python3 .agents/skills/ascend-memory-profiling/scripts/mem_collect.py \
--execution-id <id> --attach \
--resume-run .vaws-local/memory-profiling/<run-dir-from-step-2>/
```
This works because attach mode accepts stopped services — it skips health check and inference, and focuses on collecting msprof CSVs that are now available. The `--resume-run` flag ensures all data lands in one directory.
### Step 5: Analyze and generate report
```bash
python3 .agents/skills/ascend-memory-profiling/scripts/mem_analyze.py \
.vaws-local/memory-profiling/<run-dir>/ \
[--format json|text]
```
Default `--format json` outputs machine-readable JSON to stdout (matching the repo-wide wrapper contract). Use `--format text` for a human-readable report on stdout. Both modes write `report.json` and `report.txt` files to the run directory.
### Baseline strategy
| Situation | How to get baseline |
|-----------|-------------------|
| Fresh profiling | Collect `npu-smi info` before `serve_start`, save to file, use `--baseline-from <file>` |
| Repeat profiling on same machine | Reuse baseline from a previous run via `--baseline-from <old-run-dir>` |
| Quick analysis (no baseline needed) | Omit `--baseline-from` — report shows "固定开销" as 0 with note |
`--baseline-from` accepts either a previous run directory (containing `baseline_npu_smi.txt` or `manifest.json`) or a raw `npu-smi info` output text file.
### Fallback: Standalone mode
When the serving skill is unavailable (e.g. bootstrap scenario), `mem_collect.py` can manage the service internally. This is a **fallback** — prefer the serving skill workflow above.
```bash
python3 .agents/skills/ascend-memory-profiling/scripts/mem_collect.py \
--model <remote-weight-path> \
--tp <N> [--dp <N>] [--tag <name>] \
[--speculative-config '...'] [--compilation-config '...'] ...
```
Task identity is `--context-file` / `VAWS_CONTEXT_FILE`.
This runs all phases internally: baseline → start (with msprof) → health check → snapshot → inference → stop → msprof export. msprof is always enabled in standalone mode; a pre-flight check verifies msprof availability before starting. The service port is leased through the session lease mechanism (no ad-hoc free-port scanning) and the lease is released on exit.
### Standalone: Analyze and generate report
```bash
python3 .agents/skills/ascend-memory-profiling/scripts/mem_analyze.py \
.vaws-local/memory-profiling/<run-dir>/
```
Outputs:
- `report.txt` -- human-readable report with evidence chains
- `report.json` -- machine-readable structured report
### Example output (S2: MoE 35B, TP=4, DP=2, MTP=3, FULL_DECODE_ONLY, with msprof)
```
[Device 0] 总 HBM: 32.00 GiB | 已用: 27.42 GiB
组件 | 占用 (MB) | 占用 (GiB) | 占比 | 主数据源
--------------------------------------------------------------------------------------------------------------
固定开销 (driver/runtime) | 2930.0 | 2.861 | 10.4% | npu-smi Phase 0 (baseline)
模型权重 | 9692.6 | 9.465 | 34.5% | safetensors 文件头精确计算
KV Cache 预留 | 14080.0 | 13.750 | 50.2% | vLLM 日志
ACL Graph 编译缓冲 | 501.8 | 0.490 | 1.8% | vLLM 日志
HCCL 缓冲 | 597.7 | 0.584 | 2.1% | msprof npu_module_mem
CANN Runtime | 125.3 | 0.122 | 0.4% | msprof npu_module_mem
激活峰值 | 72.0 | 0.070 | 0.3% | npu-smi delta
└ 未归因残差 | 74.7 | 0.073 | 0.3% | 残差
[交叉验证]
npu-smi 已用: 28,074 MB
组件加总: 27,999.3 MB
未归因: 74.7 MB (0.3%)
msprof APP: 27,500 MB
```
## Interaction with other skills
| Skill | Interaction |
|-------|-------------|
| `vllm-ascend-serving` | **Service lifecycle**: Use `serve_start.py` / `serve_stop.py`. Attach mode uses `--execution-id` / `--service`. |
| `vaws_session` | Bind actual worktrees through native task context. |
| `vaws-coordinator` | Own managed environments, device allocation and source preparation. |
## Critical rules
- **Service lifecycle belongs to `vllm-ascend-serving`** — this skill only collects and analyzes data.
- In attach mode, `mem_collect` will **never** start or stop the service itself. Service stop is done by the agent calling `serve_stop.py` directly. After stop, `mem_collect --attach` can be run again to export and collect msprof data.
- In standalone mode (fallback), the service is started and stopped within the profiling run.
- `msprof` export (8 卡) 可能需要几分钟。
- Keep collected data under `.vaws-local/memory-profiling/` only (untracked).
## Weight analysis methodology
The skill uses `weight_inspector.py` to parse safetensors file headers on the remote machine, extracting tensor names, shapes, dtypes, and byte sizes. Individual tensor byte sizes are **byte-accurate** from the file headers. Component classification (e.g. "Attention Q", "MoE Expert") and shard strategy assignment (col/row/expert parallel) are **rule-based inferences** from tensor name patterns, not direct measurements.
### How it works
1. **Parse safetensors headers**: Each `.safetensors` file starts with a JSON header containing tensor metadata (name, shape, dtype, offsets). No weight data is read — only the header (typically a few KB per file).
2. **Classify tensors**: Each tensor is categorized by name pattern:
- `embed_tokens` → Embedding
- `self_attn.{q,k,v,o}_proj` → Full Attention
- `linear_attn.{in_proj_qkv,in_proj_z,out_proj,conv1d,...}` → Linear/Mamba Attention
- `mlp.experts.*` → MoE Experts
- `mlp.shared_expert.*` → MoE Shared Expert
- `visual.*` → Vision Encoder
- `mtp.*` → MTP (Multi-Token Prediction)
- `norm`, `layernorm` → Layer Norms
3. **Determine shard strategy**: Each tensor is assigned a parallelism strategy:
- `col_parallel` / `row_parallel` → divided by TP
- `expert_parallel` → divided by EP (= TP × DP when EP enabled)
- `replicated` → full copy on each device (norms, gates, small params)
4. **Calculate per-device weight**: Sum up per-device bytes across all tensors.
5. **Cross-validate with vLLM**: Compare safetensors-derived per-device value with vLLM's `DeviceMemoryProfiler` measurement. Expected differences:
- **MTP embedding/lm_head sharing**: MTP reuses base model's embeddings (reduces actual memory)
- **F32 → BF16 conversion**: Some small parameters (A_log, dt_bias) stored as F32 but may load as BF16
- **Vision encoder in text-only mode**: Still loaded for multimodal architectures
### When the script isn't sufficient
If the safetensors-based analysis shows unexpected results, the agent should:
1. Check `weight_manifest.json` for unclassified ("other") tensors
2. Inspect vLLM's model implementation (`load_weights`) for weight sharing or skipping logic
3. Compare `model.named_parameters()` output on the remote machine with the safetensors manifest
4. Check if `enable_ep_weight_filter` affects loading behavior
## Limitations
- Activation measurement relies on npu-smi delta between idle and inference states. This captures peak but not fine-grained activation lifetime.
- Endpoint-based `torch_npu.profiler` via vLLM `/start_profile`/`/stop_profile` has been observed with empty device-side data (`device_0/data`). That observation is not a universal capability prohibition, and this skill does not record a version, date, or topology for it. A 2xx response from those endpoints is control-plane evidence only; it is not device-data evidence. For torch-profiler captures, use [`ascend-profiling-collection`](../ascend-profiling-collection/SKILL.md), which verifies per-rank analyse outputs and expected rank count ([output checks](../ascend-profiling-collection/references/acceptance.md)). Missing or empty per-rank artifacts mean the capture is incomplete. This skill still uses msprof wrapping for HBM attribution; keep the msprof workflow and the limitations below.
- msprof wrapping profiles the main process only. TP worker processes are separate -- each gets its own PROF directory with per-device data.
- msprof export for 8-card runs may take several minutes. The timeout is set to 1800s.
## References
- `references/methodology.md` -- Detailed methodology and data source descriptions
- `references/msprof_fields.md` -- msprof CSV field reference