benchmark git:20260702.ae1592bA

Immutable. This exact content is served forever at /api/v1/blob/cdd189ab7058736c.

---
name: benchmark
description: Benchmark runtime performance on hardware. If the current branch has commits ahead of upstream/main or uncommitted changes, compares against the fork point (merge-base). Otherwise benchmarks current state only. Use when the user asks to benchmark, measure performance, or compare latency.
---

# Benchmark Workflow

Benchmark runtime performance on Ascend hardware. Automatically detects whether to run a single benchmark or a comparison.

## Modes

| Condition | Mode | What happens |
| --------- | ---- | ------------ |
| 0 commits ahead AND no uncommitted changes | **Single** | Benchmark current state, report Elapsed + Orch times |
| >= 1 commits ahead OR uncommitted changes | **Compare** | Benchmark merge-base (worktree) AND current workspace, show comparison table |

## Input

Optional benchmark arguments forwarded to `tools/benchmark_rounds.sh`:

```text
/benchmark
/benchmark -d 4 -n 50
/benchmark -d 4 -d 6
/benchmark --serial-orch-sched
```

Extra arguments (`-n`, `-r`, `--serial-orch-sched`, etc.) are forwarded to
`tools/benchmark_rounds.sh`.

### Device arguments (`-d`)

The `-d` flag specifies NPU device IDs.

**Hard rule: one benchmark process per device at any time.** Never run two benchmark processes on the same `-d` device concurrently — not two runtimes, not baseline + current, nothing. This prevents resource contention and ensures stable measurements.

| `-d` count | Compare mode behavior |
| ---------- | --------------------- |
| One device (`-d 4`) | **Sequential**: baseline first, then current, both on the same device. Multiple runtimes also run serially on that device. |
| Two devices (`-d 4 -d 6`) | **Parallel per-runtime**: for each runtime, baseline on first device and current on second device can run in parallel (different devices). Multiple runtimes still run serially — finish one runtime on both devices before starting the next. |
| Zero (not specified) | Auto-detect idle devices (see Step 2) |

**Defaults** (when not specified): use `benchmark_rounds.sh` defaults (device 0, 100 rounds, a2a3, tensormap_and_ringbuffer).

## Runtime Selection

`tools/benchmark_rounds.sh` supports `-r <runtime>`:

- `tensormap_and_ringbuffer` (default)

The example list is defined at the top of the script (`TMR_EXAMPLE_CASES`).

## Step 1: Detect Mode

```bash
git fetch upstream main --quiet
COMMITS_AHEAD=$(git rev-list HEAD --not upstream/main --count 2>/dev/null || echo "0")
HAS_CHANGES=$(git status --porcelain)

if [ "$COMMITS_AHEAD" -eq 0 ] && [ -z "$HAS_CHANGES" ]; then
  MODE="single"
else
  MODE="compare"
  MERGE_BASE=$(git merge-base upstream/main HEAD)
fi
```

## Step 2: Device Detection

If `-d` was provided in args, skip detection and use the user-specified device(s).

Otherwise, detect idle NPU devices (HBM-Usage = 0):

```bash
npu-smi info
```

Pick devices with **HBM-Usage = 0**. Find the longest consecutive sub-range (at most 4). If no idle device is found, prompt user to specify a device ID.

## Step 3: Confirm PTO-ISA Pin

PTO-ISA is selected by the repo-root `pto_isa.pin`. Record the pin in the
benchmark notes so baseline and current runs can be compared against the same
source revision:

```bash
PTO_ISA_PIN=$(tr -d '[:space:]' < pto_isa.pin)
```

## Step 4: Prepare — Compute Absolute Paths

The Bash tool resets its working directory to the project root on every call. Relative paths like `cd worktree && ...` are fragile and easy to forget. **Compute absolute paths once, then use them everywhere.**

```bash
PROJECT_ROOT="$(pwd)"                    # e.g. /home/user/simpler
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
WORKTREE_ABS="${PROJECT_ROOT}/tmp/worktree_baseline_${TIMESTAMP}"
mkdir -p "${PROJECT_ROOT}/tmp"
```

Store `PROJECT_ROOT` and `WORKTREE_ABS` as shell variables in every Bash call that needs them (the Bash tool does not persist variables across calls). Use this pattern:

```bash
# Correct — self-contained, uses absolute path
WORKTREE_ABS="/home/user/simpler/tmp/worktree_baseline_20260331_102302"
"${WORKTREE_ABS}/tools/benchmark_rounds.sh" -d 2 ...
```

**Do NOT use `cd` + relative `./tools/...`** — this is the #1 source of silent errors (running the wrong workspace).

## Step 5: Run Benchmarks

### Single Mode

```bash
./tools/benchmark_rounds.sh $BENCH_ARGS -r "$RUNTIME" 2>&1 | tee "tmp/benchmark_${TIMESTAMP}.txt"
```

Use `--serial-orch-sched` to run each case once in the default overlapped mode
and once with `PTO2_SERIAL_ORCH_SCHED=1`, then emit serial-vs-parallel
Delta/Change tables.

### Compare Mode

Use a **git worktree** for the baseline so the current workspace is never disturbed.

#### CRITICAL: Worktree needs its own build environment

The worktree is a fresh checkout at merge-base — it has **no pre-built runtime binaries** and no compiled nanobind extension. Two things must be built:

1. **Runtime `.so` binaries** (`build/lib/`) — loaded via ctypes by `bindings.py`
2. **Nanobind `_task_interface` extension** — compiled C++ Python bindings

Pure Python files under `simpler_setup/` (e.g. `scene_test.py`, `kernel_compiler.py`) are resolved via `sys.path` from the worktree when an editable install is active there, so they correctly come from the worktree. But `_task_interface.*.so` is installed into site-packages by `pip install -e .` and is **shared system-wide**. Without isolation, the worktree would use the main workspace's nanobind extension — which may have incompatible API changes.

**Solution: always create a venv in the worktree** (~26s overhead). This builds both the nanobind extension AND runtime binaries, fully isolating the baseline.

#### 5a. Create worktree, venv, and build

Inline the **absolute** worktree path (copy-paste the value, do not rely on shell variables persisting):

```bash
# Create worktree
git worktree add "$WORKTREE_ABS" "$MERGE_BASE" --quiet

# Create venv with system site-packages (for torch, numpy, etc.)
python3 -m venv "${WORKTREE_ABS}/.venv" --system-site-packages

# Install into venv — builds nanobind extension + runtime binaries
"${WORKTREE_ABS}/.venv/bin/pip" install -e "${WORKTREE_ABS}" -q 2>&1 | tail -3
```

This gives the worktree its own `_task_interface.*.so` in `.venv/lib/python3.*/site-packages/`, completely independent from the main workspace.

#### 5b. Run baseline

Activate the venv so `benchmark_rounds.sh` (which calls `python3`) picks up the worktree's nanobind extension and Python bindings:

```bash
# WORKTREE_ABS must be the literal absolute path.
cd "$WORKTREE_ABS" && \
  source .venv/bin/activate && \
  pwd && \
  ./tools/benchmark_rounds.sh -d "$BASELINE_DEVICE" -r "$RUNTIME" \
  2>&1 | tee "${PROJECT_ROOT}/tmp/benchmark_baseline_${TIMESTAMP}_${RUNTIME}.txt"
```

**Always include `pwd &&` after `cd` to verify you are in the correct directory.** If `pwd` does not print the worktree path, something went wrong — do not proceed.

#### 5c. Run current

```bash
# Runs from the main workspace (Bash tool default cwd)
./tools/benchmark_rounds.sh -d $CURRENT_DEVICE -r "$RUNTIME" \
  2>&1 | tee "tmp/benchmark_current_${TIMESTAMP}_${RUNTIME}.txt"
```

#### 5d. Cleanup

```bash
git worktree remove "$WORKTREE_ABS" --force
```

If `git worktree remove` fails (e.g., cwd was inside the deleted worktree), use:

```bash
git -C "$PROJECT_ROOT" worktree remove "$WORKTREE_ABS" --force
```

#### Parallel execution (two devices)

When two devices are available, run baseline and current **for the same runtime** in parallel on separate devices. The venv ensures the worktree has its own nanobind extension, so both workspaces are fully independent.

```bash
# For each runtime (serially):
for RUNTIME in "${RUNTIMES_TO_BENCH[@]}"; do
  # Baseline on device A (from worktree with venv), current on device B (from main) — parallel
  (cd "$WORKTREE_ABS" && source .venv/bin/activate && pwd && ./tools/benchmark_rounds.sh -d $DEVICE_BASELINE -r "$RUNTIME" ...) &
  ./tools/benchmark_rounds.sh -d $DEVICE_CURRENT -r "$RUNTIME" ... &
  wait  # Both finish before starting next runtime
done
```

**Never launch the next runtime until the current one finishes on all devices.**

#### Sequential execution (one device)

```bash
# 1. Worktree + venv already created in step 5a

# 2. For each runtime (serially — one device, one process at a time):
#    Baseline first (from worktree with venv activated)
cd "$WORKTREE_ABS" && \
  source .venv/bin/activate && \
  pwd && \
  ./tools/benchmark_rounds.sh -d "$DEVICE" -r "$RUNTIME" \
  2>&1 | tee "${PROJECT_ROOT}/tmp/benchmark_baseline_${TIMESTAMP}_${RUNTIME}.txt"

#    Then current (from main workspace — default cwd, no venv)
./tools/benchmark_rounds.sh -d $DEVICE -r "$RUNTIME" \
  2>&1 | tee "tmp/benchmark_current_${TIMESTAMP}_${RUNTIME}.txt"

# 3. Cleanup
git -C "$PROJECT_ROOT" worktree remove "$WORKTREE_ABS" --force
```

## Step 6: Report Results

Parse all five `Avg <Metric>:` lines per example (`Host`, `Device`, `Effective`, `Orch`, `Sched`) from benchmark output.

| Metric | Source | What it captures |
| ------ | ------ | ---------------- |
| Host | `[STRACE]` `run_prepared` span | steady_clock around dispatch (Python overhead included); rendered from markers by `strace_timing --rounds-table` |
| Device | `[STRACE]` `run_prepared.runner_run.device_wall` span | full on-NPU AICPU run wall (`AicpuPhase::RunWall`, `max(end) − min(start)` across threads) — the whole run + teardown, strictly larger than the windows below |
| Effective | orch/sched markers' device-domain `ts`+`dur` | `max(orch_end,sched_end) − min(orch_start,sched_start)` — the orch∪sched merged window (the old device-log "Total"), now pure-marker |
| Orch | `[STRACE]` `…device_wall.orch` span (`--rounds-table`) | orchestrator (graph-build) window |
| Sched | `[STRACE]` `…device_wall.sched` span (`--rounds-table`) | scheduler dispatch/execution window |

The scene test only *emits* `[STRACE]` markers to stderr; `benchmark_rounds.sh`
tees the run and renders the Host/Device/Effective/Orch/Sched table with
`python -m simpler_setup.tools.strace_timing <log> --rounds-table`. All columns
come from the markers (onboard and sim) — no CANN device log is read.

For a per-stage breakdown of `Host`/`Device` (host `bind`/`runner_run`/`validate`
plus the AICPU `preamble`/`so_load`/`graph_build`
(`config_validate`/`arena_wire`/`sm_reset` prep sub-phases)/`post_orch`
subdivision), parse the `[STRACE]` markers with
`simpler_setup/tools/strace_timing.py` (add `--tree` for the nested view) — see
[docs/dfx/host-trace.md](../../../docs/dfx/host-trace.md). Same `SIMPLER_PROFILING`
gate, no extra flag (set `SIMPLER_DEVICE_PROFILING=0` to drop only the device
`clk=dev` markers).

### Single Mode

```text
Benchmark at: <short SHA>
Args: -d 4 -n 100

Example                          Host (us)   Device (us)   Effective (us)    Orch (us)   Sched (us)
-------------------------------  ---------   -----------   --------------   ----------   ----------
alternating_matmul_add           480000.0        9050.0         1235.5          820.3       1235.4
benchmark_bgemm                  370000.0        7100.0          892.1          650.2        892.0
...
```

### Compare Mode

Show comparison table per metric (one row per metric per example), **grouped by runtime**. `Effective` is the headline metric used in the overall summary; the other four are sub-rows for context:

```text
Merge-base: <short SHA>  →  HEAD: <short SHA> (+ uncommitted)
Args: -d 4 -n 100
Device: baseline=4, current=4  (or baseline=4, current=6)

### tensormap_and_ringbuffer

Example                      Base (us)   HEAD (us)   Delta (us)   Change (%)
---------------------------  ---------   ---------   ----------   ----------
alternating_matmul_add         1240.1      1235.5        -4.6       -0.37%
  (host)                     480000.0    470000.0    -10000.0       -2.08%
  (device)                     9000.0      8800.0       -200.0       -2.22%
  (orch)                        830.0       820.3        -9.7       -1.17%
  (sched)                      1240.0      1235.4        -4.6       -0.37%
benchmark_bgemm                 890.3       892.1        +1.8       +0.20%
  (host)                     370000.0    370500.0      +500.0       +0.14%
  (device)                     7100.0      7080.0       -20.0       -0.28%
  (orch)                        650.0       650.2        +0.2       +0.03%
  (sched)                       890.2       892.0        +1.8       +0.20%
...

Overall: X of Y examples improved, Z regressed   (based on Effective)
```

If baseline and current ran on **different devices**, add a note:

> Note: Baseline and current ran on different NPU devices (4 vs 6). Results within ±2% may reflect device-to-device variance rather than code changes. For definitive comparison, re-run on the same device with `/benchmark -d <single_device>`.

**Interpretation:**

| Change (%) | Assessment |
| ---------- | ---------- |
| < -2% | Notable improvement |
| -2% to +2% | Within noise margin |
| > +2% | Potential regression — flag for review |

If any example shows > 5% regression, highlight it explicitly.

## Error Handling

| Error | Action |
| ----- | ------ |
| No idle device and no `-d` specified | Prompt user to specify device ID |
| Benchmark script fails | Report which examples failed; continue with remaining |
| No timing data | Warn: "No timing markers — ensure `SIMPLER_PROFILING` is enabled" |
| All examples fail | Check: did you run `pip install -e .` in the worktree venv? |
| Worktree creation fails | Fall back to stash/checkout approach or report error |
| `Pre-built runtime binaries not found` | The venv `pip install -e .` should have built these; re-run it |
| `ModuleNotFoundError: _task_interface` | Venv not activated; add `source .venv/bin/activate &&` before the command |

## Checklist

- [ ] Mode detected (single vs compare)
- [ ] Idle device found or user-specified
- [ ] PTO-ISA pinned to CI commit
- [ ] `PROJECT_ROOT` and `WORKTREE_ABS` absolute paths computed
- [ ] (Compare mode) Worktree created, venv built with `pip install -e .`
- [ ] (Compare mode) Baseline completed — venv activated, `pwd` confirmed worktree path before running
- [ ] Current completed in main workspace
- [ ] Worktree cleaned up (compare mode)
- [ ] Results table presented with Host / Device / Effective / Orch / Sched times
- [ ] (Compare mode) Device difference noted if applicable
- [ ] (Compare mode) Regressions > 2% flagged