doca-gpunetio-ib-write-bw · diff
git:20260727.a5736e4 to git:20261014.bbb601a
58 added, 38 removed. Audit A to A.
---
license: Apache-2.0
name: doca-gpunetio-ib-write-bw
description: >
- Use this skill when the user is building, running, or interpreting
- the doca/tools/gpunetio_ib_write_bw client+server benchmark — a CUDA
+ Use this skill when the user is building, running, or interpreting the
+ doca/tools/gpunetio_ib_write_bw client+server benchmark — a CUDA
kernel on the client posts RDMA WRITE work requests through the
doca-gpunetio device-side surface to measure sustained GPU-driven
WRITE bandwidth on a GPU+IB-device pair. Trigger even when the user
- does not explicitly mention "doca-gpunetio-ib-write-bw" or
- "GPUNetIO" — typical implicit phrasings include "measure WRITE BW
- when the GPU posts the WRs", "BW swings between runs on the same
- flags", "is the NIC saturated or am I CPU-bound on the CUDA
- kernel", "meson compile fails for the GPUNetIO bw tool",
- "nvidia_peermem isn't picking up my GPU buffer", or "GPU-initiated
- WRITE throughput vs CPU-initiated perftest". Refuse and route
- elsewhere for general doca-gpunetio library work, DOCA install, the
- GPU-initiated WRITE latency analog, the CPU-initiated upstream
- perftest, or application-level end-to-end throughput — those belong
- to other skills.
+ does not explicitly mention "doca-gpunetio-ib-write-bw" or "GPUNetIO"
+ — typical implicit phrasings include "measure WRITE BW when the GPU
+ posts the WRs", "BW swings between runs on the same flags", "is the
+ NIC saturated or am I CPU-bound on the CUDA kernel", "meson compile
+ fails for the GPUNetIO bw tool", "nvidia_peermem isn't picking up my
+ GPU buffer", or "GPU-initiated WRITE throughput vs CPU-initiated
+ perftest". Refuse and route elsewhere for general doca-gpunetio
+ library work, DOCA install, the GPU-initiated WRITE latency analog,
+ the CPU-initiated upstream perftest, or application-level end-to-end
+ throughput — those belong to other skills.
metadata:
+ author: NVIDIA <doca@nvidia.com>
kind: tool
+ tags:
+ - doca
+ - gpunetio
+ - bandwidth
compatibility: >
Requires DOCA SDK on Linux with a BlueField DPU or ConnectX NIC,
NVIDIA GPU, CUDA toolkit and nvcc, loaded `nvidia_peermem`, and an
InfiniBand RNIC paired with the GPU. Uses `pkg-config` for
doca-gpunetio, doca-rdma, and doca-common, plus the installed
gpunetio_ib_write_bw sources. Run only on a trusted, non-shared IB
- fabric during the benchmark window.
+ fabric during the benchmark window. The validation steps require a
+ live DOCA install.
---
# DOCA GPUNetIO ib_write_bw
+ ## Examples
+
+ - **"What sustained RDMA-WRITE bandwidth can GPUNetIO deliver on
+ this GPU-NIC pair?"** — this is in scope. Stay on
+ `gpunetio_ib_write_bw`; do not retune a live GPI app here.
+ - **"Where is the bottleneck — GPU occupancy, NIC issue rate, or
+ link saturation?"** — decompose throughput before changing
+ the CUDA kernel.
+
+ ## Instructions
+
+ Follow Loading order below. Open the tasks companion for configure /
+ run / debug. Open the capabilities companion for GPU-NIC pairing
+ and observability. Do not open every companion file.
+
**Where to start:** This is a tool skill for the GPUNetIO-
flavored `ib_write_bw` benchmark shipped under
`doca/tools/gpunetio_ib_write_bw/` (a client + server pair,
built from source against the installed DOCA via `meson`).
It measures sustained RDMA WRITE bandwidth when the WRs are
posted **from a CUDA kernel through the doca-gpunetio
device-side surface**, with the GPU on the data path. Open
- [`TASKS.md`](TASKS.md) and start at
- [`## configure`](TASKS.md#configure) for the GPU-NIC
+ the tasks companion and start at
+ `## configure` for the GPU-NIC
pairing precondition and the build pattern; jump to
- [`## run`](TASKS.md#run) for the smoke-before-bulk flow.
- Open [`CAPABILITIES.md`](CAPABILITIES.md) when the question
+ `## run` for the smoke-before-bulk flow.
+ Open the capabilities companion when the question
is *what this tool actually measures*, *how the result
decomposes (GPU occupancy vs NIC issue rate vs link
saturation)*, or *how the result reads against the GPI
sister tool and the upstream CPU-initiated `perftest`
`ib_write_bw`*. If DOCA is not installed yet, route to
[`doca-setup`](../../doca-setup/SKILL.md) first; if the
user is still deciding between the GPI and GPUNetIO
programming surfaces, the picture in
- [`../../libs/doca-gpunetio/CAPABILITIES.md#capabilities-and-modes`](../../libs/doca-gpunetio/CAPABILITIES.md#capabilities-and-modes)
+ [`doca-gpunetio`](../../libs/doca-gpunetio/SKILL.md)
and
- [`../../libs/doca-gpi/CAPABILITIES.md#capabilities-and-modes`](../../libs/doca-gpi/CAPABILITIES.md#capabilities-and-modes)
+ [`doca-gpi`](../../libs/doca-gpi/SKILL.md)
is the first stop.
## Example questions this skill answers well
The CLASSES of `doca-gpunetio-ib-write-bw` questions this
skill is built to answer, each with one worked example. The
class is the load-bearing piece; the worked example is one
instance.
- **"What sustained RDMA-WRITE bandwidth can the GPUNetIO
path deliver on this GPU-NIC pair?"** — worked example:
*"measure sustained WRITE BW between two hosts with an
H100 + ConnectX-7 on each side"*. Answered by the
GPU-NIC pairing precondition in
- [`CAPABILITIES.md ## Capabilities and modes`](CAPABILITIES.md#capabilities-and-modes)
+ the capabilities companion
+ the bring-up flow in
- [`TASKS.md ## configure`](TASKS.md#configure) +
- [`TASKS.md ## run`](TASKS.md#run). The same shape
+ the tasks companion +
+ the tasks companion. The same shape
answers *"measure GPUNetIO-driven WRITE BW between a
host GPU and a BlueField DPU"*.
- **"Where is the bottleneck — GPU compute occupancy, NIC
issue rate, or link saturation?"** — worked example:
*"I see 120 Gbit/s on a 200 Gbit/s link; is the NIC
saturated, am I CPU-bound on the client, or is the CUDA
kernel not driving enough WRs in flight?"*. Answered by
the throughput-decomposition rules in
- [`CAPABILITIES.md ## Observability`](CAPABILITIES.md#observability)
+ the capabilities companion
+ the eval-loop overlay in
- [`TASKS.md ## test`](TASKS.md#test).
+ the tasks companion.
- **"How does the result differ from the classic CPU-
initiated `perftest` `ib_write_bw`?"** — worked example:
*"my team has a CPU-initiated WRITE BW number on this
same NIC; should I expect the GPUNetIO number to match
or be different?"*. Answered by the *"GPU-initiated
path adds (or removes) overhead vs the CPU-initiated
path"* rule in
- [`CAPABILITIES.md ## Capabilities and modes`](CAPABILITIES.md#capabilities-and-modes).
+ the capabilities companion.
- **"Is the doca-gpunetio path the right surface for my
sustained-throughput workload class?"** — worked example:
*"my application streams sensor data from GPU memory at
line rate to a remote consumer"*. Answered by the
*"when GPUNetIO is the right surface vs GPI vs CPU-
initiated"* rule in
- [`CAPABILITIES.md ## Capabilities and modes`](CAPABILITIES.md#capabilities-and-modes)
- + the use-side decision in [`TASKS.md ## use`](TASKS.md#use).
+ the capabilities companion
+ + the use-side decision in the tasks companion.
- **"My BW number swings between runs. What do I check
before quoting it?"** — worked example: *"three runs at
the same flags gave 145, 187, and 160 Gbit/s; is the
benchmark noisy or is my platform inconsistent?"*.
Answered by the measurement-soundness rules in
- [`CAPABILITIES.md ## Error taxonomy`](CAPABILITIES.md#error-taxonomy)
+ the capabilities companion
layer 5 + the steady-state guidance in
- [`TASKS.md ## test`](TASKS.md#test).
+ the tasks companion.
- **"What version of DOCA + CUDA Toolkit do I need for this
binary to build and run?"** — worked example: *"my
install has DOCA at one semver and CUDA at another; will
the ToT-shipped `gpunetio_ib_write_bw` even link?"*.
Answered by the version overlay in
- [`CAPABILITIES.md ## Version compatibility`](CAPABILITIES.md#version-compatibility)
+ the capabilities companion
which cross-links the canonical detection chain in
[`doca-version`](../../doca-version/SKILL.md).
## Audience
This skill serves **external developers and performance
engineers who need a reproducible measurement of sustained
RDMA WRITE bandwidth when the WRs are posted from a CUDA
kernel through doca-gpunetio**, on the user's actual install
and GPU-NIC pair. Concretely:
- A developer comparing the GPUNetIO path against the GPI
path or the host-initiated `perftest`-style path before
committing an application design to one of them.
- A platform operator validating a tuning change (NUMA
pinning, GPU PCIe placement, IB device choice, GID
index, NIC firmware burn) by re-running this benchmark
against the new state.
- An SRE / performance engineer producing a *"this is the
GPUNetIO-driven WRITE BW on this GPU-NIC pair today"*
artifact downstream consumers can cite.
- An AI agent answering *"is the doca-gpunetio path a win
for my sustained-throughput workload class"* honestly —
with a measured number, the build + invocation that
produced it, and the GPU + NIC + DOCA version that
scopes it — rather than guessing from datasheet
headlines.
It is **not** for users debugging the `doca-gpunetio`
library itself (route to
[`../../libs/doca-gpunetio/SKILL.md`](../../libs/doca-gpunetio/SKILL.md)),
and **not** a substitute for the `perftest` upstream
`ib_write_bw` (which measures CPU-initiated WRITE BW).
## Language scope
The `doca-gpunetio-ib-write-bw` tool is shipped as **C plus
a CUDA `.cu` translation unit** under
`doca/tools/gpunetio_ib_write_bw/`, split into a `client/`
subtree and a `server/` subtree. The verified surface (per
`client/{main.c,common.h,common.c,kernel.cu,perftest.c}` and
`server/{main.c,common.h,common.c,perftest.c}`): host-side
build via `meson` against the installed DOCA `pkg-config`
modules (`doca-gpunetio`, `doca-rdma`, `doca-common`); the
device-side build via `nvcc` against the DOCA GPU NetIO
device-side header set; the OOB descriptor exchange via a
TCP socket between client and server. There is no Python /
Rust / Go binding — the tool is a pair of CLI binaries.
The skill's job is to keep the operator-side workflow
language-neutral; the device-side CUDA surface is not
wrappable in another language.
## When to load this skill
Load this skill when the user is — or the agent needs to —
build and run the `gpunetio_ib_write_bw` client + server on
real hosts with DOCA installed plus a CUDA Toolkit matched
to the DOCA install, and a GPU + IB device pair on the
host's PCIe topology. Concretely:
- Measuring sustained kernel-initiated RDMA WRITE
bandwidth between two hosts (or a host and a BlueField
DPU) with the GPUNetIO surface.
- Deciding whether the GPUNetIO path is the right runtime
surface for a class of workload vs the GPI programming
surface (the [`doca-gpi`](../../libs/doca-gpi/SKILL.md)
library — `doca/tools/` ships no GPI benchmark binary) or
the classic CPU-initiated `perftest` path.
- Capturing a documented baseline (build + invocation +
DOCA version + GPU + NIC + as-deployed environment +
numbers) for later regression hunts.
- Diagnosing a build / link / run failure that surfaces
the GPUNetIO + RDMA bring-up sequence under this tool's
shipped scaffolding.
Do **not** load this skill for general DOCA orientation,
library API work, or installation. For those, use
[`doca-public-knowledge-map`](../../doca-public-knowledge-map/SKILL.md),
[`../../libs/doca-gpunetio/SKILL.md`](../../libs/doca-gpunetio/SKILL.md),
or [`doca-setup`](../../doca-setup/SKILL.md). Do not load
it for *application-level* end-to-end throughput either —
this benchmark measures the WR-submission path through
GPUNetIO, not the user's full pipeline.
## What this skill provides
This is a **thin loader**. Substantive material lives in
two companion files:
- - `CAPABILITIES.md` — what the tool measures (the
+ - the capabilities companion — what the tool measures (the
sustained-WRITE-BW primitive driven by a client-side
CUDA kernel through doca-gpunetio), the
runtime-surface selection rule (GPUNetIO vs GPI vs
CPU-initiated), the GPU-NIC pairing precondition, the
throughput-decomposition guide (GPU compute occupancy
vs NIC issue rate vs link saturation), the version
overlay (DOCA `.pc` PLUS CUDA Toolkit), the layered
error taxonomy (config-syntax / build-time / GPU-NIC-
pairing / GPUNetIO-lifecycle / RDMA-connection /
measurement-soundness / version / cross-cutting), the
observability surface (stdout report, DOCA log levels,
OOB-socket exchange), and the safety overlay (the
*"GPU-side handle is a credential"* rule from
doca-gpunetio; the cross-cutting hardware-safety
meta-policy).
- - `TASKS.md` — step-by-step workflows for the in-scope
+ - the tasks companion — step-by-step workflows for the in-scope
task verbs: `install` (preconditions — DOCA install,
CUDA Toolkit, GPU + NIC pair, OOB connectivity),
`configure` (build-tree under
`doca/tools/gpunetio_ib_write_bw/` and the `meson`
build wrapping the shipped DOCA), `build` (the
`meson setup` + `meson compile` pattern from the
public DOCA build documentation), `modify` (do not
patch the shipped tool source; modify the invocation
and the surrounding environment instead), `run` (smoke-
before-bulk; client + server bring-up order; reading
the per-iteration report), `test` (the eval loop —
steady-state, NUMA placement, NIC saturation cross-
check), `debug` (walk the error taxonomy layer by
layer), `use` (how a BW result feeds a class-of-
workload decision), plus a `Deferred task verbs`
block routing out-of-scope questions.
The skill assumes a host where DOCA is already installed,
a CUDA Toolkit matched to the install is present, and the
operator has whatever privileges the public install profile
expects for binding a `doca_dev`, a `doca_gpu`, and an OOB
TCP socket.
## What this skill deliberately does not ship
This skill is **agent guidance**, not a samples or scripts
bundle. To keep the boundary clean, it deliberately does
not contain — and pull requests should not add:
- **Specific flag strings or expected throughput numbers**
beyond what the tool's shipped `--help` and `main.c` ARGP
registration establish. The flag surface is small
(device name, GPU PCIe address, GID index, server IP on
the client side); the agent re-reads the binary's
`--help` on the installed version before quoting flag
strings. Throughput numbers are device-, firmware-,
version-, and topology-specific.
- **Pre-written DOCA GPUNetIO or CUDA kernel source code**
that would compete with the shipped tool tree. The
shipped `client/{main.c,kernel.cu,perftest.c,common.{c,h}}`
and `server/{main.c,perftest.c,common.{c,h}}` files are
the verified worked example; the agent's job is to
route the user there and prescribe minimum-diff
modification per the universal modify-a-sample workflow
in
[`doca-programming-guide`](../../doca-programming-guide/SKILL.md).
- **Wrappers, parsers, or scripts** in any language that
consume the tool's stdout. The output format is small
and documented in
- [`CAPABILITIES.md ## Observability`](CAPABILITIES.md#observability);
+ the capabilities companion;
if the user wants to script against it, the right
answer is *"read the live source, write the parser
against your installed binary"*.
- **A `samples/`, `bindings/`, or `reference/` subtree.**
This is a thin loader for a shipped tool tree;
substantive material lives in the source tree and in
the GPUNetIO library docs.
## Loading order
- 1. Read this `SKILL.md` first to confirm the user's
+ 1. Read this skill first to confirm the user's
question is in scope (the user actually wants to
measure sustained kernel-initiated WRITE BW through
GPUNetIO, not learn GPUNetIO as a library or do a
CPU-initiated measurement).
2. **For what the tool measures, the surface-selection
rule against the GPI sister tool and the CPU-initiated
`perftest`, the throughput-decomposition guide, the
version overlay, the error taxonomy, the observability
surface, and the safety overlay, see
- [CAPABILITIES.md](CAPABILITIES.md).**
+ the capabilities companion.**
3. **For step-by-step workflows — `install`, `configure`,
`build`, `modify`, `run`, `test`, `debug`, `use` — see
- [TASKS.md](TASKS.md).**
+ the tasks companion.**
## Related skills
- [`../../libs/doca-gpunetio/SKILL.md`](../../libs/doca-gpunetio/SKILL.md) —
the library this tool wraps. The per-GPU `doca_gpu`
context, the GPU-visible `doca_gpu_eth_*` and RDMA-side
handles, the CUDA-side persistent-kernel pattern, the
dual capability-discovery rule (DOCA cap-query AND
`cudaGetDeviceProperties`), and the env preconditions
(`nvidia_peermem` loaded, CUDA buffers registered with
DOCA) live there.
- [`../../libs/doca-rdma/SKILL.md`](../../libs/doca-rdma/SKILL.md) —
the underlying RDMA library. The RDMA queue this tool
binds is created and connected via `doca-rdma`; the
queue lifecycle, transport type (RC vs UC vs UD),
permission matrix, and connection method are owned
there.
- [`../../libs/doca-verbs/SKILL.md`](../../libs/doca-verbs/SKILL.md) —
the raw-verbs escape hatch beneath `doca-rdma` /
`doca-gpunetio`. This tool stays on the higher-level
surfaces; `doca-verbs` is the right place only if the
user needs a specific WR flag / QP attribute the
GPUNetIO + RDMA surfaces do not expose.
- [`../doca-gpunetio-ib-write-lat/SKILL.md`](../doca-gpunetio-ib-write-lat/SKILL.md) —
the latency analog of this tool. Same physical
operation; same runtime framework; different metric
class (BW vs latency). The two together carry the
full GPUNetIO-side throughput / latency picture.
- [`doca-gpi`](../../libs/doca-gpi/SKILL.md) — the GPI
programming surface (CUDA-kernel-initiated RDMA), the
alternative runtime framework for the same physical
operation. `doca/tools/` ships no GPI `ib_write_lat` /
`ib_write_bw` benchmark binary, so the GPI comparison is
against the library surface, not a sibling tool. The
selection rule in
- [`CAPABILITIES.md ## Capabilities and modes`](CAPABILITIES.md#capabilities-and-modes)
+ the capabilities companion
is the decision aid.
- [`doca-version`](../../doca-version/SKILL.md) — the
canonical version-detection chain, four-way match rule,
NGC container semantics, and headers-win-over-docs
rule. The `## Version compatibility` section in this
skill is a thin overlay; the body lives there.
- [`doca-setup`](../../doca-setup/SKILL.md) — env
preparation, install verification, GPU + CUDA Toolkit
pairing, `nvidia_peermem` load, hugepages, NUMA, and
the *I have no install yet* path with the public NGC
DOCA container.
- [`doca-public-knowledge-map`](../../doca-public-knowledge-map/SKILL.md) —
routing to the public DOCA documentation set (DOCA GPU
NetIO, DOCA RDMA pages on `docs.nvidia.com`) and the
`docs.nvidia.com/cuda/` pointer for the CUDA Toolkit.
- [`doca-debug`](../../doca-debug/SKILL.md) — the
cross-cutting debug ladder. The tool surfaces its own
error taxonomy; when the cause is below DOCA, the
taxonomy hands off here.
- [`doca-hardware-safety`](../../doca-hardware-safety/SKILL.md) —
the bundle-wide hardware-safety meta-policy. The
`## Safety policy` overlay cross-links it.