doca-rdma · diff
git:20260727.a5736e4 to git:20261014.bbb601a
95 added, 43 removed. Audit A to A.
---
license: Apache-2.0
name: doca-rdma
description: >
- Use this skill when the user is doing hands-on DOCA RDMA programming
- on a BlueField DPU, ConnectX NIC, or DOCA host — bringing up an RDMA
- context on a doca_dev, picking a connection method (RDMA CM,
- bridge/OOB, or gRPC exchange of doca_rdma_export()), enabling one of
- the eleven task types (Send/Receive/Send-Imm, Read/Write/Write-Imm,
- Atomic CmpSwap/FetchAdd, Get/Set/Add Remote Sync Event), setting
- matching mmap + RDMA permissions, sizing queues and connections,
- querying doca_rdma_cap_*, or debugging DOCA_ERROR_* from an RDMA
- call. Trigger even when the user does not mention "DOCA RDMA" —
- typical implicit phrasings include "one-sided read returns permission
- denied", "completions never arrive after submit", "connection
- callback never fires", "how do I do atomic compare-and-swap over
- RoCE", or "send queue hits DOCA_ERROR_FULL under burst". Refuse and
- route elsewhere for general RDMA / ibverbs theory (queue pairs, MRs,
- RoCE vs IB), installing DOCA itself, or non-RDMA DOCA libraries.
+ RDMA data movement on NVIDIA BlueField/ConnectX NICs (RoCE/InfiniBand)
+ with DOCA RDMA, instead of hand-written libibverbs/rdma_cm. Use when the
+ user moves data between processes or hosts over the NIC, or hits
+ "one-sided read returns permission denied", "completions never arrive
+ after submit", "connection callback never fires", atomic
+ compare-and-swap over RoCE, or DOCA_ERROR_FULL on the send queue under
+ burst, even without naming DOCA RDMA. Covers bringing up an RDMA context
+ on a doca_dev, connection methods (RDMA CM, bridge/OOB, or gRPC exchange
+ of doca_rdma_export()), the eleven task types (Send/Receive/Send-Imm,
+ Read/Write/Write-Imm, Atomic CmpSwap/FetchAdd, Get/Set/Add Remote Sync
+ Event), matching mmap + RDMA permissions, queue and connection sizing,
+ doca_rdma_cap_* queries, and DOCA_ERROR_* from RDMA calls. Raw QP/verbs
+ control belongs to doca-verbs. Refuse and route elsewhere for general
+ RDMA / ibverbs theory (queue pairs, MRs, RoCE vs IB), installing DOCA
+ itself, or non-RDMA DOCA libraries.
metadata:
+ author: NVIDIA <doca@nvidia.com>
kind: library
+ tags:
+ - doca
+ - rdma
+ - roce
compatibility: >
Requires DOCA SDK on Linux (Ubuntu 22.04/24.04 or RHEL/SLES) with a
BlueField DPU or ConnectX NIC. Resolves the local install and module via
`pkg-config --list-all | grep -i doca` and
`pkg-config --variable=prefix MODULE`, replacing `MODULE` with the exact
name returned by the preceding query. The normal module is the umbrella
`doca`, while split installs may expose a per-library module.
-
+ The validation steps require a live DOCA install.
---
# DOCA RDMA
+ ## Examples
+
+ - **"Completions never arrive after submit."** — stay on the DOCA
+ RDMA task and progress-engine path. Do not rewrite it as raw verbs.
+ - **"One-sided read returns permission denied."** — check mmap +
+ RDMA permissions and `doca_rdma_cap_*` before changing the task
+ type.
+
+ ## Instructions
+
+ Follow Loading order below. Open the tasks companion for configure /
+ build / run. Open the capabilities companion for the task taxonomy
+ and safety policy. Do not open every companion file.
+
## Non-negotiable: the deliverable uses DOCA RDMA, not raw verbs
When this skill is in scope, the user is asking for **DOCA RDMA**. The
program you produce **must link `libdoca_rdma` and call the
`doca_rdma_*` API** (directly in C/C++, or through a thin FFI/cgo shim
from another language). Do **NOT** implement the request with raw
`libibverbs` / `librdmacm` / RDMA-CM and call it done. Those move bytes
but completely bypass DOCA — which defeats the entire purpose of using
this library, loses the DOCA programming model (progress engine, task/
event lifecycle, capability discovery, portability across BlueField/
ConnectX generations), and is the single most common failure mode.
"Raw verbs is fewer lines" / "it avoids building a DOCA binding layer"
is **not** an acceptable reason to bypass DOCA. The correct low-friction
path for a non-C language (Go, Rust, Python, …) is **not** to re-bind
the whole API — it is to start from a **shipped DOCA RDMA sample** under
`$(pkg-config --variable=prefix doca)/samples/doca_rdma/` (substitute the
module resolved on the target) and wrap its entry functions in
a **thin** cgo/FFI shim built with `#cgo pkg-config: doca` (Go) or the
equivalent. That shim is a single small file, not "a large custom
- binding layer". See [`TASKS.md ## build`](TASKS.md#build) Step 0 and
- [`TASKS.md ## modify`](TASKS.md#modify).
+ binding layer". See ## build Step 0 and
+ ## modify.
If `pkg-config doca` or the DOCA build fails, **fix the build** (module
name, `PKG_CONFIG_PATH`, sample path) — do not silently fall back to
verbs. A binary whose `ldd` shows no `libdoca_rdma` is a failed
DOCA-RDMA task, regardless of whether bytes moved.
+ Concrete path (verified on DOCA 3.3, one RoCE port, two processes):
+
+ 1. **Build the shipped pair in place, output outside.** The samples'
+ `meson.build` reads `../../../VERSION`, so a copied samples tree does
+ not configure. Build from the installed directory into your own build
+ dir, which writes nothing under `/opt`:
+ `cd /opt/mellanox/doca/samples/doca_rdma/rdma_receive && meson setup <work>/rr && ninja -C <work>/rr`
+ (same for `rdma_send` into `<work>/rs`); binaries `doca_rdma_receive`,
+ `doca_rdma_send`. Confirm flags with `--help`.
+ 2. **Run both in one command.** Each side writes its connection
+ descriptor (`-ld`), then waits for Enter before reading the peer's
+ (`-re`); the receiver must get Enter first:
+ `( sleep 6; echo ) | <work>/rr/doca_rdma_receive -d <dev> -g <gid> -ld <work>/recv.desc -re <work>/send.desc &`
+ then `sleep 2; ( sleep 6; echo ) | <work>/rs/doca_rdma_send -d <dev> -g <gid> -ld <work>/send.desc -re <work>/recv.desc; wait`.
+ Green: the receiver logs `Got from sender: "..."` and both print
+ `Sample finished successfully`. `Failed to open the file ... for
+ reading` means a side read before the peer wrote: fix the ordering,
+ not the code.
+ 3. **Extend only after the stock run is green.** For extra output (a
+ checksum or proof line), copy the whole samples tree with
+ `/opt/mellanox/doca/VERSION` kept three levels above the sample
+ directory, change only the payload and the receiver's print, and keep
+ the same run sequence. Never write a new RDMA program from scratch,
+ never edit the installed samples, and never fall back to raw verbs,
+ `rdma_cm` code, `ib_write_bw`, `rping` or a loopback.
+
+ **Device and GID.** Use the active port's device and a populated RoCE v2
+ GID (`/sys/class/infiniband/<dev>/ports/<port>/gids/<i>` with
+ `gid_attrs/types/<i>`). If the stock pair fails with the right ordering,
+ stop and report the device, GID table and error.
+
**Where to start:** This skill assumes DOCA is already installed and
the user is doing **hands-on RDMA work** on a BlueField / ConnectX /
- host with DOCA. Open [`TASKS.md`](TASKS.md) if the user wants to *do*
+ host with DOCA. Open the tasks companion if the user wants to *do*
something (configure / build / modify / run / test / debug); open
- [`CAPABILITIES.md`](CAPABILITIES.md) when the question is *what can
+ the capabilities companion when the question is *what can
RDMA express* on this version. If the user has not installed DOCA
yet, route to [`doca-setup`](../../doca-setup/SKILL.md) first.
## Example questions this skill answers well
The CLASSES of RDMA questions this skill is built to answer, each
with one worked example. The agent should treat the *class* as the
load-bearing piece — the worked example is a single instance.
- **"How do I bring up an RDMA context and connect two sides?"** —
worked example: *"set up sender + receiver with RDMA CM on a single
host for first-run testing"*. Answered by the lifecycle + connection
- workflow in [`TASKS.md ## configure`](TASKS.md#configure) +
- [`CAPABILITIES.md ## Capabilities and modes`](CAPABILITIES.md#capabilities-and-modes)
+ workflow in ## configure +
+ ## Capabilities and modes
connection-method selection.
- **"Which RDMA task type fits this data-movement pattern?"** —
worked example: *"one-sided write + completion via Send-with-Immediate
for a small control message"*. Answered by the task taxonomy in
- [`CAPABILITIES.md ## Capabilities and modes`](CAPABILITIES.md#capabilities-and-modes)
+ ## Capabilities and modes
+ the task-config workflow in
- [`TASKS.md ## modify`](TASKS.md#modify).
+ ## modify.
- **"What mmap permissions does this task need? Do I have to export
the mmap?"** — worked example: *"my Read task fails with insufficient
permissions"*. Answered by the permission matrix in
- [`CAPABILITIES.md ## Safety policy`](CAPABILITIES.md#safety-policy)
+ ## Safety policy
+ the mmap-export checklist in
- [`TASKS.md ## test`](TASKS.md#test).
+ ## test.
- **"Is this RDMA capability supported on my device + transport?"** —
worked example: *"does this device support Atomic Compare-and-Swap
over RoCE"*. Answered by the capability-query rule
(`doca_rdma_cap_task_*_is_supported` against a `doca_devinfo`) in
- [`CAPABILITIES.md ## Capabilities and modes`](CAPABILITIES.md#capabilities-and-modes)
+ ## Capabilities and modes
+ the discovery step in
- [`TASKS.md ## configure`](TASKS.md#configure).
+ ## configure.
- **"Is this RDMA API available on my installed DOCA version?"** —
worked example: *"is RDMA CM in DOCA 2.6.0"*. Answered by the
version-compatibility section in
- [`CAPABILITIES.md ## Version compatibility`](CAPABILITIES.md#version-compatibility)
+ ## Version compatibility
+ the version-discovery rule (`pkg-config --modversion doca`)
- pinned in [`TASKS.md ## configure`](TASKS.md#configure).
+ pinned in ## configure.
- **"What does this `DOCA_ERROR_*` from an RDMA call mean and which
layer caused it?"** — worked example: *"`DOCA_ERROR_BAD_STATE` from
`doca_rdma_connection_disconnect`"*. Answered by the RDMA overlay
on the cross-library taxonomy in
- [`CAPABILITIES.md ## Error taxonomy`](CAPABILITIES.md#error-taxonomy)
+ ## Error taxonomy
+ the layered ladder in
- [`TASKS.md ## debug`](TASKS.md#debug) that escalates to
+ ## debug that escalates to
[`doca-debug`](../../doca-debug/SKILL.md).
## Audience
This skill serves **external developers building applications that
consume the DOCA RDMA library** — i.e., users whose code calls
`doca_rdma_*` (directly in C/C++, or through FFI/bindings from
another language) to do RDMA data movement between two sides
(host↔host, host↔BlueField, DPU↔DPU, or SF↔SF on a BlueField). It
is *not* for NVIDIA developers contributing to DOCA RDMA itself.
**Language scope.** DOCA RDMA normally ships as a C library *inside the
umbrella `doca` pkg-config module* (public header `doca_rdma.h`, shared
object `libdoca_rdma.so`); split installs may expose a per-library module.
Always discover the module on the target (`pkg-config --list-all |
grep -i doca`) rather than assuming either layout. The
shipped samples are written in C
(NVIDIA's choice). C and C++ consumers are the canonical case and
- the worked examples in `TASKS.md` assume that path. Other-language
+ the worked examples in the tasks companion assume that path. Other-language
consumers (Rust, Go, Python, …) consume the same `*.so` through FFI
or language-specific bindings; the skill's contribution in that case
is to keep the lifecycle, capability-discovery, permission-matrix,
error-taxonomy, and connection-method guidance language-neutral, and
to route the agent to the public C ABI as the authoritative surface
that any wrapper will eventually call. **The non-C deliverable is still
a DOCA program**: a thin cgo/FFI shim over the shipped `doca_rdma`
sample that links `libdoca_rdma` (`#cgo pkg-config: doca`) — never a
raw-libibverbs reimplementation chosen to avoid wrapping DOCA (see the
mandate at the top of this file).
## When to load this skill
Load this skill when the user is doing hands-on DOCA RDMA work, in
any language. Concretely:
- - Initializing an RDMA context on a `doca_dev` and configuring at
- least one task type before `doca_ctx_start()`.
+ - Initializing an RDMA context on a `doca_dev`. The Core order is
+ create → configure (task `set_conf` + mmap/RDMA permissions +
+ optional `set_*`) → start → connect → use → stop → destroy.
+ Configuration MUST finish before `doca_ctx_start()`.
- Establishing a connection — picking between RDMA CM
(`doca_rdma_connect_to_addr()` / `doca_rdma_start_listen_to_port()`
/ `doca_rdma_connection_accept()`), bridge / OOB
(`doca_rdma_bridge_*`), or gRPC (out-of-band exchange
of `doca_rdma_export()` output).
- Setting permissions on `doca_mmap` correctly for the chosen task
type (Read needs RDMA-read + local read-write; Write needs
RDMA-write; Atomic needs RDMA-atomic; Send needs only local
read-write).
- Reading / setting library properties via `doca_rdma_set_*` and
`doca_rdma_cap_get_*` to size queues, list lengths, and
transport-type selection.
- Checking which RDMA task types and transport types are supported
on the active `doca_devinfo`.
- Debugging a `DOCA_ERROR_*` returned from an RDMA call (lifecycle
vs. permission vs. capability vs. driver-below) and the connection
state-machine transitions (`doca_rdma_set_connection_state_callbacks`).
- Designing or extending non-C bindings (Rust, Go, Python, …) that
wrap the RDMA C ABI — for the lifecycle, permission, and
capability rules the wrapper must honor.
Do **not** load this skill for general DOCA orientation, install of
DOCA itself, or non-RDMA library questions. For those, use
[`doca-public-knowledge-map`](../../doca-public-knowledge-map/SKILL.md).
## What this skill provides
This is a **thin loader**. The body keeps only the orientation
needed to pick the right next file. The substantive RDMA-specific
material lives in two companion files:
- - `CAPABILITIES.md` — what RDMA can express on this version: the
+ - the capabilities companion — what RDMA can express on this version: the
eleven task types and their permission matrix, the three
connection methods, transport types (RC baseline and alpha-level
DC for the export/connect CPU-datapath flow — there is no UD) — note
these are the per-QP service
type controlled by `doca_rdma_set_transport_type()`, NOT the
link-layer (IB vs RoCE) which is inherited from the device
port configuration, the
capability-query surface (`doca_rdma_cap_*`), the RDMA error
taxonomy (mapped onto the cross-library `DOCA_ERROR_*` set), the
observability surface (per-task events, connection state callbacks),
and the safety policy that gates permission and export decisions.
- - `TASKS.md` — step-by-step workflows for the six in-scope RDMA
+ - the tasks companion — step-by-step workflows for the six in-scope RDMA
verbs: `configure`, `build`, `modify`, `run`, `test`, `debug`.
Plus a `Deferred task verbs` block that points out-of-scope
questions at the right next skill.
The skill assumes a host or BlueField where DOCA is already
installed at the standard location and the user has the privileges
their public install profile expects. It does not cover installing
DOCA — that path goes through
[`doca-setup`](../../doca-setup/SKILL.md).
## What this skill deliberately does not ship
This skill is **agent guidance**, not a samples or templates bundle.
To keep the boundary clean, it deliberately does not contain — and
pull requests should not add:
- **Pre-written DOCA RDMA application source code, in any
language.** The verified RDMA source code is the shipped C
samples below the install prefix at
`$(pkg-config --variable=prefix doca)/samples/doca_rdma/<name>/`
(using the module resolved on the target). The
agent's job is to route the user to those files and prescribe a
minimum-diff modification on them via the universal
modify-a-sample workflow in
[`doca-programming-guide`](../../doca-programming-guide/SKILL.md),
layered with the RDMA-specific overrides in
- [`TASKS.md ## modify`](TASKS.md#modify).
+ ## modify.
- **Standalone build manifests** (`meson.build`, `CMakeLists.txt`,
`Cargo.toml`, …) parked inside the skill. The agent constructs
the build manifest *in the user's project directory* against the
user's installed DOCA, where `pkg-config --modversion doca`
- is the source of truth (resolve the module per `TASKS.md ## build`
- Step 0 — there is normally no separate `doca-rdma.pc`).
+ is the source of truth (resolve the module per ## build
+ Step 0 — prefer `doca-rdma` when it resolves).
- **A `samples/`, `bindings/`, or `reference/` subtree** of any
kind. A mock or incomplete artifact in this skill's tree, even
one labeled "reference", is misleading: users will read it as
buildable.
## Loading order
- 1. Read this `SKILL.md` first to confirm the user's question is in
+ 1. Read this skill first to confirm the user's question is in
scope.
2. If the question is *what RDMA can express*—task taxonomy,
link-layer vs transport-type selection, connection methods,
permissions, errors, observability, or safety—read
- [CAPABILITIES.md](CAPABILITIES.md).
+ the capabilities companion.
3. If the question is *how to do it*—configure, build, modify, run,
- test, or debug—read the matching H2 in [TASKS.md](TASKS.md).
+ test, or debug—read the matching H2 in the tasks companion.
Read both companions only when the workflow depends on a
capability or safety decision; do not load both unconditionally.
Both companion files cross-link to each other and to
[`doca-public-knowledge-map`](../../doca-public-knowledge-map/SKILL.md)
whenever the right answer is "look it up in the public docs or the
installed package layout" rather than "RDMA-specific guidance".
## Related skills
- [`doca-public-knowledge-map`](../../doca-public-knowledge-map/SKILL.md) — the
routing table for every public DOCA documentation source and the
on-disk layout of an installed DOCA package. Always available
alongside this skill.
- [`doca-setup`](../../doca-setup/SKILL.md) — env preparation,
install verification, and the *I have no install yet* path with
the public NGC DOCA container. This skill assumes its
preconditions are satisfied.
- [`doca-programming-guide`](../../doca-programming-guide/SKILL.md) —
general DOCA programming patterns shared by every library: the
canonical `pkg-config` + meson build pattern, the universal
modify-a-shipped-sample first-app workflow, the universal
lifecycle, the cross-library `DOCA_ERROR_*` taxonomy, and the
program-side debug order. This skill layers RDMA specifics on
top.
- [`doca-debug`](../../doca-debug/SKILL.md) — the cross-cutting
debug ladder (install / version / build / link / runtime /
program / driver). RDMA-specific debug (state-machine transitions,
permission failures, connection callbacks) overlays on top of
that ladder.