ceph-performance · diff

git:20260922.1fe1a44 to git:20260922.6f1744d

4 added, 9 removed. Audit A to A.

---

name: ceph-performance

description: >-

Tune and benchmark a healthy Ceph cluster, with Rook-on-Kubernetes

specifics: mClock profiles and the NVMe capacity-fallback trap

(osd_mclock_max_capacity_iops), osd_memory_target from pod requests,

BlueStore allocation and cache defaults, PG autoscaler targets

(mon_target_pg_per_osd, bulk, target_size_ratio), public/cluster network

and Multus choices, msgr2 modes, Tentacle Fast EC (allow_ec_optimizations,

stripe_unit), krbd vs rbd-nbd in ceph-csi, librbd cache, and benchmark

method (rados bench, rbd bench, fio). Squid 19.2 and Tentacle 20.2.

when_to_use: >-

Use when making a working Ceph cluster faster or sizing one: "ceph

performance", "tune ceph", "mclock profile", "osd_mclock_max_capacity_iops",

"NVMe OSD slow", "IOPS lower than expected", "osd_memory_target", "OSD

memory sizing rook", "bluestore_min_alloc_size", "pg autoscaler",

"mon_target_pg_per_osd", "how many PGs", "bulk pool", "cluster network",

"public network", "multus ceph", "host network rook", "msgr2 secure mode

overhead", "erasure coding performance", "allow_ec_optimizations", "fast

EC", "stripe_unit", "crimson", "seastore", "krbd vs rbd-nbd", "rbd cache",

"rados bench", "fio rbd", "benchmark ceph". NOT for a degraded or

erroring cluster (slow recovery, slow ops, full OSDs) — that is

ceph-troubleshooting; upgrades are rook-ceph-best-practices; RGW/S3

behaviour is ceph-s3.

---

# ceph-performance

Defaults read from `src/common/options/*.yaml.in` at v19.2.6 and v20.2.4,

docs.ceph.com (tentacle pages) and Rook docs at v1.20.7, on **2026-09-23**.

Measure before and after every change; `ceph config show osd.N <opt>` is

the value actually in force.

## mClock: the NVMe capacity trap

mClock sizes every reservation from each OSD's measured IOPS capacity. At

OSD start an `osd bench` sets `osd_mclock_max_capacity_iops_{hdd,ssd}`.

If the result exceeds the sanity threshold — **500 (HDD), 80 000 (SSD)** —

it is discarded and the OSD falls back to the default capacity

(**21 500** SSD) with a cluster-log warning. Modern NVMe routinely exceeds

80 000, so mClock under-drives it.

1. `ceph config show osd.N osd_mclock_max_capacity_iops_ssd` on every OSD;

`21500` means the fallback hit.

2. Measure the device with fio at 4 KiB random write, then

`ceph config set osd.N osd_mclock_max_capacity_iops_ssd <iops>`.

3. Re-check after replacing a device — the value is per OSD.

Profiles: `balanced` is the default from 17.2.7 and in Reef, Squid and

Tentacle (17.2.0–17.2.6 shipped `high_client_ops`). Use `high_client_ops`

for latency-sensitive steady state, `high_recovery_ops` only for a

maintenance window.

## Memory

- Under Rook, Ceph reads `POD_MEMORY_REQUEST` and uses it 1:1 as

`osd_memory_target`; with only a limit, target = limit ×

`osd_memory_target_cgroup_limit_ratio` (0.8). Set requests and limits

explicitly on `spec.resources.osd`; the limit needs headroom above the

request because the target is a cache-trim goal, not a cap. Confirm with

`ceph config show osd.N osd_memory_target` after the pods restart.

- Rook's own minimums (warns below): osd 2048 MB, mon 1024 MB, mgr 512 MB.

4 GiB+ per OSD is the Ceph default target; NVMe OSDs benefit from more.

- `bluestore_cache_autotune` is on by default; `osd_memory_target_autotune`

(cephadm) is off and irrelevant under Rook.

- `mds_cache_memory_limit` defaults to 4 GiB; the MDS uses more than this —

size its pod limit well above it.

## BlueStore allocation

`bluestore_min_alloc_size_hdd` and `_ssd` are both **4 KiB**. The "64 KiB

on HDD" advice is obsolete. The value is baked in at OSD creation; changing

it needs the OSD redeployed.

## PG autoscaler

| Knob | Default | Guidance |

|---|---|---|

| `mon_target_pg_per_osd` (mgr option) | 100 | Docs recommend **200** for all but the smallest clusters; above 500 costs peering and RAM |

| pool `bulk` flag | false | Set on pools that will hold most of the data so they start with a full PG count instead of growing |

| `target_size_ratio` | unset | Set on each large pool; otherwise the autoscaler sizes from current usage only |

| `pg_num_min` / `pg_num_max` | unset | Bound pools the autoscaler keeps shrinking |

## Network

- A separate cluster network is **optional**: current docs say public-only

"functions just fine … especially with 25GE or faster"; add one for high

client traffic or ≤10 GE links.

- Rook: host networking removes pod-network latency but gives no isolation;

Multus gives isolation — use CNI `macvlan` with `whereabouts` IPAM (the

documented path). Validate with `kubectl rook-ceph multus validation run`.

- msgr2 mode defaults are `crc secure` for cluster, service and client:

CRC is used unless a peer requires secure. Forcing `secure` everywhere

encrypts all traffic and costs CPU/throughput — benchmark before and after.

## Erasure coding (Tentacle)

- New pools default to the **ISA-L** plugin (Jerasure before); existing

pools keep their plugin on upgrade.

- Fast EC: `ceph osd pool set <pool> allow_ec_optimizations true` — partial

reads/writes, parity-delta writes; 2–3× small-I/O gains in Ceph's own

benchmark. **Irreversible** for the pool. Works with Jerasure and ISA-L

`reed_sol_van`.

- `stripe_unit` is fixed at pool creation: 16 KiB for general workloads,

up to 256 KiB for read-heavy; the old 4 KiB default gains less. Create a

new pool rather than flipping the flag on an old 4 KiB one.

- Do not enable it on 20.2.0: turning it on for a pool that already had

`ec_overwrites` crash-looped OSDs (`interval_set.h:365`).

- Replicated size 3 still outperforms EC; EC wins on cost per byte.

## Crimson / SeaStore

Tentacle docs: "tech preview stage and is not suitable for production use".

Do not recommend it for production.

## Clients (ceph-csi)

- RBD PVCs use **krbd** by default. `mounter: rbd-nbd` is Alpha, not for

production. CephFS uses the kernel client by default, ceph-fuse on request.

- librbd cache settings (`rbd_cache` true, `rbd_cache_policy`

`writearound`, `rbd_readahead_max_bytes` 512 KiB) apply to librbd and

rbd-nbd only — they do nothing for krbd PVCs.

## Benchmarking

- | Layer | Tool | Watch |

- |---|---|---|

- | RADOS | `rados bench -p <pool> 60 write --no-cleanup`, then `seq` / `rand`, then `rados -p <pool> cleanup` | Without `--no-cleanup` the read phases have nothing to read |

- | RBD | `rbd bench --io-type write --io-size 4K --io-pattern rand` | Bypasses the kernel; not what a krbd PVC sees |

- | PVC path | fio inside a pod on the PVC (`--direct=1`) | Matches krbd + filesystem, the real client path |

- | Cluster-wide | CBT (`ceph/cbt`) | Repeatable multi-client runs |

- Run several clients in parallel — one client cannot saturate a cluster —

- and use a dataset larger than the OSD caches.

+ `rados bench` (with `--no-cleanup` before the read phases) and `rbd bench`

+ measure librados/librbd, not what a PVC sees: a krbd PVC is only measured

+ by fio inside a pod on that PVC (`--direct=1`). Use several clients and a

+ dataset larger than the OSD caches; CBT (`ceph/cbt`) for repeatable runs.

## Field reports

| Report | Source |

|---|---|

| 18.2.4 → 20.2.1 on hybrid HDD + NVMe-DB clusters: RGW PUT latency 3–4×, DB-device IOPS near zero; unresolved | ceph-users 2026-07-09 |

## References

- `references/sources.md` — every source with its verification stamp.

- Sibling skills: `ceph-troubleshooting`, `rook-ceph-best-practices`,

`ceph-s3`.