rook-ceph-best-practices git:20261027.de37ca2A

Immutable. This exact content never changes and is served at /api/v1/blob/b96db9f4b12ce952.

---
name: rook-ceph-best-practices
description: >-
  Operate, configure and upgrade Rook-managed Ceph on Kubernetes (CephCluster
  CR; rook-ceph, rook-ceph-cluster and ceph-csi-drivers charts;
  ceph-csi-operator). Core
  knowledge: the Rook -> Ceph -> key-rotation upgrade order and its version
  gates; the CVE-2025-30156 fix (new AES256K CephX key type, daemon key
  rotation, the six AUTH_INSECURE_* health codes, kernel 7.0 for CSI keys);
  the Rook v1.20 move of CSI to ceph-csi-operator and the new ceph-csi-drivers
  chart; disabling the rook mgr module before Tentacle; Helm monitoring RBAC.
when_to_use: >-
  Use for any Rook/Ceph operator task: "upgrade rook", "upgrade ceph under
  rook", "bump cephVersion.image", "rook helm chart upgrade", "rook v1.20",
  "ceph-csi-drivers chart", "ceph-csi-operator", "CVE-2025-30156", "aes256k",
  "cephx key rotation", "keyRotationPolicy", "keyGeneration",
  "AUTH_INSECURE_SERVICE_KEY_TYPE", "muteHealthWarning", "allowedCiphers",
  "rook toolbox permission denied", "rook mgr module", "rook-ceph monitoring
  ServiceMonitor forbidden", "which Ceph versions does Rook support".
  Symptoms: HEALTH_ERR after a Ceph point-release bump, CSI ctrlplugin 0/2
  after a Rook upgrade, toolbox `handle_auth_bad_method` / errno 13, mgr
  crash-looping on Tentacle, PVC mounts failing after key rotation.
  NOT for other Ceph questions just because the cluster runs under Rook:
  PG states, slow ops, OSD OOM, stuck PVCs after node loss go to
  ceph-troubleshooting; OSD memory sizing, Multus/network, EC pools go to
  ceph-performance; RGW, S3 errors, CephObjectStoreAccount go to ceph-s3.
argument-hint: "[upgrade|cephx|csi|monitoring] (optional focus area)"
---

# rook-ceph-best-practices

Facts cover Rook through v1.21.1 and Ceph through 19.2.7 / 20.2.4.
Everything here is version-gated: re-check `references/sources.md` before
relying on it for a newer release.

## Version gates

| Rook | Kubernetes | Ceph supported | Notes |
|---|---|---|---|
| v1.18.x | 1.29–1.34 | Reef 18.2, Squid 19.2 | Last line that runs Reef. ceph-csi-operator becomes the default CSI config path. Key rotation experimental. |
| v1.19.x | 1.30–1.35 | Squid 19.2.0+, Tentacle | **Reef dropped** (min Ceph 19.2.0). AES256K from **v1.19.9**; CVE floor **v1.19.10**. v1.19.11 ships ceph-csi 3.16.3 (AES256K backport). |
| v1.20.x | 1.31–1.37 (1.36 before v1.20.8) | Squid 19.2.0+, Tentacle 20.2.1+ | **Rook no longer deploys CSI** — ceph-csi-operator + new `ceph-csi-drivers` chart required. AES256K from v1.20.5; CVE floor **v1.20.6**. v1.20.7 ships ceph-csi 3.17.1. |
| v1.21.x | 1.32–1.37 | Squid 19.2.0+, Tentacle 20.2.1+, Umbrella 21.2.0+ | **msgr2 required by default**: kernel clients need 5.11+, else set `network.connections.requireMsgr2: false` before the hop. Rook pins no CSI images. |

Ceph lines: Reef 18.2.x is EOL (last 18.2.8, 2026-03-20). Squid 19.2.x
latest 19.2.7, estimated EOL **2026-10-31** — plan the Tentacle hop.
Tentacle 20.2.x latest 20.2.4. Umbrella has no stable release (no 21.2.0).
Do not run **20.2.0** (data corruption with CSI read affinity,
rook#16839) or **18.2.5/18.2.6**
(BlueStore corruption regression, fixed 18.2.7). `x.2.z` is a stable
release; a `v19.3.0`-style tag is a development tag, never deploy it.

## Upgrade order — never reorder

1. **Rook operator** (Helm `rook-ceph` chart; CRDs ship as chart templates
   when `crds.enabled`, so `helm upgrade` applies them).
2. **Ceph image** (`CephCluster.spec.cephVersion.image`).
3. **CephX key rotation** (can ride on step 2's restart — same patch).
4. **Toolbox.** The `rook-ceph-cluster` chart toolbox follows `cephImage`
   unless `toolbox.image` / `repository` / `tag` is set. Rook never touches a
   manifest toolbox: re-apply `deploy/examples/toolbox.yaml` from the new tag.

A Rook minor hop must start from a supported patch: v1.19 → v1.20 requires
**v1.19.5+** first; v1.20 → v1.21 accepts any v1.20 patch (checklist in
`references/upgrade-path.md`). Clusters below v1.18 heading for the CVE fix take the
path in `references/upgrade-path.md` — do not jump straight to Ceph 19.2.6.

### Gates between hops

| When | Check | Pass |
|---|---|---|
| Before any hop | `ceph status`, CephCluster `.status.phase` | HEALTH_OK, phase Ready, all PGs active+clean, all mons in quorum |
| Operator hop done | `kubectl -n <ns> get deploy -l rook_cluster=<ns> -o jsonpath='{range .items[*]}{.metadata.labels.rook-version}{"\n"}{end}' \| sort \| uniq -c` | one line |
| Ceph hop done | `ceph versions` | `overall` has one line whose count = total daemons |
| Rotation done | CephCluster `.status.cephx.<type>.keyGeneration` for admin, mon, mgr, osd, crashCollector, cephExporter; also CephFilesystem / CephObjectStore `.status.cephx.daemon` | every entry at the target generation |

`ceph versions` does not change during an operator-only hop — use the
`rook-version` label. Watch progress with the `ceph-version` label, not the
toolbox (the toolbox loses auth mid-rotation).

**HEALTH_ERR semantics.** The operator checks health *when an update is
requested* and refuses to start from HEALTH_ERR. HEALTH_ERR raised *during*
a started rolling upgrade by the new AUTH_INSECURE_* codes did not stop it
(observed on v1.19.11). Do not pre-emptively set `skipUpgradeChecks: true`;
in rook#18240 forcing past HEALTH_ERR mid-rotation preceded a full OSD
outage.

### Pin images by digest

Use full build tags plus digest: `quay.io/ceph/ceph:v19.2.6-20260818@sha256:…`.
A floating tag (`:v20`) with `imagePullPolicy: IfNotPresent` keeps running
a stale cached image — root cause of the toolbox `Malformed input` keyring
error in rook#18240. The CVE quick-fix patch in Rook docs up to v1.19.11 /
v1.20.7 shows `quay.io/ceph/ceph:v1.20.4-20260818` — a typo for
`v20.2.4-20260818` (fixed in v1.19.12 / v1.20.8); do not copy it.

## CVE-2025-30156 and AES256K — the short path

Unaided models get this wrong in three ways, so do not repeat them:
`AUTH_INSECURE_*_KEY_TYPE` / `AUTH_INSECURE_SERVICE_TICKETS` are new in
19.2.6 / 20.2.4 and are **not** the 2021 `AUTH_INSECURE_GLOBAL_ID_RECLAIM*`
codes (CVE-2021-20288) — `auth_allow_insecure_global_id_reclaim` does
nothing here; `aes256k` **is** a real CephX key type; and
`spec.security.cephx.allowedCiphers` / `keyType` **are** real CephCluster
fields (Rook ≥ v1.19.9 / v1.20.5).

The image bump alone does **not** fix the CVE. The fix is Ceph ≥ 19.2.6 /
20.2.4 **plus** rotating the core daemon keys (mon, mgr, osd, mds) to
AES256K. Client kernels do not matter for this step: once daemon keys are
AES256K every client, including `aes` ones, receives AES256K service tickets.

Before the Ceph bump:

1. Check node kernels, never a note:
   `kubectl get nodes -o custom-columns=NODE:.metadata.name,KERNEL:.status.nodeInfo.kernelVersion`.
2. Pin CSI key type if any node is < 7.0 (FIPS: < 7.2) without a
   vendor-documented libceph AES256K backport (CentOS Stream 9/10 have one):
   `spec.security.cephx.csi.keyType: aes`. With `keyRotationPolicy` unset
   this triggers no rotation; Rook only creates CSI keys that are missing.
3. Remove auth entities for OSDs that no longer exist — Rook only rotates
   live daemons, so a stale `osd.N` stays `aes` and is the one HEALTH_ERR
   left afterwards. Compare `ceph auth ls | grep ^osd.` with `ceph osd ls`;
   for each orphan: `ceph osd find N` (must be ENOENT) → `ceph auth get osd.N`
   (backup) → `ceph auth rm osd.N`.
4. Save a baseline: CephCluster YAML (`status.cephx`), `ceph auth ls`,
   `ceph osd tree`, pods, full operator log. Take `ceph auth ls` now — after
   the admin key rotates the toolbox cannot authenticate until it reloads
   the keyring (~60 s; manifest toolbox from v1.19.9 / v1.20.5, chart
   toolbox from v1.20.7) or, if older, is restarted.

Then apply image + rotation in one patch:

```yaml
spec:
  cephVersion:
    image: quay.io/ceph/ceph:v19.2.6-20260818   # or v20.2.4-20260818; pin the digest
  security:
    cephx:
      daemon:
        keyRotationPolicy: KeyGeneration
        keyGeneration: 2          # any integer above current status
```

Expect, in order: HEALTH_ERR `AUTH_INSECURE_SERVICE_KEY_TYPE` +
`AUTH_INSECURE_SERVICE_TICKETS` as soon as the first daemons run the new
version; rotation admin → mon → mgr → OSDs one at a time → RGW/MDS/others;
OSD pods briefly 1/2 or Init (normal); the toolbox failing with
`handle_auth_bad_method … [errno 13] RADOS permission denied` (an older
toolbox needs a restart; a current one recovers by itself).
The two errors clear when the core daemons are rotated. Remaining warnings
are expected — see the health-code table in `references/cephx-aes256k.md`
and mute them with `spec.healthCheck.muteHealthWarning.<CODE>.policy: mute`
(Rook ≥ v1.19.10).

**If rotation stalls** (a daemon type stuck at the old generation, OSDs
losing auth): per the maintainer in rook#18240 — apply no workarounds,
restart no pods in the Rook namespace, collect the operator log immediately.
`allowedCiphers` edits did not help that case.

**Never** set `allowedCiphers: [aes256k]` until
`ceph auth dump-keys --format=json-pretty` shows no `aes` key anywhere, and
never while any node kernel is < 7.0 with CSI keys that could rotate.
Recovery and the CSI key migration procedure: `references/cephx-aes256k.md`.

`status.cephx.osd` shows `keyGeneration` but not `keyType`; `ceph health
detail` / `ceph auth dump-keys` are the authority on key type.

## Rook v1.20: CSI moves to ceph-csi-operator

Rook v1.20 removed every CSI setting from `rook-ceph-operator-config` and
from the `rook-ceph` chart. Helm install/upgrade order becomes:
**`rook-ceph` → `ceph-csi-drivers` (new, required) → `rook-ceph-cluster`**.
Skipping `ceph-csi-drivers` leaves the CSI ServiceAccounts/RBAC missing:
ctrlplugin Deployments sit at 0/2 and every attach/detach in the cluster
fails (rook#17644). Migrate custom CSI settings to ceph-csi-operator
`OperatorConfig` / `Driver` CRs by hand before the hop.

Install `ceph-csi-drivers` with Rook's values file
(`deploy/charts/ceph-csi-drivers/values.yaml` at the Rook tag). The chart's
own defaults lack the Rook-prefixed driver names and
`operatorConfig.driverSpecDefaults.imageSet.name:
rook-csi-operator-image-set-configmap`; without the image set
ceph-csi-operator runs its built-in cephcsi — 3.17.0 in the operator
versions Rook v1.20.x bundles, below the AES256K floor
(ceph-csi-operator#605). From v1.21 Rook leaves that configmap empty and
the bundled operator's default (cephcsi 3.18.1) applies. Verify the running
cephcsi image after the hop; AES256K needs cephcsi ≥ 3.17.1.

## Tentacle: disable the rook mgr module first

Rook docs (v1.20.6+) recommend, before upgrading to Tentacle:

```yaml
spec:
  mgr:
    modules:
    - name: rook
      enabled: false
```

On Tentacle the enabled module makes the mgr crash-loop in the prometheus
module (`NotImplementedError` from `node_proxy_fullreport`, rook#18124,
ceph tracker 79106), and a day of crash records then breaks `ceph crash ls`
(tracker 79357). Its verbose logging is also one of two causes of mgr
OOM kills after Rook v1.20.1 / Ceph 20.2.2 (rook#17786; the other, a
`remote()` refcount leak, is tracker 77724). The only loss is some dashboard orchestrator pages; Rook
does not use the module. The chart default was flipped — check the live
CephCluster, not the chart.

## Monitoring with a Helm-installed operator

Needs the operator chart value `monitoring.enabled: true` (ServiceMonitor
RBAC) as well as `CephCluster.spec.monitoring.enabled`; without it the
operator loops on `servicemonitors.monitoring.coreos.com … is forbidden`
and never enables the mgr prometheus module. After the RBAC lands, restart
the operator — it did not create the ServiceMonitors until restarted.
`metricsDisabled: false` turns on both the mgr prometheus module and the
per-node `ceph-exporter`. The operator never creates PrometheusRules; ship
them separately.

## Other hazards worth checking before a hop

Per-release breaking changes (CRUSH rule deletion, MDS standby removal,
chart field splits, S3 credential purge) are in `references/upgrade-path.md`.

| Hazard | Source |
|---|---|
| OSDs with `encryptedDevice: true`: daemon key rotation can leave them in `Init:CrashLoopBackOff` with `Unable to retrieve dmcrypt secret` — ceph-volume's tmpfs remount discards the refreshed lockbox keyring; pod/operator restarts do not fix it (reporter fixed the lockbox LVM tag by hand). Open; rotate a single encrypted OSD's node first | rook#18282 |
| 19.2.6 → Tentacle below 20.2.4 drops AES256K (20.2.0–20.2.3 predate it): go 19.2.6 → 20.2.4+ only | ceph-users 2026-09-03 |
| After 19.2.6 → 20.2.4, re-check `ceph config get mon auth_allowed_ciphers`: a cephadm upgrade reset `aes256k` to `aes, aes256k` | tracker 80295 |
| RGW SigV4 fix (19.2.6 / 20.2.4) rejects presigned PUTs with an unsigned Content-Type (restic, PHP clients). Squid: go to 19.2.7 (fixed). Tentacle: `rgw_sigv4_insecure=true` until a release after 20.2.4 ships the merged backport | tracker 79674 |
| Squid→Tentacle below Rook v1.19.12 / v1.20.8: Rook rolls RGW before all OSDs are upgraded, breaking S3 multipart during the window — upgrade Rook first. MDS still rolls once the mons are done (v1.21.1) | rook#18367 |
| RGW multisite on 19.2.6: SigV4 rejects RGW's own forwarded requests (`403 AccessDenied`, e.g. bucket creation on a non-master zone); `rgw_sigv4_insecure` alone does not help. Upgrade Squid to 19.2.7 (fixed). A zone that must run 19.2.6 needs `rgw_sigv4_insecure: "true"` **and** `rgw_s3_client_max_sig_ver: "2"` in each store's `gateway.rgwConfig` **before** upgrading; revert to `"false"` / `"-1"` once every zone is fixed. Tentacle has the fix since 20.2.2, though Rook docs also list 20.2.4 | tracker 79698 → 74579; Rook ceph-object-multisite.md |
| Mons of a fresh cluster never hold quorum when the container runtime passes an unbounded open-files limit into pods (`Max open files 2147483584`): every `ceph` CLI spawn hangs, the admin-socket probe times out, kubelet kills healthy mons. Cap the runtime's default nofile (e.g. 1048576). Not a kernel 7.0 regression | rook#18370 |
| 1500-OSD-scale 18.2.8 → 19.2.6 upgrade: mon slow ops climbing into tens of thousands on each mon restart; did not reproduce in staging | ceph-users 2026-09-14 |
| NFS-Ganesha per-export keys cannot be rotated (Ceph has no mechanism) | Rook cephx-key-rotation.md |

## References

- `references/cephx-aes256k.md` — key types, all six health codes, CSI key
  migration and revert, `allowedCiphers` recovery, CephClient / RBD-mirror /
  external-cluster keys. Read before any rotation beyond the daemon keys.
- `references/upgrade-path.md` — release-by-release breaking changes
  v1.16–v1.21 and the multi-hop CVE path from older Rook.
- `references/sources.md` — every source with its verification stamp.
- Sibling skills: `ceph-troubleshooting`, `ceph-performance`, `ceph-s3`.