k8s-upgrade-readiness · git:20261031.8d9def9 · 2026-10-31 · sha256 b044e1fb389d2bd6
k8s-upgrade-readiness git:20261031.8d9def9A
Immutable. This exact content never changes and is served at /api/v1/blob/b044e1fb389d2bd6.
---
name: k8s-upgrade-readiness
description: >-
Use when a Kubernetes cluster is about to be upgraded to its next minor version, or
someone asks "is the cluster ready to upgrade?", "what will break when we upgrade?" or
why a node drain hangs during an upgrade: PodDisruptionBudgets with ALLOWED
DISRUPTIONS 0, pods with no controller ("cannot delete Pods that declare no
controller"), admission webhooks with failurePolicy Fail and no endpoints, deprecated
or removed APIs still in use (apiserver_requested_deprecated_apis, CRD versions marked
deprecated), kubelet and control plane version skew, nodes on cgroup v1 or containerd
1.x, and add-ons that do not support the target version. Reports what to fix first.
Read-only: never upgrades, drains or changes anything itself.
license: Apache-2.0
metadata:
verified-with: "kubernetes 1.37"
verified-on: "2026-10-31"
---
# Upgrade readiness
Check a cluster before a Kubernetes minor version upgrade, and report what has to be
fixed first. **Read only.** Run the investigation commands freely. Never upgrade,
drain, cordon or evict anything, and never change a workload, a PodDisruptionBudget or a
webhook yourself: everything here is a proposed fix for the user and the owners of each
object. Never read Secrets.
An upgrade replaces the control plane first, one minor version at a time, and then
every node: each node is cordoned and drained (its pods are evicted and recreated
elsewhere), then replaced or upgraded. Managed clusters (AKS, EKS, GKE) do both for you,
but they drain nodes the same way. So most of what breaks an upgrade is in the cluster:
pods that cannot be evicted, pods that cannot be recreated, and clients or add-ons that
use an API or a node setup the new version no longer supports.
The commands use placeholders: `my-namespace`, `my-app`, `my-node`. Replace them with
the real names. Some checks need more than the built-in `view` role (nodes, CRDs,
webhook configurations, the API server's metrics); if a command is forbidden, say which
check it was and give the user the command to run.
## When to use
- Before a minor version upgrade, or when planning one.
- A node drain during an upgrade hangs, or a managed upgrade fails or stalls on
evictions.
- After an upgrade, clients fail with `no matches for kind` or `the server could not
find the requested resource`.
Hand-offs, by skill name. When the evidence points elsewhere, continue with that skill;
if it is not installed, keep going with the steps here and tell the user which skill
would help:
- On AKS: available versions, node pool upgrades and upgrades that failed:
`k8s-aks-troubleshooting`.
- A webhook or an APIService is already failing requests today: `k8s-api-troubleshooting`.
- A node is NotReady or under pressure today: `k8s-node-troubleshooting`.
- Pods are already failing today: `k8s-workload-troubleshooting`, or `k8s-triage` for a
first look.
- The Gateway API CRDs and their versions: `k8s-gateway-api-troubleshooting`.
## Investigate
Run every step: the goal is a complete list, not the first problem. Note the current
version and the target version (the next minor) first.
**1. Versions and skew**: the API server, every kubelet, the container runtime, the
kernel and the OS image.
```bash
kubectl version
kubectl get nodes -o wide
kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{" kubelet="}{.status.nodeInfo.kubeletVersion}{" runtime="}{.status.nodeInfo.containerRuntimeVersion}{" kernel="}{.status.nodeInfo.kernelVersion}{" os="}{.status.nodeInfo.osImage}{"\n"}{end}'
kubectl get pods -n kube-system -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.spec.containers[*].image}{"\n"}{end}'
```
**2. Nodes that are not ready, cordoned or under pressure.**
```bash
kubectl get nodes
kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{" unschedulable="}{.spec.unschedulable}{" "}{range .status.conditions[?(@.status=="True")]}{.type}{" "}{end}{"\n"}{end}'
```
**3. What a drain cannot evict**: PodDisruptionBudgets that allow no disruption, and
pods with no controller.
```bash
kubectl get pdb -A
kubectl get pdb -A -o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name}{" allowed="}{.status.disruptionsAllowed}{" healthy="}{.status.currentHealthy}/{.status.desiredHealthy}{" selector="}{.spec.selector}{" unhealthyPodEvictionPolicy="}{.spec.unhealthyPodEvictionPolicy}{"\n"}{end}'
kubectl get pods -A -o custom-columns=NAMESPACE:.metadata.namespace,NAME:.metadata.name,OWNER:.metadata.ownerReferences[0].kind,NODE:.spec.nodeName
kubectl get deployments,statefulsets -A
```
**4. What would refuse the recreated pods**: admission webhooks that fail closed, and
whether their Services have endpoints.
```bash
kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations
kubectl get validatingwebhookconfigurations -o jsonpath='{range .items[*]}{range .webhooks[*]}{.name}{" failurePolicy="}{.failurePolicy}{" service="}{.clientConfig.service.namespace}/{.clientConfig.service.name}{" rules="}{.rules}{" namespaceSelector="}{.namespaceSelector}{"\n"}{end}{end}'
kubectl get mutatingwebhookconfigurations -o jsonpath='{range .items[*]}{range .webhooks[*]}{.name}{" failurePolicy="}{.failurePolicy}{" service="}{.clientConfig.service.namespace}/{.clientConfig.service.name}{" rules="}{.rules}{" namespaceSelector="}{.namespaceSelector}{"\n"}{end}{end}'
kubectl get endpointslices -n my-namespace
```
**5. Deprecated APIs still in use**: requests the API server has served for deprecated
versions since it started, and custom resource versions marked deprecated or still
stored.
```bash
kubectl get --raw /metrics | grep apiserver_requested_deprecated_apis
kubectl get crds -o jsonpath='{range .items[*]}{.metadata.name}{" stored="}{.status.storedVersions}{" deprecated="}{.spec.versions[?(@.deprecated==true)].name}{"\n"}{end}'
kubectl api-versions
```
**6. Add-ons and their versions**: what runs as part of the cluster, to check against
each project's supported Kubernetes versions.
```bash
kubectl get deployments,daemonsets,statefulsets -A -o jsonpath='{range .items[*]}{.kind}{" "}{.metadata.namespace}/{.metadata.name}{" "}{.spec.template.spec.containers[*].image}{"\n"}{end}'
kubectl get apiservices
```
**7. Open the matching reference** for the details:
| What you check | Reference |
|---|---|
| Version skew, the upgrade order, cgroup v1, containerd 1.x, deprecated and removed APIs, CRD stored versions, add-on compatibility | [references/versions-and-apis.md](references/versions-and-apis.md) |
| PodDisruptionBudgets, unmanaged pods, local storage, single replicas, webhooks during a drain | [references/drains.md](references/drains.md) |
## Read the results
What the common findings mean (Kubernetes 1.37):
- **A PDB with `ALLOWED DISRUPTIONS` `0`** while its pods are healthy (for example
`minAvailable` equal to the replicas, or `maxUnavailable: 0`): no pod it selects can
be evicted, so a drain of a node running one of them waits forever, and managed
upgrades stall or fail on it. A PDB with 0 because pods are unhealthy blocks the same
way unless its `unhealthyPodEvictionPolicy` is `AlwaysAllow`.
- **A pod with no owner** (`OWNER` `<none>`): kubectl drain refuses it with `cannot
delete Pods that declare no controller (use --force to override)`; with `--force` it
is deleted and nothing recreates it. Pods owned by a `Node` are static (mirror) pods,
and DaemonSet pods are skipped by drains; both are fine.
- **A pod with an `emptyDir` volume**: drain refuses it with `cannot delete Pods with
local storage (use --delete-emptydir-data to override)`; the data is lost on eviction.
Usually expected; say so only if the data matters.
- **A webhook with `failurePolicy: Fail` whose Service has no endpoints**: requests it
matches are refused. If it matches pods, every pod a drain evicts in those namespaces
cannot be recreated, although the running ones look fine today.
- **A kubelet newer than the API server, or more than three minor versions older**:
outside the version skew policy. Its upgrade order is stricter: before the API server
moves to the target, every kubelet should be at most two minor versions older than
the target. Upgrade those nodes first. The API server itself moves one minor version
at a time.
- **A `containerRuntimeVersion` of `containerd://1.x`** with a target of 1.36 or later:
the kubelet no longer supports containerd 1.x; upgrade containerd to 2.0 or later
first. A **kernel older than 5.8**, or an OS image that still uses cgroup v1: since
1.35 the kubelet does not start on cgroup v1 by default.
- **`apiserver_requested_deprecated_apis` lines**: a client used that group, version
and resource since the API server started. A `removed_release` at or below the target
version means that client breaks after the upgrade. `version="v1"` `resource="endpoints"`
is the core Endpoints API, deprecated in favour of EndpointSlices with no removal
planned.
- **A CRD whose `storedVersions` contains a version marked `deprecated`**, or a version
the next release of its add-on drops: objects are still stored in that version. They
must be migrated, and `storedVersions` trimmed, before the add-on removes it.
- **Add-on versions**: compare each against the project's compatibility table for the
target version; an add-on that does not support it is upgraded first.
Report the result as a list, most important first: blockers (the upgrade will fail or
cause an outage), then risks (downtime, data loss, a client that will break), then what
was checked and found fine. For each, the object, the evidence and the proposed fix.
## Proposed fix
Show each fix to the user; the agent runs none of them. Every finding here belongs to
the owner of the object (an application team, a platform team), and changing a PDB, a
webhook or a workload's replicas during an upgrade window affects availability. Write
each undo with the values you found.
Let a single-replica app keep a pod running while another starts (two or more replicas;
the PDB then allows one disruption):
> Proposed fix: review before running. The agent does not run this.
```bash
kubectl scale deployment my-app -n my-namespace --replicas=2
# Undo: kubectl scale deployment my-app -n my-namespace --replicas=1
```
Allow one disruption instead of keeping every pod (for the app's owners; prefer the
change in the app's source):
> Proposed fix: review before running. The agent does not run this.
```bash
kubectl patch pdb my-app -n my-namespace --type=json -p '[{"op": "remove", "path": "/spec/minAvailable"}, {"op": "add", "path": "/spec/maxUnavailable", "value": 1}]'
# Undo: kubectl patch pdb my-app -n my-namespace --type=json -p '[{"op": "remove", "path": "/spec/maxUnavailable"}, {"op": "add", "path": "/spec/minAvailable", "value": 1}]'
```
A pod with no controller is recreated under a Deployment (or a Job) by its owner, or
deleted if nobody needs it, before the upgrade. A webhook without endpoints gets its
backend back before the upgrade (see `k8s-api-troubleshooting`). Clients and manifests
that use a removed API are moved to its replacement version first. The references give
the commands for each.
## Sources
Checked against the Kubernetes 1.37 documentation, and on a Kubernetes 1.37 kind
cluster:
- [Version Skew Policy](https://kubernetes.io/releases/version-skew-policy/): the
skew for each component and the upgrade order.
- [Deprecated API Migration Guide](https://kubernetes.io/docs/reference/using-api/deprecation-guide/)
and [Kubernetes Deprecation Policy](https://kubernetes.io/docs/reference/using-api/deprecation-policy/).
- [Warning: Helpful Warnings Ahead](https://kubernetes.io/blog/2020/09/03/warnings/):
deprecation warnings and the `apiserver_requested_deprecated_apis` metric.
- [Safely Drain a Node](https://kubernetes.io/docs/tasks/administer-cluster/safely-drain-node/),
[Disruptions](https://kubernetes.io/docs/concepts/workloads/pods/disruptions/) and
[Specifying a Disruption Budget](https://kubernetes.io/docs/tasks/run-application/configure-pdb/):
evictions, PDBs, `unhealthyPodEvictionPolicy`.
- [About cgroup v2](https://kubernetes.io/docs/concepts/architecture/cgroups/): cgroup
v1 deprecated in 1.35, the kubelet's `failCgroupV1`, the kernel 5.8 requirement.
- [Kubernetes v1.34: Autoconfiguration for Node Cgroup Driver Goes GA](https://kubernetes.io/blog/2025/09/12/kubernetes-v1-34-cri-cgroup-driver-lookup-now-GA/):
containerd 1.x supported until 1.35, the `kubelet_cri_losing_support` metric.
- [Dynamic Admission Control](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/):
`failurePolicy` and `matchPolicy`.
- [Versions in CustomResourceDefinitions](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definition-versioning/):
deprecated versions and `storedVersions`.
- The drain messages and the metric lines quoted were seen on the kind cluster with
a server-side dry-run drain, run by the test kit's admin, not the agent.