k8s-triage · git:20261025.b852b9e · 2026-10-25 · sha256 bc94c8a1b74c6368
k8s-triage git:20261025.b852b9eA
Immutable. This exact content never changes and is served at /api/v1/blob/bc94c8a1b74c6368.
--- name: k8s-triage description: >- Use when something is wrong in a cluster and it is not yet clear where: "the cluster is broken", "a lot of pods are failing", "users say the app is down", "something is off after the upgrade", or any problem report that names no workload. Sweeps nodes, pods in every namespace, recent Warning events and CPU and memory use, ranks what it finds by severity, proposes a first fix for each issue and names the skill that goes deeper. Reads first and asks before changing anything. license: Apache-2.0 metadata: verified-with: "kubernetes 1.37" verified-on: "2026-10-23" --- # Kubernetes triage Find out where a cluster is unhealthy when nobody knows yet, rank the problems, and hand each one to the right place. **Read first, ask before changing.** Run the investigation commands freely. Run a command under an "Ask first" line only after the user approves that exact command; if your credentials do not allow it, give it to the user to run. Never run a "Proposed fix"; hand it to the user. Triage is a first look, not a root-cause analysis. For each problem, report it, show the evidence, propose a first fix, and name the skill that investigates it fully. Do not dig deeper than the short first look this skill describes. The commands use placeholders: `my-namespace`, `my-app`, `my-pod` and `my-node`. Replace them with the real names. Never read Secrets. If output contains passwords, tokens or keys, do not repeat them. ## When to use - The user reports a problem without naming a workload: "the cluster is broken", "everything is slow", "lots of pods are failing", "users say the app is down". - Something changed cluster-wide (an upgrade, a new node pool, a network change) and "something is off". - You were asked for a general health check of a cluster. When the user names one workload that is unhealthy, start with `k8s-workload-troubleshooting` instead. Hand-offs, by skill name. For each issue you find, name the skill that goes deeper. If it is not installed, give the issue, its evidence, a first fix and the short first look below, then tell the user which skill would help, and stop there: - Pods crash, restart, stay `Pending`, fail probes, or a rollout or Job is stuck: `k8s-workload-troubleshooting`. - A node is `NotReady`, under memory, disk or PID pressure, cordoned, or pods are being evicted: `k8s-node-troubleshooting`. - Pods are healthy but traffic does not arrive (Service with no endpoints, DNS, NetworkPolicy, Ingress): `k8s-network-troubleshooting`. - A PersistentVolumeClaim is `Pending`, or volumes fail to attach or mount: `k8s-storage-troubleshooting`. - The API server refuses requests (`failed calling webhook`, an APIService that is not `True`, a namespace stuck `Terminating`): `k8s-api-troubleshooting`. ## Investigate Work through the steps in order. They are cheap and cluster-wide; do all of them before you rank anything. **1. Check what you can see.** A user with access to one namespace cannot list nodes or other namespaces. Know this before you read any result. ```bash kubectl auth can-i list nodes kubectl auth can-i list pods --all-namespaces kubectl auth can-i list events --all-namespaces ``` If a check says `no`, or a command below fails with `Forbidden`, say exactly what you could not see, and narrow the sweep to the namespaces you can read. Start with the namespace your kubeconfig points at: commands without `-n` use it, and users with access to one namespace usually have it set. ```bash kubectl auth can-i list pods kubectl get pods kubectl auth can-i list pods -n my-namespace ``` If none of these works and you know no namespace, ask the user which one to look at. Never report something you could not see as healthy. **2. Nodes.** Look at `STATUS` (`Ready`, `NotReady`, and `SchedulingDisabled` for a cordoned node), then describe every node that is not plainly `Ready`, and read its `Conditions`, `Taints` and `Allocated resources`. ```bash kubectl get nodes -o wide kubectl describe node my-node ``` **3. Pods that are not running.** Across every namespace you can read: ```bash kubectl get pods -A --field-selector=status.phase!=Running,status.phase!=Succeeded kubectl get pods -A | grep -v -E "Running|Completed" kubectl get pods -A --sort-by='.status.containerStatuses[0].restartCount' ``` The first command finds `Pending`, `Failed` and `Unknown` pods. The second also catches pods whose phase is `Running` but whose `STATUS` column says `CrashLoopBackOff`, `ImagePullBackOff`, `Error` or `Terminating`. The third puts the most restarted pods last. Also look for `Running` pods that are not ready, such as `0/1` in the `READY` column. When a node is not `Ready`, list the pods on it: ```bash kubectl get pods -A --field-selector=spec.nodeName=my-node -o wide ``` **4. Warning events.** For each one, check whether the object it is about is still unhealthy now. Only those count as issues; a Warning about an object that has since recovered is history, even if it is recent (a new cluster has several from start-up). ```bash kubectl events -A --types=Warning ``` **5. CPU and memory.** `kubectl top` works only when metrics-server is installed and running. If it fails, write "metrics not available" in the overall status and go on; `Allocated resources` from step 2 still shows requests and limits against capacity. ```bash kubectl top nodes kubectl top pods -A --sort-by=memory kubectl top pods -A --sort-by=cpu ``` **6. Controllers short of replicas.** A Deployment, StatefulSet or DaemonSet whose ready count is below the desired count is affected even when its pods look fine one by one. ```bash kubectl get deployments -A kubectl get statefulsets -A kubectl get daemonsets -A ``` **7. PersistentVolumeClaims and Services**, only if steps 3 and 4 point there (pods `Pending` with volume events, or an app that is up but unreachable): ```bash kubectl get pvc -A kubectl get endpointslices -n my-namespace ``` **8. Pick the area of each issue** and its next skill, from this table: | What you see | Tier | Next skill | |---|---|---| | Node `NotReady` or `Unknown`, a pressure condition `True`, `NetworkUnavailable`, `SchedulingDisabled` | 1 | `k8s-node-troubleshooting` | | `CrashLoopBackOff`, `OOMKilled`, `Error`, `ErrImagePull`, `ImagePullBackOff`, `CreateContainerConfigError` | 2 | `k8s-workload-troubleshooting` | | `Pending`, `Running` but not ready, a controller short of replicas | 3 | `k8s-workload-troubleshooting` (or `k8s-storage-troubleshooting` for an unbound PVC) | | Warning events about objects that are still unhealthy, a node near its memory or CPU capacity, pods at their limits | 4 | the area the event belongs to | | Service with no ready endpoints while its pods are healthy, DNS failures | 3 | `k8s-network-troubleshooting` | | `FailedCreate` events that name a webhook, an APIService that is not `True`, a namespace stuck `Terminating` | 3 | `k8s-api-troubleshooting` | A short first look inside a container can confirm a network issue. Ask first: show the command, and run it only after the user agrees (pick a running pod in the namespace that has `nslookup`; many minimal images do not): > Ask first: the agent runs this only after the user approves it. ```bash kubectl exec my-pod -n my-namespace -- nslookup my-app.my-namespace.svc.cluster.local ``` ## Read the results **Rank by severity.** Report issues in this order, most urgent first: 1. **Nodes.** `NotReady` or `Unknown`, `MemoryPressure`, `DiskPressure` or `PIDPressure` `True`, `NetworkUnavailable`, cordoned (`SchedulingDisabled`). 2. **Crashing pods.** `CrashLoopBackOff`, `OOMKilled`, `Error`, `ErrImagePull` or `ImagePullBackOff`, `CreateContainerConfigError`. 3. **Pods that do not become ready.** `Pending`, `Running` but not ready, controllers with fewer ready replicas than desired, Services without ready endpoints. 4. **Early warnings.** Warning events about objects that are still unhealthy, and resource pressure that has not broken anything yet (a node above about 90% of its memory, pods at their CPU or memory limit). Within a tier, more affected nodes or pods rank higher. **One cause, one issue:** when a node problem explains pod symptoms (the pods on a `NotReady` node show `Unknown`, `Terminating`, or are being recreated elsewhere), report the node issue once and list the pods as its evidence. Do not report those pods as separate application failures. **Read nodes.** `Ready` is `Unknown` when the node controller has not heard from the node within its grace period (50 seconds by default): the kubelet is down or the node is unreachable. `Ready` is `False` when the kubelet reports the node is not healthy. A pressure condition `True` means the kubelet may evict pods from that node. The taints `node.kubernetes.io/not-ready`, `node.kubernetes.io/unreachable` and `node.kubernetes.io/unschedulable` (a cordoned node) show the same state to the scheduler. By default, pods on a `NotReady` or unreachable node are evicted after five minutes; until then they look stuck, and pods of a StatefulSet can stay `Terminating`. **Read events.** Typical reasons: `FailedScheduling`, `BackOff`, `Failed` (image pull), `Unhealthy` (probes), `FailedMount`, `Evicted`, `NodeNotReady`, `OOMKilling`. An event is evidence only when its object is unhealthy now: a `Running`, ready pod that once failed a readiness probe is not an issue. **Write the report** in this shape: 1. **Overall status.** Nodes (how many, how many `Ready`), pods (how many are not healthy, in how many namespaces), metrics (available or not), and what you could not see because of access. 2. **Issues**, ranked. For each: one sentence for the issue, the evidence you saw (the status, the condition, the event reason and message, the number), the next skill, and a first fix in the format under [Proposed fix](#proposed-fix). 3. **When the sweep is clean**, do not invent a problem. Say what you checked and that it looked healthy from where you could see, then ask the user to narrow it down: which app, namespace or URL, what exactly they saw (an error message, a timeout, slowness), and since when. ## Proposed fix Triage proposes first fixes only. Show each one to the user first: what it changes and how to undo it. A reversible fix goes under "Ask first" and runs only after the user says yes. Anything destructive or hard to undo is never run by you; say why and hand it over. Prefer fixing the source the workload is deployed from (manifest, Helm values, GitOps repository) when there is one. Write each undo with the values you found. Uncordon a node that was left cordoned after maintenance and is otherwise healthy: > Ask first: the agent runs this only after the user approves it. ```bash kubectl uncordon my-node # Undo: kubectl cordon my-node ``` Roll back a Deployment whose pods started failing right after a rollout. First note the current revision with `kubectl rollout history deployment/my-app -n my-namespace`; the undo below returns to it by number (here revision 3), so it holds even if another rollout happens in between: > Ask first: the agent runs this only after the user approves it. ```bash kubectl rollout undo deployment/my-app -n my-namespace # Undo: kubectl rollout undo deployment/my-app -n my-namespace --to-revision=3 ``` Force-delete a pod stuck `Terminating` on a node that is gone for good. Never run this yourself: for a StatefulSet pod it can start a second copy while the first still runs on a node that is only unreachable. A human must first confirm the node is really gone. > Proposed fix: review before running. The agent does not run this. ```bash kubectl delete pod my-pod -n my-namespace --force --grace-period=0 # Undo: none; the pod's controller recreates it on another node ``` For a node that stays `NotReady`, a workload that keeps crashing, a Service without endpoints or a stuck volume, the fix needs the deeper skill named in the issue. ## Sources Checked against the Kubernetes 1.37 documentation: - [Nodes](https://kubernetes.io/docs/concepts/architecture/nodes/) and [Node Status](https://kubernetes.io/docs/reference/node/node-status/): conditions, the 50-second grace period before `Unknown`, cordoned nodes shown as `SchedulingDisabled`, eviction five minutes after `Unknown`. - [Node-pressure Eviction](https://kubernetes.io/docs/concepts/scheduling-eviction/node-pressure-eviction/): pressure conditions and eviction signals. - [Taints and Tolerations](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/): the `node.kubernetes.io/*` taints, the default 300-second tolerations, pods still running on a partitioned node. - [Pod Lifecycle](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/): pod phases (`Unknown` when the node cannot be reached), `Terminating`, `CrashLoopBackOff`. - [Field Selectors](https://kubernetes.io/docs/concepts/overview/working-with-objects/field-selectors/): `status.phase` and `spec.nodeName` for pods. - [kubectl events](https://kubernetes.io/docs/reference/kubectl/generated/kubectl_events/): `--types`, `-A`. - [Resource metrics pipeline](https://kubernetes.io/docs/tasks/debug/debug-cluster/resource-metrics-pipeline/) and [kubectl top](https://kubernetes.io/docs/reference/kubectl/generated/kubectl_top/): `kubectl top` needs metrics-server. - [Troubleshooting Clusters](https://kubernetes.io/docs/tasks/debug/debug-cluster/): listing and describing nodes. - [Using RBAC Authorization](https://kubernetes.io/docs/reference/access-authn-authz/rbac/): the `view` role and `kubectl auth can-i`.