--- name: k8s-workload-troubleshooting description: >- Use when a pod or workload is unhealthy in Kubernetes: pods stuck in Pending, CrashLoopBackOff, ImagePullBackOff or ErrImagePull, containers OOMKilled (exit code 137), failing liveness, readiness or startup probes, pods Running but not Ready, a Deployment rollout that does not progress (ProgressDeadlineExceeded), or a Job or CronJob that fails. Read-only: investigates with kubectl get, describe, logs and events, and proposes fixes for a human to review and run. license: Apache-2.0 metadata: verified-with: "kubernetes 1.37" verified-on: "2026-10-23" --- # Kubernetes workload troubleshooting Find out why a pod, Deployment, StatefulSet, DaemonSet, Job or CronJob is unhealthy, and propose a fix. **You only read.** Run the read-only commands below; never change the cluster yourself. Every fix goes to the user as a proposal, in the format under [Proposed fix](#proposed-fix), for a human to review and run. The commands use placeholders: `my-namespace`, `my-app` (the workload) and `my-pod` (one of its pods). Replace them with the real names. Never read Secrets. If logs or output contain passwords, tokens or keys, do not repeat them. ## When to use - A pod is `Pending`, `ContainerCreating`, `ErrImagePull`, `ImagePullBackOff`, `CrashLoopBackOff`, `OOMKilled`, `Error`, or `Running` but not ready (`0/1`). - A container keeps restarting, or a probe keeps failing (`Unhealthy` events). - A rollout never finishes: `kubectl rollout status` waits or reports that the Deployment "exceeded its progress deadline". - A Job fails or never completes, or a CronJob's runs fail. Hand-offs, by skill name. When the evidence points elsewhere, continue with that skill; if it is not installed, keep going with the steps here and tell the user which skill would help: - The node is the cause (`NotReady`, memory, disk or PID pressure, evictions, taints no pod tolerates): `k8s-node-troubleshooting`. - A PersistentVolumeClaim is the cause (unbound claim, attach or mount errors): `k8s-storage-troubleshooting`. - The pods are healthy and Ready, but traffic still does not arrive (Service, DNS, NetworkPolicy, Ingress): `k8s-network-troubleshooting`. ## Investigate Work through the steps in order. Stop when you have the cause and its evidence. **1. See the pods and their status.** The `STATUS`, `READY` and `RESTARTS` columns say which reference applies. ```bash kubectl get pods -n my-namespace -o wide kubectl get pods -n my-namespace -l app=my-app ``` `app=my-app` is an example label. Take the real selector from the workload, with `kubectl get deployment my-app -n my-namespace -o jsonpath='{.spec.selector.matchLabels}'` or the `Selector:` line of `kubectl describe deployment my-app -n my-namespace`, and use it wherever a command here says `-l app=my-app`. **2. Describe the failing pod.** Read the container `State` and `Last State` (reason and exit code), `Restart Count`, `Limits` and `Requests`, the probes, the pod `Conditions` and the `Events` at the bottom. ```bash kubectl describe pod my-pod -n my-namespace kubectl events -n my-namespace --for pod/my-pod ``` **3. Read the logs**, of the current container and of the one that ran before the last restart. A crash usually explains itself in the previous container's last lines. ```bash kubectl logs my-pod -n my-namespace --tail=100 kubectl logs my-pod -n my-namespace --previous --tail=100 kubectl logs my-pod -n my-namespace --all-containers --tail=100 ``` **4. Look at the owning controller.** Find what changed and whether a rollout is stuck. ```bash kubectl get deployment my-app -n my-namespace kubectl describe deployment my-app -n my-namespace kubectl rollout status deployment/my-app -n my-namespace --timeout=10s kubectl rollout history deployment/my-app -n my-namespace kubectl get replicasets -n my-namespace -l app=my-app kubectl get deployment my-app -n my-namespace -o jsonpath='{.spec.progressDeadlineSeconds}' ``` `describe deployment` does not print the progress deadline; the last command reads it. For a StatefulSet or DaemonSet use the same `describe`, `rollout status` and `rollout history` commands with `statefulset/my-app` or `daemonset/my-app`. For a Job: ```bash kubectl get jobs -n my-namespace kubectl describe job my-app -n my-namespace kubectl get pods -n my-namespace -l job-name=my-app ``` **5. Check the namespace's recent warnings**, which also show problems of objects that never got a pod (quota, admission, ReplicaSet creation). ```bash kubectl events -n my-namespace --types=Warning ``` **6. Open the matching reference** and follow its steps: | What you see | Reference | |---|---| | `Pending`, `FailedScheduling` | [references/pending.md](references/pending.md) | | `CrashLoopBackOff`, `Error`, restarts | [references/crashloop.md](references/crashloop.md) | | `ErrImagePull`, `ImagePullBackOff`, `InvalidImageName` | [references/image-pull.md](references/image-pull.md) | | `Unhealthy` events, `Running` but `0/1` ready, liveness restarts | [references/probes.md](references/probes.md) | | `OOMKilled`, exit code 137, CPU throttling, quota or LimitRange errors | [references/oom-and-limits.md](references/oom-and-limits.md) | | Rollout stuck, `ProgressDeadlineExceeded`, `ReplicaFailure` | [references/rollouts.md](references/rollouts.md) | | Job failed, `BackoffLimitExceeded`, `DeadlineExceeded`, CronJob not running | [references/jobs.md](references/jobs.md) | ## Read the results What the common findings mean (Kubernetes 1.37): - **Pod phase `Pending`**: the pod is accepted but not all containers are set up and ready to run. This covers both waiting to be scheduled and pulling images. If the `PodScheduled` condition is `False` with a `FailedScheduling` event, the scheduler found no node; see [pending.md](references/pending.md). - **Container state `Waiting`** with reason `ErrImagePull` or `ImagePullBackOff`: the image could not be pulled. The pull event's message says why (tag not found, unauthorized, cannot reach the registry). The kubelet keeps retrying with a growing delay, capped at 5 minutes. - **`CrashLoopBackOff`**: the container starts, exits, and the kubelet waits longer before each restart (10s, 20s, 40s, ... capped at 5 minutes by default; reset after 10 minutes of running). The cause is in `Last State` and in `kubectl logs --previous`, not in the word CrashLoopBackOff. - **`Last State: Terminated`, `Reason: OOMKilled`, `Exit Code: 137`**: the container used more memory than its memory limit and the kernel killed it. Raise the limit or reduce the app's memory use. A node under memory pressure evicting pods is a different cause (an `Evicted` event and pod status); that one belongs to `k8s-node-troubleshooting`. - **`Reason: Error` with another exit code**: the app exited by itself. The logs of the previous container say why (missing config, bad argument, failed dependency). - **`Running` but `READY 0/1`, no restarts, `Unhealthy` "Readiness probe failed" events**: the readiness probe fails, so the pod is not a ready endpoint of its Services and gets no traffic (unless a Service sets `publishNotReadyAddresses: true`). The container is not restarted. - **Restarts with `Unhealthy` "Liveness probe failed" (or startup probe) events**: the kubelet kills the container after `failureThreshold` failures in a row and restarts it. - **Deployment condition `Progressing=False`, reason `ProgressDeadlineExceeded`**: the new ReplicaSet made no progress for the Deployment's `progressDeadlineSeconds` (read it from the spec; 600 is only the default when unset). Kubernetes only reports this; it does not roll back. The new ReplicaSet's pods show the real cause (image pull, crash, readiness, quota). - **Job condition `Failed`, reason `BackoffLimitExceeded`**: pods failed more times than `backoffLimit` (default 6). Reason `DeadlineExceeded`: the Job ran longer than `activeDeadlineSeconds`. State the cause in one sentence, then the evidence you saw (the event reason and message, the exit code, the condition), then the proposed fix. ## Proposed fix Propose, never run. Each fix states what it changes and how to undo it, and is shown in this exact format: the marker line, then a fenced block with the commands and an `# Undo:` line. Prefer fixing the source the workload is deployed from (manifest, Helm values, GitOps repository) when there is one; a direct command is for when there is not, or for an urgent mitigation. Typical fixes: Correct an image tag that does not exist (the container name is the one in the pod spec): > Proposed fix: review before running. The agent does not run this. ```bash kubectl set image deployment/my-app app=registry.example.com/my-app:1.4.2 -n my-namespace # Undo: kubectl set image deployment/my-app app=registry.example.com/my-app:1.4.3 -n my-namespace ``` Roll a stuck Deployment back to the previous revision: > Proposed fix: review before running. The agent does not run this. ```bash kubectl rollout undo deployment/my-app -n my-namespace # Undo: kubectl rollout undo deployment/my-app -n my-namespace (rolls forward to the revision you left) ``` Raise a memory limit that the app outgrows (choose the value from the app's real use): > Proposed fix: review before running. The agent does not run this. ```bash kubectl set resources deployment/my-app -c app --limits=memory=256Mi --requests=memory=256Mi -n my-namespace # Undo: kubectl set resources deployment/my-app -c app --limits=memory=32Mi --requests=memory=32Mi -n my-namespace ``` Point a readiness probe at a path the app serves: > Proposed fix: review before running. The agent does not run this. ```bash kubectl patch deployment my-app -n my-namespace --type=json -p '[{"op": "replace", "path": "/spec/template/spec/containers/0/readinessProbe/httpGet/path", "value": "/"}]' # Undo: kubectl patch deployment my-app -n my-namespace --type=json -p '[{"op": "replace", "path": "/spec/template/spec/containers/0/readinessProbe/httpGet/path", "value": "/healthz"}]' ``` Write the undo with the values you actually found, so it restores the current state. ## Sources Checked against the Kubernetes 1.37 documentation: - [Pod Lifecycle](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/): phases, container states, restart backoff, CrashLoopBackOff, pod conditions, probes. - [Liveness, Readiness, and Startup Probes](https://kubernetes.io/docs/concepts/configuration/liveness-readiness-startup-probes/) and [Configure Liveness, Readiness and Startup Probes](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/): probe fields, defaults, HTTP success range. - [Resource Management for Pods and Containers](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/): OOM kills, CPU throttling, `FailedScheduling`, `OOMKilled` with exit code 137. - [EndpointSlices](https://kubernetes.io/docs/concepts/services-networking/endpoint-slices/): endpoint `ready` condition, `publishNotReadyAddresses`. - [Images](https://kubernetes.io/docs/concepts/containers/images/): `ImagePullBackOff`, pull back-off, image pull policy. - [Deployments](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/): `progressDeadlineSeconds`, `ProgressDeadlineExceeded`, rollout history and undo. - [Jobs](https://kubernetes.io/docs/concepts/workloads/controllers/job/) and [CronJob](https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/): `backoffLimit`, pod failure back-off, `activeDeadlineSeconds`, CronJob policies. - [Debug Pods](https://kubernetes.io/docs/tasks/debug/debug-application/debug-pods/) and [Debug Running Pods](https://kubernetes.io/docs/tasks/debug/debug-application/debug-running-pod/): Pending and Waiting pods, `kubectl logs --previous`.