aks-troubleshooting v1.0.1B

Immutable. This exact content is served forever at /api/v1/blob/e3af604634f7cd12.

---
name: aks-troubleshooting
license: MIT
metadata:
  author: Microsoft
  version: "1.0.1"
description: "Debug and root-cause live Azure Kubernetes Service (AKS) incidents with a read-only, evidence-first investigation and structured report. WHEN: pod crashes or Pending, CrashLoopBackOff, OOMKilled, ImagePullBackOff, node NotReady, DNS failure, ingress 502/503, connectivity timeout, network policy, SNAT exhaustion, upgrade stuck, cordon/drain failure, spot or zone disruption, expired certificate, or 'investigate my AKS cluster'. DO NOT USE FOR: packet capture (use aks-network-capture); GPU or model-serving issues (use aks-gpu-inference); cluster creation or provisioning (use aks-cluster-setup); cost or rightsizing (use aks-cost-optimization); an exact documented AKS error code, nested VM-extension/CSE signature, or message-qualified allocation failure (use aks-known-issues). Bare wrappers or codes without the nested cataloged signature, generic failures without a stable signature, and open-ended incidents stay here."
---

# AKS Troubleshooting

Root-cause live AKS incidents with a read-only, evidence-first investigation. This skill covers the full Day-2 troubleshooting surface — workloads, nodes, networking, ingress, upgrades, and spot/zone disruptions — and produces a structured incident report.

## Operating rules

**Read-only by default.** Do not restart, delete, cordon, drain, scale, upgrade, or reconfigure any resource unless the user explicitly asks for remediation. Gather evidence, name the root cause, and propose the fix — but do not apply it uninvited.

**Evidence before conclusion.** Do not state a root cause without quoting the evidence that supports it. "Pod is Pending" and "node is NotReady" are symptoms, not causes — trace them to the specific selector, taint, exhausted resource, or Azure-side condition.

**Tool preference.** Inspect the host's available tools for Azure MCP capabilities that advertise AKS, AppLens, Azure Monitor, or Resource Health operations. Use each matching capability under its host-assigned name, preferring the smallest operation that fits the read. Never treat a specific prefix or spelling as an availability check, and do not invent a name-mapping layer. Fall back to raw `az aks` and `kubectl` only when the host exposes no matching capability or the discovered MCP surface cannot perform the check. Default the MCP access mode to `readonly`. See [references/aks-mcp.md](references/aks-mcp.md).

**Evidence order.** Gather Azure-side state first (cluster state, resource health, recent operations, node-pool state, detector/monitoring output), then Kubernetes-side state (reachability, nodes, `kube-system`, events, the affected namespace, pod detail, logs). This ordering catches platform-level causes — a failed upgrade operation, a stopped cluster, a quota block — before you spend time inside the cluster.

## Route by symptom

| Symptom | Reference |
|---------|-----------|
| Broad investigation, unknown root cause | [general-diagnostics.md](general-diagnostics.md) |
| Pod crash, OOMKilled, ImagePullBackOff, Pending, readiness probe | [pod-failures.md](pod-failures.md) |
| Node NotReady, node pressure, node scaling / autoscaler not triggering | [node-issues.md](node-issues.md) |
| Service connectivity, DNS, pod-to-pod networking | [networking.md](networking.md) |
| Ingress 502/503, load-balancer health probe, external access | [load-balancer-and-ingress.md](load-balancer-and-ingress.md) |
| Network policy blocking traffic | [network-policy.md](network-policy.md) |
| Upgrade stuck, cordon/drain failure | [upgrade-operations.md](upgrade-operations.md) |
| Spot eviction, zone rebalance failure | [spot-and-zone-issues.md](spot-and-zone-issues.md) |
| Any symptom → exact commands, in order | [references/symptom-map.md](references/symptom-map.md) |

`references/symptom-map.md` is the fastest path: 16 symptom sections, each a self-contained block of the exact `kubectl`/`az` commands to run plus the common causes. Start there when the symptom is clear; use the topic files above for deeper investigation.

## Scripts

Both are POSIX `sh`, read-only, and safe to run at the start of any investigation. Invoke with `sh <script>` or `./<script>`.

- `scripts/cluster-snapshot.sh` — quick cluster-health overview (nodes, system pods, recent events, node-pool state).
- `scripts/pod-deep-dive.sh <namespace> <pod>` — full diagnostic dump for one pod: describe, events, current and previous logs across all containers, resource usage.

## AKS-specific gotchas

The highest-signal failure patterns that are specific to AKS — a frontier model will not reliably know these. Review before investigating.

- **Azure CNI vs kubenet is a fork in every networking fix.** Check `az aks show -o json --query networkProfile.networkPlugin` **first** — the plugin (kubenet, Azure CNI, CNI Overlay, Cilium) changes how pod IPs, routes, and network policy behave.
- **Managed-identity RBAC is behind a large share of AKS failures.** ACR pull, disk attach, private DNS, and Key Vault access all depend on the cluster or kubelet identity having a role assignment. Check `az aks show --query identityProfile` and the relevant role assignments early.
- **Node NotReady is not always a VM problem.** It can be kubelet, containerd, the CNI plugin, Azure host maintenance, or an expired kubelet/API-server certificate. Correlate `kubectl describe node` conditions with `az vm get-instance-view`, and check `kubectl get csr` for pending certificate requests.
- **The Azure LB health probe can disagree with Kubernetes.** A Service can look healthy in-cluster but fail at the Azure load balancer because the LB rule's probe path/port does not match the app endpoint. Check `az network lb probe list`.
- **Subnet exhaustion silently blocks scheduling.** Azure CNI allocates a VNet IP per pod; a full pod subnet stops new pods scheduling with no obvious error. Check `az network vnet subnet show --query '{addressPrefix: addressPrefix, used: ipConfigurations | length(@)}'`.
- **System-pool PodDisruptionBudgets block drains during upgrades.** CoreDNS and metrics-server ship PDBs that can stall a node drain. Check `kubectl get pdb -A`.
- **The API server IP can change after stop/start.** When a cluster is stopped and restarted, the API server IP may change; flush DNS and re-run `az aks get-credentials` if `kubectl` cannot connect afterward.
- **Private clusters need in-VNet access.** `kubectl` must run from a VM inside — or peered to — the cluster VNet. Check `az aks show --query apiServerAccessProfile` for private-cluster and authorized-IP-range settings.
- **NSG/firewall egress blocks surface as VM extension errors.** AKS nodes need outbound access to required FQDNs (AKS API, MCR, `management.azure.com`, and others). A restrictive NSG or firewall causes VM extension errors during create/upgrade — error codes 50 (`OutboundConnFailVMExtensionError`), 51 (`K8SAPIServerConnFailVMExtensionError`), 52 (`K8SAPIServerDNSLookupFailVMExtensionError`). Check `az network nsg rule list` and firewall logs.
- **SNAT port exhaustion appears past a few hundred nodes.** Large clusters using the Azure Load Balancer for outbound can exhaust SNAT ports, causing intermittent egress failures. Check `az network lb show --query outboundRules`; fix by moving to a NAT gateway (`az aks update --outbound-type managedNATGateway`).
- **Upgrade `max-surge` defaults to one node at a time.** Large-cluster upgrades take hours at the default. Check `az aks nodepool show --query upgradeSettings` and raise `--max-surge` if the workload tolerates it.
- **`kubectl` must be within two minor versions of the cluster.** A stale client produces confusing errors. Compare `kubectl version --client` with `az aks show --query kubernetesVersion`.

## Log discipline

- Always fetch `kubectl logs --previous` alongside current logs — after a restart the current stream may be empty.
- Do not truncate logs with `--tail` or `| tail`; the causal error is often early.
- For multi-container pods, use `--all-containers` (or name each `--container`) so sidecar and init-container logs are not missed.
- Get current UTC time with `date -u` before using `--since-time`.

## Deep diagnostics

When standard checks do not reveal a root cause, use **Inspektor Gadget** for real-time, low-level node and pod observability (DNS traces, TCP traces, process and file-access snapshots). See [references/inspektor-gadget.md](references/inspektor-gadget.md) for the gadget catalog and symptom-to-gadget mapping. Inspektor Gadget runs a privileged debug pod — get explicit user approval before invoking it. Additional MCP-driven investigation modes are in [references/structured-input-modes.md](references/structured-input-modes.md) and [references/command-flows.md](references/command-flows.md).

## Report

Structure the final incident report using [references/report-template.md](references/report-template.md): symptom and impact, evidence gathered, failure domain, root cause with supporting evidence, confidence, remediation, and escalation. Quote relevant log snippets inline rather than pasting full dumps.

## Reference

Microsoft's AKS troubleshooting hub: https://learn.microsoft.com/troubleshoot/azure/azure-kubernetes/welcome-azure-kubernetes