git:20260805.5b3309e to git:20260925.7bf99b8

- The symptom-first companion to the `workload` skill (which owns workload types, the spec shape, and the create/update tools). Given an unhealthy workload, map what you observe to its platform-specific root cause and a fix the schema will actually accept. Diagnosis is **read-only and MCP-first**; most failures trace to a Control Plane rule a generic engineer would not guess — deny-by-default firewalls, the secret identity+policy chain, blocked ports, the sleep-binary shutdown rule. The single most common is OOMKilled. Deep remediation for each area lives in the domain skill named in that section; this skill is the diagnostic map.

-

- ## Step 1 — Gather state (read-only)

- | Tool | What it tells you |

- |---|---|

- | `mcp__cpln__list_deployments` | **Start here.** Per-location readiness with reason/message. Pass `location` to drill into one failing location. |

- | `mcp__cpln__get_workload_events` | Image pulls, crashes, scheduling, probe failures, `OOMKilled`. |

- | `mcp__cpln__get_workload_logs` | App logs (LogQL); the `_accesslog` container holds HTTP status codes and latency. |

- | `mcp__cpln__get_resource` (kind=`workload`) | The spec and current status. |

- | `mcp__cpln__list_metrics` then `mcp__cpln__query_metrics` | Resource pressure — memory before OOM, CPU, latency. |

- | `mcp__cpln__list_workload_replicas` | Confirm which replicas are currently running. Use the `cpln` CLI after reading the `cpln` skill if in-container inspection is essential. |

- | `mcp__cpln__query_traces` then `mcp__cpln__get_trace` | For a slow or intermittently failing request: which span in the path spent the time or errored. Needs tracing enabled on the GVC (opt-in). Deep dive in `metrics-observability`. |

- CLI fallback (MCP unavailable, interactive shell, or CI/CD):

- ```bash

- cpln workload get WORKLOAD --gvc GVC -o json

- cpln workload eventlog WORKLOAD --gvc GVC -o json

- cpln logs '{gvc="GVC", workload="WORKLOAD"}' --limit 50 # |= "error" filters; container="_accesslog" for HTTP codes

- cpln workload connect WORKLOAD --gvc GVC --location LOCATION # interactive shell

- ```

- ### Out of memory (OOMKilled) — the most common issue

-

- **Symptoms:** container restarts repeatedly, events show `OOMKilled`, crashes under load.

-

- `memory` is a hard cap — exceed it (app + runtime + GC + buffers) and the kernel kills the container. Usual culprits: Java without `-Xmx`, Node without `--max-old-space-size`, Python loading large datasets. Each container, sidecars included, has its own limit. With **Capacity AI** on, spiky workloads can be downsized too aggressively — set `minMemory` as a floor (Capacity AI never downscales CPU below 25 millicores).

-

- **Fix:** check real usage with `query_metrics`, then raise `memory` — but memory (MiB) must stay within **8× CPU (millicores)**, so raise `cpu` alongside it or the update is rejected (the default `cpu: 50m` caps memory at 400Mi). See *Apply fixes within the schema's limits*.

- ### Image pull failures

- **Symptoms:** events show `ImagePullBackOff` / `ErrImagePull`, deployment stuck.

- - **Reference format** — `//image/NAME:TAG` (org registry), bare `NAME:TAG` (Docker Hub, no `docker.io/`), full URL (other registries).

- - **Platform** — images must be `linux/amd64` for managed locations (BYOK allows more).

- - **Pull secret** — a private external registry needs a pull secret in the GVC's `pullSecretLinks`; only `docker`, `ecr`, `gcp` secret types work, and org `//image/` images need none.

- The registry secret must already exist (created by the user — offer a manifest scaffold for them to fill and apply, `setup-secret` skill); attach with `update_gvc`. Deep setup: `image` skill.

- **Symptoms:** env vars empty, logs show missing config, secret-access errors in events.

-

- A workload reaches a secret only with all three in place: an **identity linked** to it (`spec.identityLink`), a **policy granting that identity `reveal`** on the secret, and a **correct reference**. Fastest fix: `grant_workload_secret_access` builds the whole chain in one call. Reference format by type:

-

- | Type | Reference |

- |---|---|

- | Opaque (decoded / raw) | `cpln://secret/NAME.payload` / `cpln://secret/NAME` |

- | Dictionary | `cpln://secret/NAME.KEY` |

- | Username & password | `cpln://secret/NAME.username` / `.password` |

- | Keypair | `cpln://secret/NAME.secretKey` / `.publicKey` / `.passphrase` |

- | TLS | `cpln://secret/NAME.cert` / `.key` / `.chain` |

- | AWS | `cpln://secret/NAME.accessKey` / `.secretKey` / `.roleArn` / `.externalId` |

-

- The manual chain: `access-control` and `setup-secret` skills.

- ### Port mismatch — healthy but 502/503

- - The spec port must match what the process listens on — compare the workload spec with application startup logs. If that is inconclusive, use the `cpln` CLI after reading the `cpln` skill for an in-container socket check.

- - On serverless, the runtime injects `PORT` and rejects a `PORT` env var that doesn't equal the exposed port.

- - **Type rules:** serverless exposes exactly one port (zero is rejected; TCP needs a dedicated LB — see *Dedicated load balancer & domain*); standard and stateful expose zero or more; cron serves no endpoint.

- - **Blocked ports** (cannot bind, invalid for TCP probes): `8012, 8022, 9090, 9091, 15000, 15001, 15006, 15020, 15021, 15090, 41000`.

- **Symptoms:** unreachable externally, can't reach external APIs, or can't talk to other workloads.

-

- Deny-by-default: external inbound disabled, external outbound disabled, internal `none`. Fix via `update_workload`:

-

- - **Inbound** — `external.inboundAllowCIDR` (e.g. `0.0.0.0/0`, or specific CIDRs).

- - **Outbound** — `external.outboundAllowCIDR`, or `outboundAllowHostname` (hostname rules reach only ports 80, 443, 445).

- - **Internal** — `internal.inboundAllowType`: `same-gvc` / `same-org` / `workload-list` (default `none` blocks all workload-to-workload traffic).

-

- Full model: `firewall-networking` skill.

- **Symptoms:** events show probe failures, replicas unready, restarts.

-

- Default probes: serverless gets readiness + liveness TCP on the container port; standard, stateful, and cron have none (cron strips them). Common fixes: raise `initialDelaySeconds` (0-600; default 10 readiness / 60 liveness) for slow starts; raise `periodSeconds` (1-600, default 10) or `timeoutSeconds` (1-600, default 1) for over-aggressive probes; ensure an HTTP path returns 200-399. **Readiness** failure removes the replica from the pool and pauses rollout; **liveness** failure restarts it. An autoscaled workload with no real readiness probe gets traffic before it's ready, causing 502s on scale-up — add an `httpGet` readiness probe (it needs a port; defaults to the container's). Probe tuning: `workload-security` skill.

-

- ### Resource limits & Capacity AI

-

- **Symptoms:** won't schedule, throttled, or Capacity AI not adjusting.

- - CPU/memory must fit org quota; `maxScale` × per-replica resources is enforced at scheduling.

- - **Capacity AI** runs on every workload type, but it is off with CPU-utilization or multi-metric autoscaling. Check the stored `spec.defaultOptions.capacityAI` before blaming the tuner. On cron the new reservation only lands at the next execution.

- - **Stateful sizing** — `minCpu`/`cpu` at most 4000m apart (ratio ≤ 4:1); `minMemory`/`memory` at most 4096Mi apart (ratio ≤ 4:1).

- - **Ephemeral storage** — 1GB per CPU core (minimum 1GB); exceeding it replaces the replica.

- Deep model: `autoscaling-capacity` skill.

- ### Container won't start (restrictions)

- - **UID 1337** is the mesh proxy's UID — running as it excludes the container from the sidecar, disabling mesh communication and mTLS. Override `runAsUser` to another UID in 1-65534 (0/root is rejected).

- - **Reserved container names** — not `istio-proxy` / `queue-proxy` / `istio-validation` (or other reserved names); cannot start with `cpln-` or `debugger-`.

- - **Reserved env vars** — names starting `CPLN_`, plus `K_SERVICE` / `K_CONFIGURATION` / `K_REVISION`, are rejected; each value caps at 4096 characters.

- - **Suspended** — `spec.defaultOptions.suspend: true` stops the workload (min/max scale 0). Clear it, or `cpln workload start WORKLOAD --gvc GVC`.

- **Symptoms:** won't scale, 502s on scale-up, scale-to-zero not working, or an invalid-strategy error on create.

-

- Strategies: `concurrency`, `cpu`, `memory`, `rps`, `latency`, `keda`, `disabled`. Per type: **serverless** has no `latency` or multi-metric; **standard** has no `concurrency`; **stateful** has no `concurrency`; **cron** has no autoscaling at all (the block is removed). **Scale-to-zero:** serverless with `rps` or `concurrency`; standard and stateful only with `metric: keda` (otherwise the update is rejected); cron cannot. For 502s on scale-up, fix the readiness probe (above). Details: `autoscaling-capacity` skill.

-

- ### Termination / graceful shutdown

- **Symptoms:** requests fail during deploys or scale-down, 502/503 on rollout, containers killed abruptly.

- - **Missing `sleep`** — if `sleep` is absent from **any** container, **all** containers get SIGKILL immediately (no drain). Many distroless/minimal images lack it; confirm with `which sleep`. Fix: include `sleep` or add a custom `preStop`.

- - **preStop error** — a failing custom `preStop` in any container SIGKILLs all of them.

- - **Ignores SIGTERM** — after the preStop (default `sleep 45`) the container gets the termination signal, then SIGKILL once `terminationGracePeriodSeconds` (default 90; max 900 without the `cpln/relaxGracePeriodMax` tag) elapses.

- **Symptoms:** can't read mounted files, permission denied, empty cloud volume.

-

- - **Secret volumes** need the identity + `reveal` policy chain (as *Secret access failures*).

- - **Cloud volumes** (S3, GCS, Azure Blob/Files) need an identity, a cloud-access policy, and outbound firewall to the provider hosts (`*.amazonaws.com`, `*.googleapis.com`, `*.blob.core.windows.net` / `*.file.core.windows.net` plus `*.azure.com`); auth is identity-only — embedded keys do not work. Read-only except Azure Files.

- - **Reserved mount paths**: `/dev`, `/dev/log`, `/tmp`, `/var`, `/var/log`. Max 15 volumes; no path may be a parent of another.

- - `filesystemGroupId` defaults to 0 (root) when unset — set it (1-65534) for a non-root app.

- **Symptoms:** a workload can't reach another internally (connection refused or timeout).

-

- - **Target's internal firewall** must allow the caller — `same-gvc` / `same-org` / `workload-list` (default `none`); listing a workload needs `view` on it.

- - **Endpoint** — `http://WORKLOAD.GVC.cpln.local:PORT` (use `http://`; the sidecar adds mTLS). An omitted port defaults to the target's first container port; only listed ports are reachable.

- - **Cross-GVC** — the target must allow `same-org` or list the caller; traffic may span locations and incur egress charges.

-

- ### Dedicated load balancer & domain

-

- **Symptoms:** unreachable after enabling a dedicated LB, TCP broken, wrong Host header.

-

- - **TCP** needs a dedicated LB on the GVC plus a custom Domain with a TCP port — not on default endpoints (HTTP/HTTP2/gRPC only).

- - Enabling/disabling a dedicated LB causes brief DNS-propagation downtime and per-location charges.

- - **Serverless Host header** — a custom domain delivers the canonical endpoint as `Host` (the custom domain moves to `X-Forwarded-Host`); standard and stateful workloads get the custom domain as `Host`.

- - **Protocol compatibility** — the domain port protocol must match the container's: HTTP2 fronts HTTP2 or gRPC, HTTP fronts HTTP.

-

- Routing, TLS, and LBs: `domain` and `ipset-load-balancing` skills.

-

- ## Apply fixes within the schema's limits

-

- A fix the Joi schema rejects at `update_workload` time is worse than none. Before applying a resource or option change, confirm it stays within these (the validator names the violated rule):

-

- - `memory` (MiB) at most 8× `cpu` (millicores); CPU ≥ 25m; memory ≥ 32MiB; `minCpu`/`minMemory` never above `cpu`/`memory`.

- - `runAsUser` / `filesystemGroupId`: 1-65534 (0/root rejected).

- - `terminationGracePeriodSeconds`: ≤ 900 (higher only with the `cpln/relaxGracePeriodMax` tag).

- - `capacityAI` with `metric: cpu` is rejected, as is `capacityAI: true` with a GPU container; every workload type otherwise accepts it.

- - Standard/stateful `minScale: 0` requires `metric: keda`; cron and vm cannot scale to zero.

- - A metric outside the workload type's allow-list is rejected.

-

- Prefer `update_workload` (PATCH — only the fields you set change). For full manifest control, author against `get_resource_schema` then `cpln apply -f workload.yaml --gvc GVC`.

-

- ## Verify

-

- After applying, poll `mcp__cpln__list_deployments` until every location reports ready, confirm the original symptom cleared (events/logs), and report the **canonical endpoint** `list_deployments` returns — never a constructed URL. For a public workload, confirm it actually responds, not just that it is ready.

-

- ## Troubleshooting

-

- | Symptom | Likely cause | First check |

- |---|---|---|

- | Restarts; `OOMKilled` in events | memory cap too low (or Capacity AI downsized) | `query_metrics` memory; raise `memory` + `cpu` |

- | `ImagePullBackOff` / stuck | bad image ref, wrong platform, missing pull secret | events; GVC `pullSecretLinks` |

- | Env vars empty | broken identity + `reveal` chain or wrong reference | `grant_workload_secret_access` |

- | Healthy but 502/503 | spec port ≠ listening port, or a blocked port | workload spec and startup logs |

- | Unreachable / can't call out | deny-by-default firewall | `firewallConfig` |

- | Won't become ready | probe path/port wrong or too aggressive | events; probe config |

- | Can't reach another workload | target internal firewall `none`, or `https://` used | target `inboundAllowType`; use `http://` |

- | 502/503 during deploys | missing `sleep`, or app ignores SIGTERM | `which sleep`; SIGTERM handling |

- | `update_workload` rejected | the fix violates a schema limit | *Apply fixes within the schema's limits* |

-

- ## Quick reference

-

- ### MCP tools

-

- | Tool | Purpose |

- |---|---|

- | `mcp__cpln__list_deployments` | Primary per-location readiness monitor |

- | `mcp__cpln__get_workload_events` | Image / crash / probe / schedule events |

- | `mcp__cpln__get_workload_logs` | App and `_accesslog` logs |

- | `mcp__cpln__list_workload_replicas` | List running replicas |

- | `mcp__cpln__query_metrics` (after `mcp__cpln__list_metrics`) | Memory / CPU / latency pressure |

- | `mcp__cpln__update_workload` | Apply a spec fix (PATCH) |

- | `mcp__cpln__grant_workload_secret_access` | Build the identity + `reveal` chain in one call |

- | `mcp__cpln__get_resource_schema` | Exact fields before a manifest-level fix |

- ### CLI (fallback)

- Use when MCP is unavailable, for an interactive shell, or in CI/CD (service-account `CPLN_TOKEN`).

- ```bash

- cpln workload get WORKLOAD --gvc GVC -o json

- cpln workload eventlog WORKLOAD --gvc GVC -o json

- cpln workload connect WORKLOAD --gvc GVC --location LOCATION

- cpln apply -f workload.yaml --gvc GVC

- ```

- ### Related skills

- - `workload` — primary skill (types, spec shape, tool division); start here.

- - `workload-security` — probe tuning, termination, `securityOptions`, direct LBs.

- - `firewall-networking` — inbound / outbound / internal rules.

- - `autoscaling-capacity` — scaling strategies and Capacity AI.

- - `access-control` / `setup-secret` — the identity + `reveal` chain.

- - `stateful-storage` — volume sets; `domain` / `ipset-load-balancing` — routing and LBs; `image` — registries and pull secrets.

- ## Documentation

- - [Workload Types](https://docs.controlplane.com/reference/workload/types.md)

- - [Containers](https://docs.controlplane.com/reference/workload/containers.md)

- - [Capacity AI](https://docs.controlplane.com/reference/workload/capacity.md)

- - [Firewall](https://docs.controlplane.com/reference/workload/firewall.md)

- - [Termination](https://docs.controlplane.com/reference/workload/termination.md)