workload-security · git:20260612.b890dfa · 2026-06-12 · sha256 4fd3348da68c58ba
workload-security git:20260612.b890dfaA
Immutable. This exact content never changes and is served at /api/v1/blob/4fd3348da68c58ba.
---
name: workload-security
description: "Hardens workloads for production on Control Plane. Use when the user asks about JWT authentication, workload security options, TLS, geo-location headers, graceful shutdown, readiness/liveness probes, or securing public-facing workloads."
---
# Workload Security & Production Hardening
> **Tool availability:** some MCP tools named here live in the `full` toolset profile — if one is not advertised on this connection, tell the user to reconnect the MCP server with `?toolsets=full` (or use the `cpln` CLI fallback). Reads and deletes work on every profile via the generic `list_resources` / `get_resource` / `delete_resource` tools.
Deep hardening detail. Probe basics, the readiness-vs-liveness model, the LB picker, and post-deploy verification live in the `workload` skill — this skill carries the full probe timing/defaults, JWT/Envoy, security, direct-LB, geo, and graceful-termination detail.
## Health Probes
Define both `readinessProbe` and `livenessProbe` as distinct probes (the readiness-vs-liveness model is in the `workload` skill).
**Defaults by workload type:**
- **Serverless** — a TCP probe on the listening port is enabled by default. Better than nothing, but a real `httpGet` against an app endpoint catches more failure modes (DB unreachable, dependency timeout, deadlock).
- **Standard / Stateful** — probes are **disabled by default**; add them explicitly for any production workload.
- **Cron** — probes are ignored.
If a workload genuinely has no HTTP healthcheck, use `tcpSocket` against the listening port as a baseline — never ship without probes.
### Probe schema
Each probe takes exactly one of `exec`, `grpc`, `tcpSocket`, `httpGet` (xor). Timing fields and ranges:
| Field | Range | Default |
|---|---|---|
| `initialDelaySeconds` | 0-600 | 10 (readiness) / 60 (liveness) |
| `periodSeconds` | 1-600 | 10 |
| `timeoutSeconds` | 1-600 | 1 |
| `successThreshold` | 1-20 | 1 |
| `failureThreshold` | 1-20 | 3 |
Liveness is typically looser than readiness (e.g. `periodSeconds: 30`) — restarts are expensive.
### Production-grade probe example
```yaml
spec:
containers:
- name: api
image: //image/api:v1.0
ports:
- number: 8080
protocol: http
readinessProbe:
httpGet:
path: /healthz/ready
port: 8080
initialDelaySeconds: 5
periodSeconds: 10
timeoutSeconds: 2
failureThreshold: 3
livenessProbe:
httpGet:
path: /healthz/live
port: 8080
initialDelaySeconds: 30
periodSeconds: 30
timeoutSeconds: 3
failureThreshold: 3
```
## JWT Authentication
JWT Authentication validates JSON Web Tokens at the infrastructure level (Envoy sidecar) before requests reach the workload. Configured under `sidecar.envoy` — set it on a workload with `mcp__cpln__configure_workload_sidecar` (or on a GVC by editing the GVC spec, where it applies to all workloads in that GVC).
### Configuration Structure
JWT auth uses the Envoy `jwt_authn` HTTP filter:
| Field | Value |
|---|---|
| `name` | `envoy.filters.http.jwt_authn` |
| `typed_config."@type"` | `type.googleapis.com/envoy.extensions.filters.http.jwt_authn.v3.JwtAuthentication` |
| `priority` | Integer for ordering multiple filters |
| `typed_config.providers` | Map of provider name to provider config |
| `typed_config.rules` | Array of rules controlling which requests require valid JWTs |
### Provider Configuration
Each provider is a key in `typed_config.providers`:
| Field | Type | Description |
|---|---|---|
| `issuer` | string | URL of the domain that issued the JWT |
| `audiences` | string[] | Accepted audiences for the JWT |
| `claim_to_headers` | object[] | Maps JWT claims to request headers forwarded to the workload |
| `remote_jwks` | object | JWKS public key resolution and caching |
**Provider naming:** Names starting with `cpln_` are configured through the UI and have restricted settings (e.g., `cache_duration` must equal `http_uri.timeout`). Non-`cpln_` prefixed names allow full Envoy JWT configuration.
### Claim-to-Header Mapping
Extract JWT claims into headers forwarded to the workload:
| Field | Type | Description |
|---|---|---|
| `header_name` | string | Header name added to the forwarded request |
| `claim_name` | string | Claim extracted from the JWT |
### Remote JWKS
Public key resolution for JWT verification:
| Field | Type | Description |
|---|---|---|
| `http_uri.uri` | string | Endpoint for JWKS public key lookup |
| `http_uri.cluster` | string | Must match the cluster name for this provider |
| `http_uri.timeout` | string | Timeout in `Ns` format (e.g., `10s`) |
| `cache_duration` | string | JWKS cache duration in `Ns` format (e.g., `300s`) |
### Rules
Rules evaluated in order; the first matching rule applies:
| Field | Type | Description |
|---|---|---|
| `match.prefix` | string | URI prefix for this match |
| `match.headers` | string[] | Optional headers that must exist in the request |
| `requires.provider_name` | string | Provider for JWT verification; omit to allow all requests |
### Clusters
Each provider requires a cluster definition for HTTPS key lookup. Replace `${providerName}` and `${providerEndpoint}`:
```yaml
sidecar:
envoy:
clusters:
- name: cpln_${providerName}
type: STRICT_DNS
load_assignment:
cluster_name: cpln_${providerName}
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: ${providerEndpoint}
port_value: 443
transport_socket:
name: envoy.transport_sockets.tls
```
### Full JWT Example
This configuration validates JWTs from `https://foo.com/auth`, extracts `user.special` claim into `X_SPECIAL_USER` header, requires JWT for all paths except `/metric`:
```yaml
sidecar:
envoy:
clusters:
- name: cpln_foo
type: STRICT_DNS
load_assignment:
cluster_name: cpln_foo
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: foo.com
port_value: 443
transport_socket:
name: envoy.transport_sockets.tls
http:
- name: envoy.filters.http.jwt_authn
priority: 50
typed_config:
"@type": >-
type.googleapis.com/envoy.extensions.filters.http.jwt_authn.v3.JwtAuthentication
providers:
cpln_foo:
audiences:
- myaudience
claim_to_headers:
- claim_name: user.special
header_name: X_SPECIAL_USER
issuer: https://foo.com/auth
remote_jwks:
cache_duration: 5s
http_uri:
cluster: cpln_foo
timeout: 5s
uri: https://foo.com/auth
rules:
- match:
headers: []
prefix: /metric
- match:
headers: []
prefix: /
requires:
provider_name: cpln_foo
```
For any OIDC-compliant provider (Auth0, Firebase, Cognito, Okta), set `issuer` to the provider's issuer URL, `remote_jwks.http_uri.uri` to its JWKS endpoint (typically `/.well-known/jwks.json`), and `audiences` to your client ID or API identifier. Use `rules` to exclude health/metrics paths, and `claim_to_headers` to pass identity context to the workload without re-parsing the token.
## Security Options
Runtime security settings for the workload, configured under `spec.securityOptions`:
| Field | Type | Range | Default | Description |
|---|---|---|---|---|
| `filesystemGroupId` | integer | 1-65534 | 0 (root) | Group ID assigned to any mounted volumes |
| `runAsUser` | integer | 1-65534 | Image default | User ID assigned to all container processes |
```yaml
spec:
securityOptions:
filesystemGroupId: 1000
runAsUser: 1000
```
Set `runAsUser` to a non-root UID for defense in depth. Set `filesystemGroupId` when containers need shared access to mounted volumes (e.g., volume sets).
### Workload Permissions
Policies control workload access using these permissions:
| Permission | Description | Implies |
|---|---|---|
| `view` | Read-only access | |
| `edit` | Modify existing workloads | `view` |
| `create` | Create new workloads | |
| `delete` | Delete existing workloads | |
| `connect` | Open interactive shell to replica | |
| `configureLoadBalancer` | Configure the workload load balancer | |
| `exec` | Execute commands | `exec.runCronWorkload`, `exec.stopReplica` |
| `exec.runCronWorkload` | Force a cron workload to run | |
| `exec.stopReplica` | Force a replica to be stopped | |
| `manage` | Full access | All of the above |
## Direct Load Balancers
Expose workload ports directly through cloud load balancers in each deployment location. Unlike the shared load balancer, direct LBs support custom TCP/UDP ports and do not require domain registration. Use for non-HTTP protocols, custom ports, or when domain registration is not needed; workloads are responsible for their own TLS certificates. Set `spec.loadBalancer` (direct, geo headers, replicaDirect) with `mcp__cpln__configure_workload_load_balancer`.
Under `spec.loadBalancer.direct`:
| Field | Type | Description |
|---|---|---|
| `enabled` | boolean (required) | Enable/disable the direct LB. When `false`, LB is stopped and no charges accrue |
| `ports` | array | Ports exposed by the load balancer |
| `ipSet` | string (optional) | Link to an IP set for reserved static IPs |
Each entry in the `ports` array:
| Field | Type | Description |
|---|---|---|
| `externalPort` | number | Public port. Range: 22-32768 |
| `protocol` | string | `TCP` or `UDP` |
| `scheme` | string (optional) | URL scheme for UI links: `http`, `tcp`, `https`, `ws`, `wss`. Default: `https` |
| `containerPort` | number | Target container port (80-65535, excludes reserved ports) |
**Reserved container ports (cannot be used):** 8012, 8022, 9090, 9091, 15000, 15001, 15006, 15020, 15021, 15090, 41000.
```yaml
spec:
loadBalancer:
direct:
enabled: true
ports:
- externalPort: 5432
protocol: TCP
scheme: tcp
containerPort: 5432
- externalPort: 9000
protocol: UDP
containerPort: 9000
```
### Geo DNS Routing
Each location gets a public address configured as a target of the Geo DNS endpoint. Latency-based routing distributes traffic across locations. The workload DNS record can be used as a CNAME target for custom domains without registering domains in Control Plane.
### Replica Direct (Stateful Only)
For stateful workloads, `replicaDirect` enables individual replica endpoints:
```yaml
spec:
loadBalancer:
replicaDirect: true
```
Endpoint format:
- External: `<workloadName>-<gvcAlias>-<replicaIndex>.<locationName>.controlplane.us`
- Internal: `replica-<replicaIndex>.<workloadName>.<locationName>.<gvcName>.cpln.local:<port>`
## Geo Location Headers
Add geographic information to inbound HTTP requests using MaxMind GeoLite2 data. Configured under `spec.loadBalancer.geoLocation`:
| Field | Type | Description |
|---|---|---|
| `enabled` | boolean | Enable geo headers (default: `false`) |
| `headers.asn` | string | Header name for ASN (max 128 chars) |
| `headers.city` | string | Header name for city (max 128 chars) |
| `headers.country` | string | Header name for country (max 128 chars) |
| `headers.region` | string | Header name for region (max 128 chars) |
**Rules:**
- At least one header must be set when enabled
- All header names must be unique
- Existing headers with the same names are replaced
- Only works on workloads exposing an HTTP port
- Replicas receive the latest IP-to-geo database on startup
```yaml
spec:
loadBalancer:
geoLocation:
enabled: true
headers:
asn: X-GeoIP-ASN
city: X-GeoIP-City
country: X-GeoIP-Country
region: X-GeoIP-Region
```
Combine geo headers with header-based firewall filtering for geo-filtering (see the **cpln-firewall-networking** skill).
## Graceful Termination
Controls how workload replicas are removed during scaling, version updates, Capacity AI rollouts, and maintenance.
### Termination Grace Period
Configured under `spec.rolloutOptions.terminationGracePeriodSeconds`:
| Field | Type | Range | Default |
|---|---|---|---|
| `terminationGracePeriodSeconds` | number | 0-900 | 90 |
Total time available for graceful shutdown before all containers receive SIGKILL. Default 90s works for most workloads; increase (up to 900) for workloads with long-running requests.
### Rollout Options
Full `spec.rolloutOptions` configuration:
| Field | Type | Default | Description |
|---|---|---|---|
| `minReadySeconds` | integer | 0 | Minimum seconds a container must run without crashing to be considered available |
| `maxSurgeReplicas` | integer or percent | | Max replicas above desired count during rollout |
| `maxUnavailableReplicas` | integer or percent | | Max unavailable replicas during rollout (not for stateful workloads) |
| `scalingPolicy` | string | `OrderedReady` | `OrderedReady` or `Parallel` |
| `terminationGracePeriodSeconds` | number | 90 | Graceful shutdown timeout (0-900) |
### Termination Sequence
1. **Load balancer update** -- replica removed from pool (up to 10 seconds)
2. **Sidecar termination** (managed by Control Plane):
- **Hold phase**: continues normally for `terminationGracePeriodSeconds - 10` seconds (default: 80s)
- **Monitoring phase**: waits for active connections to complete
- **Drain phase**: stops accepting new connections, verifies completion, shuts down
3. **Container termination**:
- **Default preStop hook**: executes `sh -c "sleep N"` where N = half the grace period (default: 45s)
- After preStop completes, the container receives **SIGTERM** (the Kubernetes default termination signal)
- Remaining time for graceful shutdown before **SIGKILL**
### Critical Warnings
- If `sleep` is not available in **any** container, ALL containers receive SIGKILL immediately.
- If a custom preStop hook throws an error in **any** container, ALL containers receive SIGKILL immediately.
### Custom PreStop Hook
Only implement if the workload requires specific termination logic (graceful connection draining, containers without `sh`/`sleep`, or custom request handling during shutdown). A custom preStop hook must include a delay or check for ongoing requests to give load balancers time to update, and must be tested thoroughly — errors cause immediate SIGKILL for all containers.
```yaml
spec:
rolloutOptions:
terminationGracePeriodSeconds: 120
minReadySeconds: 10
maxSurgeReplicas: "25%"
maxUnavailableReplicas: "25%"
scalingPolicy: Parallel
```
## Combined Hardening
- Combine JWT auth with firewall CIDR rules for layered security.
- Use geo location headers with header-based firewall filtering for geo-blocking.
- Pair direct LBs with an IP set for reserved static IPs, and set an appropriate `terminationGracePeriodSeconds`.
## Quick Reference
### MCP Tools (use these first)
| Tool | Purpose |
|---|---|
| `mcp__cpln__create_workload` | Create a workload with security options, probes, and rollout settings |
| `mcp__cpln__update_workload` | Update security options, probes, or rollout config (PATCH — only sent fields change) |
| `mcp__cpln__configure_workload_load_balancer` | Set/clear the load balancer (direct, geo headers, replicaDirect) |
| `mcp__cpln__configure_workload_sidecar` | Set/clear the Envoy sidecar (JWT auth, filter chain) |
| `mcp__cpln__get_resource` (kind="workload") | Inspect current workload configuration (read before any update for rollback) |
| `mcp__cpln__list_resources` (kind="workload") | Find workloads in a GVC before targeting one |
| `mcp__cpln__delete_resource` (kind="workload") | Delete a workload (destructive — confirm blast radius first) |
| `mcp__cpln__list_deployments` | PRIMARY post-deploy readiness monitor — poll until ready, surfaces probe failures per location |
| `mcp__cpln__get_workload_events` | Probe/liveness reason + message when a deploy fails |
| `mcp__cpln__get_workload_logs` | App-side logs to diagnose security/probe issues |
| `mcp__cpln__list_workload_replicas` | List running replicas before exec |
| `mcp__cpln__workload_exec` | Run a one-off command inside a replica (audited; targets live traffic) |
| `mcp__cpln__workload_start_cron` | Trigger an out-of-band run of a cron workload |
### CLI (fallback)
Use the CLI when the MCP server is unavailable or unauthenticated, in CI/CD pipelines (service-account `CPLN_TOKEN`), or for interactive debugging (`cpln workload exec`, `cpln workload connect`, `cpln logs`).
```bash
# Get workload YAML for editing (yaml-slim strips IDs, timestamps, metadata)
cpln workload get WORKLOAD_NAME --gvc GVC_NAME -o yaml-slim > workload.yaml
# Apply updated workload configuration (CI/CD: add --ready to block until deployed)
cpln apply --file workload.yaml --gvc GVC_NAME
```
### Related Skills
- **cpln-workload** — Start here: the primary workload skill (types, defaults, spec shape) that routes here for security & hardening.
- **cpln-firewall-networking** — CIDR rules, header filtering, geo-filtering, LB types
- **cpln-access-control** — Policies, service accounts, permissions
- **cpln-autoscaling-capacity** — Scaling and Capacity AI settings
- **cpln-stateful-storage** — Volume sets (use with `filesystemGroupId`)
## Documentation
For the latest reference, see:
- [Workload Security Reference](https://docs.controlplane.com/reference/workload/security.md)
- [JWT Auth Reference](https://docs.controlplane.com/reference/workload/jwt-auth.md)
- [Termination Reference](https://docs.controlplane.com/reference/workload/termination.md)
- [Load Balancing Reference](https://docs.controlplane.com/reference/workload/load-balancing.md)
- [Firewall Reference](https://docs.controlplane.com/reference/workload/firewall.md)
- [Workload General Reference](https://docs.controlplane.com/reference/workload/general.md)