kubernetes · diff
git:20260725.20a560e to git:20260919.28d30f9
110 added, 39 removed. Audit A to A.
---
name: kubernetes
description: "Kubernetes operations: debugging, security, RBAC, and infrastructure tooling."
user-invocable: false
context: fork
agent: kubernetes-helm-engineer
routing:
triggers:
# from kubernetes-debugging
- "kubernetes debug"
- "pod failure"
- "pod crashloop"
- "kubectl logs"
- "OOMKilled"
- "pod pending"
# from kubernetes-security
- "kubernetes security"
- "k8s RBAC"
- "RBAC setup"
- "pod security policy"
- "network policy"
# from cobalt-core
- "cobalt core"
- "cobaltcore"
- "kvm-exporter"
- "kvm exporter"
- "hypervisor metrics"
- "libvirt exporter"
- "cloud hypervisor"
category: kubernetes
pairs_with:
- - service-health-check
- - go-patterns
+ - assessment
+ - programming
- prometheus-grafana-engineer
---
# Kubernetes Skill
- Kubernetes debugging, security hardening, and infrastructure tooling. Covers pod triage, RBAC, network policies, and cobaltcore hypervisor components.
+ Three domains: **debugging** (pod triage, networking, resources), **security** (RBAC, pod hardening, network isolation, supply chain), and **cobaltcore** (KVM exporter, hypervisor metrics). Select by request signal, then follow the phases below.
- ## Reference Loading Table
+ Always specify `-n <namespace>` in every kubectl command. Use read-only commands to gather evidence before proposing changes.
- | Signal | Reference | Size |
- |--------|-----------|------|
- | CrashLoopBackOff, OOMKilled, config error, health check, liveness probe, ImagePullBackOff, Pending, FailedScheduling | `references/crash-diagnosis.md` | ~140 lines |
- | service resolution, DNS, CoreDNS, port-forward, NetworkPolicy ingress/egress | `references/network-debugging.md` | ~50 lines |
- | CPU throttling, memory limit, OOMKill, ephemeral storage, DiskPressure, debug container | `references/resource-debugging.md` | ~100 lines |
- | RBAC, Role, RoleBinding, ClusterRole, ServiceAccount, least-privilege | `references/rbac-patterns.md` | ~60 lines |
- | PodSecurity, SecurityContext, runAsNonRoot, readOnlyRootFilesystem, restricted, baseline | `references/pod-security.md` | ~90 lines |
- | NetworkPolicy, default-deny, allow-list, namespace isolation | `references/network-policies.md` | ~70 lines |
- | cosign, Kyverno, OPA, admission controller, Sealed Secrets, External Secrets | `references/supply-chain.md` | ~120 lines |
- | kvm-exporter, metrics, prometheus, libvirt, hypervisor, collector, scrape, steal time, NUMA, cgroups, cloud hypervisor | `references/cobalt-kvm-exporter.md` | ~800 lines |
- | cobaltcore concurrency, goroutine, semaphore, TryLock | `references/cobalt-concurrency-patterns.md` | ~200 lines |
- | cobaltcore testing, mock, moq, Kind cluster | `references/cobalt-testing-patterns.md` | ~200 lines |
- | kubernetes debugging process, triage flow, diagnosis routing | `references/kubernetes-debugging.md` | ~50 lines |
- | kubernetes security process, RBAC + pod security + network hardening | `references/kubernetes-security.md` | ~50 lines |
- | cobaltcore overview, KVM exporter architecture, component identification | `references/cobalt-core.md` | ~50 lines |
+ ---
- **Loading rule.** Read the references whose signals match the task before responding.
+ ## Domain Selection
+ | Signal | Domain |
+ |--------|--------|
+ | CrashLoopBackOff, OOMKilled, ImagePullBackOff, Pending | Debugging |
+ | Service unreachable, DNS failure, port-forward | Debugging (network) |
+ | CPU throttling, memory limit, disk pressure | Debugging (resources) |
+ | RBAC, permissions, roles, ServiceAccount | Security (access) |
+ | Pod hardening, container security, PodSecurity | Security (pods) |
+ | NetworkPolicy, default-deny, namespace isolation | Security (network) |
+ | Image signing, secrets, admission control | Security (supply chain) |
+ | KVM exporter, cobaltcore, hypervisor metrics | Cobaltcore |
+
---
## Phase 1: TRIAGE
- Determine which Kubernetes domain the request targets:
+ ### Debugging Triage Flow
- | Domain | Load references | Action |
- |--------|----------------|--------|
- | Pod failure, CrashLoop, OOM | crash-diagnosis, resource-debugging | Triage flow |
- | Network, DNS, service resolution | network-debugging, network-policies | Connectivity diagnosis |
- | RBAC, permissions, roles | rbac-patterns | Access control |
- | Pod hardening, container security | pod-security | Security posture |
- | Image signing, secrets, admission | supply-chain | Supply chain |
- | Cobaltcore / KVM exporter | kvm-exporter + cobalt refs | Component-specific |
+ Follow this sequence for every pod or workload issue. Do not skip steps -- many failures are only visible in events and describe output, not in logs.
- Always specify `-n <namespace>` explicitly in every kubectl command.
+ ```bash
+ kubectl get pods -n <namespace> -o wide
+ kubectl describe pod <pod-name> -n <namespace>
+ kubectl logs <pod-name> -n <namespace> -c <container-name>
+ kubectl logs <pod-name> -n <namespace> -c <container-name> --previous
+ kubectl get events -n <namespace> --sort-by='.lastTimestamp'
+ kubectl exec -it <pod-name> -n <namespace> -c <container-name> -- /bin/sh
+ ```
- **Gate**: Domain identified and relevant references loaded.
+ Always check `--previous` logs for crashed containers before current logs -- restarting destroys them permanently.
+ **Diagnosis routing**:
+
+ | Symptom | Action |
+ |---------|--------|
+ | CrashLoopBackOff, ImagePullBackOff, Pending, FailedScheduling | Check describe output for events, previous logs, image pull errors |
+ | Service unreachable, DNS failure | Check service endpoints, CoreDNS, NetworkPolicy below |
+ | CPU throttling, OOMKill, disk pressure | Check resource limits, requests vs actual, node capacity |
+ | "no endpoints available for service" | Compare svc selector with pod labels |
+
+ **Network debugging**:
+
+ ```bash
+ # Verify service has endpoints
+ kubectl get endpoints <service-name> -n <namespace>
+ # DNS lookup from inside cluster
+ kubectl run dns-debug --rm -it --restart=Never --image=busybox:1.36 -n <namespace> -- \
+ nslookup <service-name>.<namespace>.svc.cluster.local
+ # Check CoreDNS
+ kubectl get pods -n kube-system -l k8s-app=kube-dns
+ kubectl logs -n kube-system -l k8s-app=kube-dns --tail=50
+ # Port-forward for local testing
+ kubectl port-forward svc/<service-name> -n <namespace> 8080:80
+ ```
+
+ ### Security Domain Selection
+
+ For security requests, provide concrete YAML manifests from the patterns below. Reference-backed specifics, not generic advice.
+
+ **RBAC patterns**: Prefer namespace-scoped Roles over ClusterRoles. Write exact verbs and resources. Create dedicated ServiceAccounts per workload. Set `automountServiceAccountToken: false` on pods that need no API access.
+
+ ```yaml
+ apiVersion: rbac.authorization.k8s.io/v1
+ kind: Role
+ metadata:
+ namespace: app-team
+ name: deployment-reader
+ rules:
+ - apiGroups: ["apps"]
+ resources: ["deployments"]
+ verbs: ["get", "list", "watch"]
+ ```
+
+ **Pod security**: Enforce PodSecurity labels at namespace level. All containers: `runAsNonRoot: true`, `readOnlyRootFilesystem: true`, `allowPrivilegeEscalation: false`, `capabilities: drop: ["ALL"]`. Use distroless base images. Pin image digests.
+
+ ```yaml
+ labels:
+ pod-security.kubernetes.io/enforce: restricted
+ pod-security.kubernetes.io/warn: restricted
+ ```
+
+ **Network policies**: Start with default-deny for ingress and egress. Add allow-list rules per service. Always allow DNS egress (UDP/TCP 53).
+
+ ```yaml
+ apiVersion: networking.k8s.io/v1
+ kind: NetworkPolicy
+ metadata:
+ name: default-deny-all
+ spec:
+ podSelector: {}
+ policyTypes: [Ingress, Egress]
+ ```
+
+ ### Cobaltcore Domain
+
+ Components: KVM Exporter (`cobaltcore-dev/kvm-exporter`). Load `references/cobalt-kvm-exporter.md` for architecture, metric catalogs, configuration, and deployment. Pair with `go-patterns` for code, `prometheus-grafana-engineer` for metrics.
+
---
## Phase 2: DIAGNOSE / RESPOND
- For debugging: follow the triage flow — describe, logs, events, exec. Use read-only commands to gather evidence before proposing changes.
-
- For security: provide concrete YAML manifests and specific configurations. Answer with reference-backed specifics, not generic advice.
+ **Debugging**: Follow the triage flow. Gather evidence with read-only commands before proposing changes.
- For cobaltcore: use component-specific reference knowledge for architecture, metrics, configuration, and deployment details.
+ **Security**: Provide copy-paste-ready YAML using the patterns in Phase 1.
- **Gate**: Specific, reference-backed diagnosis or response provided.
+ **Cobaltcore**: Use component-specific references for architecture, metrics, concurrency patterns, and testing.
---
## Phase 3: VERIFY
- For debugging: confirm the fix resolves the symptom.
- For security: validate against the misconfiguration table in supply-chain.md.
- For cobaltcore: verify against component test patterns.
+ - **Debugging**: Confirm the fix resolves the symptom with the same triage commands.
+ - **Security**: Validate against the PodSecurity standards and RBAC least-privilege patterns above.
+ - **Cobaltcore**: Verify against component test patterns in `references/cobalt-testing-patterns.md`.
+
+ ---
+
+ ## Deep References
+
+ | Signal | Reference | Content |
+ |--------|-----------|---------|
+ | KVM exporter architecture, metrics, config | `references/cobalt-kvm-exporter.md` | Full component reference (463 lines) |
+ | Cobaltcore concurrency, goroutine, semaphore | `references/cobalt-concurrency-patterns.md` | Go concurrency patterns (268 lines) |
+ | Cobaltcore testing, mock, Kind cluster | `references/cobalt-testing-patterns.md` | Testing strategies (271 lines) |