devops · diff
git:20260314.9629b35 to git:20260906.aa56e0a
165 added, 300 removed. Audit A to A.
---
name: devops
description: >-
- Infrastructure engineering discipline: infrastructure-as-code principles, deliverable
- quality standards, environment parity, change management, security posture, observability,
- incident response, policy-as-code, supply chain integrity, and disaster recovery. Invoke
- whenever task involves any interaction with infrastructure work — provisioning, configuring,
- deploying, monitoring, or operating infrastructure systems.
+ Carry an infrastructure change from intent to verified effect: what a change must carry before apply, what "applied"
+ does not prove, which changes a revert cannot reverse, what a credential can do that the instruction never said, and
+ the evidence that closes a change. Not how any one tool is written.
+ when_to_use: >-
+ Invoke whenever infrastructure is changed at all — provisioning, configuring, deploying, upgrading, backing up, or
+ operating a host, container, cluster, VM, share, network device, DNS zone, or certificate, with any tool, and before
+ the tool's own skill. Also invoke on the symptoms: a change reported success and the target still runs the old
+ state, a revert that did not restore, a plan that proposes destroying what should exist, an operator locked out by
+ their own change, a backup nobody has restored, a secret in a plan or a log, a certificate renewed but not served,
+ or a "done" with no post-apply check behind it. Covers the discipline of the change; how a play, image, VM, share,
+ or ruleset is written belongs to the ansible, containers, proxmox, unraid, and networking skills.
+ compatibility: Uses Claude Code frontmatter beyond the Agent Skills spec (when_to_use)
---
- # DevOps Discipline
-
- **Declarative, reproducible, observable, secure by default.**
-
- Every infrastructure failure traces to one of four root causes:
-
- - Undeclared state — infrastructure exists that isn't in code
- - Unverified changes — changes deployed without testing or rollback plan
- - Invisible systems — infrastructure that can't be monitored or debugged
- - Assumed security — security treated as a later hardening step instead of a default
-
- This skill prevents all four.
-
- ## References
-
- - **IaC principles** — [`${CLAUDE_SKILL_DIR}/references/iac-principles.md`]: GitOps v1.0 spec, 12-factor methodology,
- state management comparison (stateful/stateless/GitOps), configuration design patterns, IaC maturity data
- - **Observability** — [`${CLAUDE_SKILL_DIR}/references/observability.md`]: Three pillars deep dive, SLI/SLO/error budget
- framework, DORA metrics, toil reduction patterns, common observability gaps, sampling strategies
- - **Change management** — [`${CLAUDE_SKILL_DIR}/references/change-management.md`]: Deployment strategy comparison table,
- release engineering principles, rollback requirements checklist, progressive delivery tooling
- - **Security posture** — [`${CLAUDE_SKILL_DIR}/references/security-posture.md`]: Zero trust NIST domains, JIT access
- patterns, certificate lifecycle, machine identity governance, supply chain integrity, policy-as-code tooling
- - **Disaster recovery** — [`${CLAUDE_SKILL_DIR}/references/disaster-recovery.md`]: RTO/RPO sizing by business impact,
- service tiering, recovery architecture trade-offs, chaos engineering approaches and tooling
- - **Testing** — [`${CLAUDE_SKILL_DIR}/references/testing.md`]: Testing pyramid layers with tool lists, IaC quality
- metrics, runbook format and automation maturity levels, incident response patterns
-
- ## Core Principles
-
- These apply to all infrastructure work regardless of tool.
-
- ### Everything in Code
-
- - All infrastructure is defined in version-controlled, declarative code
- - No manual changes through CLI, console, or SSH. If it's not in code, it doesn't exist
- - Configuration is separated from code — config varies between environments, code does not
- - Changes go through code review before deployment
- - Every change is auditable: who changed what, when, why
-
- ### Idempotent and Reproducible
-
- - Applying configuration twice produces zero changes on the second run. If the second apply shows diffs, the
- configuration is broken
- - Given the same inputs, the system produces identical infrastructure every time
- - Pin all dependency versions — tools, providers, modules, images. "Latest" is not a version
- - Builds are hermetic: independent of the machine running them
-
- **Pinning example:**
-
- - Bad: `image: nginx:latest` / `version: ">=2.0"` / `hashicorp/consul:*`
- - Good: `image: nginx:1.25.4@sha256:6a5...` / `version: "2.3.1"` / `hashicorp/consul:1.17.2`
-
- ### Immutable Over Mutable
-
- - Prefer replacing infrastructure over patching in place
- - Servers are cattle, not pets — replaceable units provisioned from known definitions
- - Mutable infrastructure accumulates drift, undocumented changes, and snowflake state
- - When mutation is unavoidable (stateful systems), treat it as a managed exception with explicit drift detection
-
- ### Secure by Default
-
- - Least privilege everywhere — no wildcard permissions, no shared credentials, no "tighten later". 99% of cloud
- identities use less than 5% of granted permissions — right-size from the start
- - Zero trust — network location does not grant trust. Authenticate and authorize every request regardless of origin.
- Verify across all NIST domains: identity, device, network, application workload, data, visibility/analytics,
- governance
- - Assume breach — treat all traffic (including internal) as potentially malicious. Focus on internal monitoring,
- session-based auth, limiting lateral movement
- - Just-in-time access — elevated permissions are temporary (4-8 hour maximum lifespan) and expire automatically. Static
- elevated permissions accumulate into permission sprawl (~25% growth per quarter without lifecycle management)
- - Encrypt in transit (TLS everywhere, no "internal-only" exceptions) and at rest
- - Never commit secrets to version control. Use a secrets manager. Rotate on schedule
- - Pin dependencies, verify checksums and signatures, scan for vulnerabilities. See
- [`${CLAUDE_SKILL_DIR}/references/security-posture.md`] for supply chain integrity requirements
-
- **Secrets example:**
-
- - Bad: `db_password: "hunter2"` in a YAML file committed to git
- - Bad: `ENV DB_PASS=hunter2` baked into a Dockerfile
- - Good: `db_password: "{{ vault_db_password }}"` with Ansible Vault
- - Good: `password_file: /run/secrets/db_pass` with Docker secrets or SOPS
-
- ### Observable
-
- - Every component exposes the four golden signals: latency, traffic, errors, saturation
- - Collect all three observability pillars: metrics, logs, and traces. Metrics for alerting and trends, logs for event
- context, traces for distributed request paths
- - Monitoring (reactive, predefined thresholds) is necessary but not sufficient. Observability (proactive, infer internal
- state from outputs) is required for distributed systems where failure modes are emergent and unpredictable
- - Alert on symptoms (what's broken), not causes (why). Use causes for debugging
- - Every alert must be actionable — if the response is "ignore it," delete the alert. Alert fatigue is the primary
- obstacle to fast incident response
- - Use distributions and percentiles, not averages. Averages hide tail latency
- - Monitor saturation proactively — latency spikes are leading indicators of saturation
- - Manage observability configuration as code — dashboards, alerts, SLOs, and notification policies live in Git and
- deploy via CI/CD
-
- ### Recoverable
-
- - Every stateful component has a backup strategy with defined RTO and RPO
- - Backups are automated, encrypted, stored off-site, and regularly test-restored
- - Every change has a defined rollback path before deployment
- - Recovery procedures are documented as runbooks with specific commands, not prose
- - Validate resilience through chaos engineering — controlled fault injection to verify recovery procedures work before
- real incidents occur
-
- ## Deliverable Standards
-
- <deliverable-checklist>
- An infrastructure deliverable is not done until all of these are true:
-
- - [ ] **Defined in code** — declarative, version-controlled, reviewed
- - [ ] **Idempotent** — second apply produces zero changes
- - [ ] **Tested** — static analysis passes, integration verified in test environment
- - [ ] **Secrets managed** — no secrets in code or config; injected at runtime
- - [ ] **Monitoring in place** — golden signals exposed, alerts configured, dashboard exists
- - [ ] **Rollback defined** — documented procedure, tested, with known duration
- - [ ] **Backup strategy** — automated backups for stateful components, restore tested
- - [ ] **Documented** — topology, access, dependencies, runbooks, recovery procedures
- - [ ] **Environment parity** — dev/staging/prod use the same definitions with environment-specific variables.
- Differences are explicit and minimized
- - [ ] **Security reviewed** — least privilege, encryption, access control verified
- - [ ] **Supply chain verified** — dependencies pinned, checksums validated, images signed where applicable,
- vulnerability scanning in CI/CD
- - [ ] **Compliance enforced** — policy-as-code checks pass in CI/CD pipeline; organizational policies (naming, tagging,
- allowed regions) are automated
- - [ ] **Drift detection active** — continuous monitoring for configuration drift with alerting or auto-remediation
- - [ ] **Deprovisioning defined** — teardown procedure exists, handles dependencies in correct order, verified to leave
- no orphaned resources </deliverable-checklist>
-
- ## Change Management
-
- ### Before Deploying
-
- - Review the plan diff — what will be created, modified, destroyed
- - Verify rollback path exists and is documented
- - Confirm monitoring will detect problems during rollout
- - Assess blast radius — what breaks if this change fails?
-
- **Blast radius example:**
-
- - Bad: "Should be fine, it's just a config change" — deployed to all nodes simultaneously
- - Good: "This modifies the load balancer config. Failure affects all inbound traffic. Deploy to canary first, verify
- health for 10 minutes, then roll to remaining nodes"
-
- ### Deployment Strategy Selection
-
- - **Default:** Canary for critical infrastructure, rolling for routine changes
- - **Blue-green:** When mixed-version state is unacceptable
- - **Progressive delivery:** Combine canary with automated analysis — query metrics during rollout and auto-promote or
- auto-rollback based on SLI thresholds
- - **Never:** All-at-once to production. Gradual rollout always
-
- ### After Deploying
-
- - Verify the change produced the expected result
- - Confirm monitoring shows healthy state
- - Watch for delayed effects (connection pool exhaustion, cache warming, DNS propagation)
- - Document any deviations from the plan
-
- See [`${CLAUDE_SKILL_DIR}/references/change-management.md`] for strategy comparison tables and rollback requirements.
-
- ## Drift Management
-
- Drift — divergence between declared state and actual state — is the single most common source of deployment failures.
- Less than one-third of organizations continuously monitor for drift; the rest discover it during outages.
-
- **Sources of drift:** manual changes ("quick fixes" via CLI/console), failed partial applies, external system changes,
- provider API updates, auto-scalers and controllers modifying resources legitimately.
-
- **Detection approaches by tool type:**
-
- - **Stateful tools** (Terraform) — compare state file against live infrastructure to detect additions, modifications,
- and deletions
- - **Stateless tools** (Ansible) — can only verify items in current playbook; cannot detect removed resources. Requires
- explicit `absent` declarations for cleanup
- - **GitOps reconciliation** (Flux, Argo CD) — pull-based agents continuously compare desired state in Git against live
- cluster state. Auto-remediate or alert
-
- **Drift example:**
-
- - Bad: "The load balancer stopped working after deploy" — someone added a firewall rule via the console last week; it
- wasn't in code; the deploy overwrote it
- - Good: Drift detection caught the manual rule within hours, team codified it, and the next deploy included it
-
- **Reconciliation safety:** Configure exclusion lists before enabling auto-sync. Controllers (HPA, cert-manager,
- Crossplane) legitimately modify resources — auto- remediating those changes creates reconciliation loops that
- destabilize the cluster. Start with detection + alerting, then enable auto-remediation after confidence in exclusion
- lists is established.
-
- ## Environment Parity
-
- Three gaps to close between development and production:
-
- - **Time gap** — deploy to staging immediately after code review, not days later
- - **Personnel gap** — the people who write infrastructure code also deploy it
- - **Tools gap** — same backing services, same OS, same versions across environments
-
- Unavoidable differences (scale, domain names, credentials) are handled through environment-specific variables, not
- through separate code paths. If dev and prod use different infrastructure definitions, they will diverge.
-
- **Parity example:**
-
- - Bad: dev uses SQLite, staging uses MySQL, prod uses PostgreSQL — "they're all SQL"
- - Good: all environments use PostgreSQL with the same major version; only connection strings and replica counts differ
- via environment variables
-
- **Indicators of parity problems:**
-
- - Production-only bugs that cannot be reproduced in lower environments
- - Extended MTTR because engineers spend time identifying environment differences rather than fixing the actual issue
- - High change failure rates and deployment toil from manual environment verification
- - Deployments are treated as risky events rather than routine operations
+ Three biases decide most calls:
- **Preventing drift:** Use IaC to provision all environments from the same definitions. Use containerization to package
- applications with their dependencies. Use continuous drift detection (state file comparison or GitOps reconciliation) to
- catch divergence before it causes incidents. Treat environments as immutable — replace rather than patch. The four
- pillars of parity: standardization, immutability, automation, observability.
+ - **Applied is not in effect.** A tool's success reports what the tool did to its own target. Whether the effect exists
+ is reported by the service, the socket, the resolver, the peer, or the guest. A change closes on the second kind of
+ evidence and never on the first.
+ - **A revert is not a rollback.** Re-applying the previous declaration restores what a tool can recreate from a
+ declaration and nothing the forward change emitted: deleted data, rotated secrets, withdrawn routes, migrated schemas,
+ destroyed objects.
+ - **The target is decided by the credential and the state the command runs against, not by the code.** A reviewed change
+ lands where the kubeconfig, backend, inventory, or token points, and a plan is a projection of one state at one
+ moment. Read both before the first mutation.
- ## Testing Requirements
+ ## Target
- ### Minimum Testing Bar
+ - **Name the target by identifiers that cannot be confused**: account or project ID, cluster API endpoint, host key and
+ address, backend and workspace, node and VMID, zone ID. `prod` is a label, not a target. GitHub deployed a reviewed
+ network configuration to the wrong environment on 2024-03-11.
+ - **Plan against the state that apply will use, immediately before apply.** Never apply a plan older than the last
+ change to the target, and never a plan made with refresh disabled. GitLab applied a three-week-old Terraform plan on
+ 2023-07-07; drift had turned it into a plan that destroyed 617 resources.
+ - **A missing or substituted state file is a stop, not a step.** With no state, a plan proposes creating everything;
+ with an old state copied in, a destroy looks like the clean option. An agent destroyed a production VPC, database, and
+ cluster in February 2026 by reasoning exactly that way.
+ - **Read the plan for destroy and replace before anything else.** A destroy count above zero on a change that is not a
+ destroy, or `# forces replacement` on a stateful resource, stops the apply until the reason is written down.
+ - **A blank parameter is a value.** Resolved defaults are part of the plan. Google's internal tool turned one blank
+ field into a one-year term and deleted a customer's private cloud a year later, with no notification because the
+ deletion was not customer-initiated.
+ - **Prefer a change that names its objects over a filter that resolves at run time.** Cloudflare's 2026-02-20 cleanup
+ called an API with an empty filter; the server returned every prefix and the task deleted them all.
+ - **Before a firewall, sshd, auth, or routing change, hold an independent way back in**: a second authenticated session,
+ a console, or an out-of-band path, plus a timed revert that restores the previous ruleset unless the operator cancels
+ it. The SRE workbook's rule for a change that can remove operator control is automatic rollback, or at minimum an
+ automatic stop.
+ - **Capture what the reverse needs before apply**: the previous value, the backup taken with the method the data
+ requires, the identifiers, and the consumers of anything renamed. Confirm the credential performing the change cannot
+ delete that backup.
+ - **Use the credential the change needs and no other.** A broader credential found in the environment is not permission.
+ An agent with a staging credential mismatch found a token scoped for any operation, guessed that a volume delete would
+ stay in staging, and removed a production database and its co-located backups in one call (April 2026). Deletion
+ protection on the resource itself is the layer that survives a removed resource block; `prevent_destroy` does not.
+ - **A prose prohibition does not hold against a token that can.** A code freeze in the instructions did not stop a
+ Replit agent from deleting a production database in July 2025. Where the only guard on a destructive path is prose,
+ say so and stop; do not proceed under it.
- Every infrastructure change must pass:
+ ## Apply
- 1. **Static analysis** — linting, security scanning, policy-as-code checks. Run on every commit. Tools: `tflint`,
- `checkov`, `ansible-lint`, `trivy`, `kics`
- 2. **Idempotency verification** — apply twice, confirm zero changes on second run
- 3. **Plan review** — human review of the plan diff before apply
+ - **One change per apply.** A change that bundles concerns cannot be reverted separately or diagnosed separately.
+ - **Stage where there is traffic to split**, comparing the canary against the rest, on a dozen metrics at most, for long
+ enough that the system can react. Where there is nothing to split (one host, one router, one NAS, one control plane),
+ the substitute is one object at a time, a wait between objects longer than the delayed-failure window, and the old
+ path kept alive until the new one is proven. Azure Front Door's 2025-10-29 configuration crashed the data plane five
+ minutes after passing every stage, and by then it had been written into Last Known Good.
+ - **Apply where the push can be stopped.** A global push with no stage is how a dormant June change withdrew
+ Cloudflare's resolver prefixes worldwide in July 2025, and how a feature with no flag crash-looped Google Cloud's
+ control plane on 2025-06-12. Google's remediation: every change to a critical binary behind a flag, disabled by
+ default.
+ - **Never disable a validator, a lock, a deletion protection, a health check, or a policy to make an apply pass.** A
+ failing safety is a finding about the change, not an obstacle to it.
+ - **Do not re-run a failed destructive workflow blind.** Run-once guards do not hold across separate runs; Google's
+ satellite decommission re-run read an empty machine list as "no constraint" and erased every edge machine. Read the
+ state, then decide.
+ - **A gate that blocks a change must not block its rollback.** The same Google postmortem found a 25 percent delta check
+ that let the destructive change through at 23 percent and blocked the restore at 29.
+ - **Treat generated configuration as untrusted input**: bound its size and cardinality at the consumer. Cloudflare's
+ 2025-11-18 outage was a feature file that doubled past a hard limit after a permissions change upstream.
- ### Recommended Testing
+ ## Effect
- - **Integration tests** — deploy to ephemeral environment, verify behavior, destroy
- - **Rollback tests** — deploy N+1, roll back to N, verify correct state
- - **Destruction tests** — verify teardown is clean, no orphaned resources
- - **Chaos tests** — inject controlled failures to validate resilience and recovery procedures. Start small (kill a
- process), increase scope gradually (network partition, node failure). See
- [`${CLAUDE_SKILL_DIR}/references/disaster-recovery.md`]
+ - **The check names the observable effect and runs on the consumer's path**: the certificate on the socket, the
+ resolver's answer, the route in the peer's table, the listening port, the running image digest, the guest's own view,
+ a request through the new rule with the identity the real client uses. Provider create, controller `Synced`, Compose
+ `up`, and a play recap are control-plane evidence and close nothing.
+ - **A service loads its configuration when it is told to.** A unit file needs `daemon-reload`; a daemon needs its reload
+ after its config test passes; a `.env` change needs recreation; a ConfigMap key in an environment variable needs a pod
+ restart; a Proxmox setting that cannot apply live is pending until the next start; sshd on Ubuntu 22.10 and later
+ listens where `ssh.socket` says, not where `sshd_config` says.
+ - **A second run reporting no change proves the tool converged, not that the service works.** Do both.
+ - **Wait through the propagation window before closing**: kubelet sync, DNS TTL and negative TTL, rollout deadline,
+ health-check timeout, the delayed crash. GitHub's Redis load-balancer change on 2026-03-05 was accepted and routed 95
+ percent of workflow starts to the wrong hosts.
+ - **The agent's own report is not evidence.** Capture the command and its output in the deliverable. A Replit agent
+ reported rollback impossible when it was not; a Claude Code bug report in December 2025 consisted entirely of the
+ agent's apology for running a data-loss flag it had not been asked to run.
+ - **A change is closed when the monitor that would page on its regression is named and was observed after the change.**
+ A host CPU graph does not close a route change; a page must detect an otherwise undetected, urgent, user-visible
+ condition that a human can act on.
- ### IaC Quality Indicators
+ Read [`${CLAUDE_SKILL_DIR}/references/effect.md`] when closing a change on a systemd unit, a daemon with a config test,
+ a firewall, Compose, Kubernetes or a GitOps controller, a Proxmox guest, DNS, a certificate, WireGuard, or a
+ provisioning tool — it carries the read-only check per class and the propagation delays.
- Watch for these signals of degrading infrastructure code quality:
+ ## Reversibility
- - Security smells — hardcoded secrets, overly permissive IAM, insecure defaults
- - Rising change frequency without corresponding test coverage
- - Linter profile below "production" level (for Ansible: `ansible-lint` profiles progress through min → basic → moderate
- → safety → shared → production)
- - Non-idempotent applies — second run shows diffs
+ - **A revert reverses declarations.** It does not reverse a schema migration, a rotated secret, an issued certificate, a
+ cached DNS answer, a destroyed object, a shrunk volume, a pool feature upgrade, a `pg_upgrade --link`, a CRD storage
+ version, a control-plane minor, or firmware. Each of those needs its own captured reverse before apply.
+ - **Expand, then contract.** Ship the additive step, the code that reads both shapes, and the cutover as separate
+ changes; the destructive contraction is last and is not reverted by a code revert.
+ - **A rollback that was never run against production-shaped objects is a description.** GitHub's 2024-03-11 rollback
+ failed in one datacenter on records missing a field the forward path never needed. Exercise the reverse, record the
+ date and the duration, and record that the old observable behavior returned.
+ - **Configuration management has no state to revert to.** Reverting a playbook re-runs forward; the reverse of a change
+ is its own converging task. Provisioning state carries the previous shape and never the previous data.
- ## Incident Response
+ Read [`${CLAUDE_SKILL_DIR}/references/reversibility.md`] when a change touches data, secrets, certificates, DNS,
+ storage, identifiers, addressing, or a version with an on-disk or API format — it carries whether the reverse exists,
+ what it costs, and what to capture per class.
- **Every alert with a rote, algorithmic response must be automated.** If the response to a page is always the same steps,
- that response is a candidate for automated remediation.
+ ## Secrets and credentials
- **Blameless postmortems:** Focus on system flaws, not human error. "What went wrong" not "who caused it." Document:
- timeline, root cause, contributing factors, action items to prevent recurrence. Every postmortem produces concrete
- action items — not just "be more careful."
+ - **State and plan files hold every secret the configuration defined, in plaintext.** `sensitive` redacts output only.
+ They never enter Git, a CI artifact, or a PR comment unredacted.
+ - **A secret on a command line is readable by every user on the host.** Pass it by stdin, environment, or file.
+ - **Pin every action to a full commit SHA and every image to a digest.** `tj-actions/changed-files` (2025-03-14) and the
+ Trivy actions (2026-03-19) had their existing tags repointed at commits that dumped runner memory into public logs;
+ every consumer pinned by tag ran the payload. The scanner is in the supply chain too.
+ - **Rotate a compromised credential in one pass.** Trivy's second compromise came through a rotation spread over days,
+ during which a still-valid token exfiltrated the fresh ones.
+ - **CI reaches a cloud through OIDC with a constrained subject and role**, never a stored long-lived key.
+ - **Public certificate lifetime is 200 days from 2026-03-15, 100 from 2027-03-15, 47 from 2029-03-15.** Let's Encrypt
+ sends no expiry mail and answers no OCSP; the operator's monitor watches the served certificate.
- **Toil management:** Toil is operational work that is manual, repetitive, automatable, tactical, without enduring value,
- and grows linearly with service size. Target: less than 50% of time on toil. If a team exceeds this, step back and
- automate. Use SLOs to make data-driven decisions — ignore certain operational tasks if doing so does not consume the
- error budget.
+ Read [`${CLAUDE_SKILL_DIR}/references/secrets.md`] when a change handles a secret, a token, a CI identity, or a
+ certificate — it carries the leak surfaces, the credential-scope incidents, the Let's Encrypt profiles and limits, and
+ the certificate schedule.
- **Self-service reduces toil:** Move 80-90% of routine requests to self-service via web forms, scripts, APIs, or pull
- requests to configuration files. Reserve human handling for edge cases while automating the common path.
+ ## Verification
- See [`${CLAUDE_SKILL_DIR}/references/testing.md`] for runbook automation maturity levels and incident response patterns.
+ - **A scanner proves a property of an artifact.** Trivy, checkov, tflint, KICS, conftest, and Kyverno CLI evaluate files
+ and plans; goss and testinfra assert host state; cosign proves who signed; SLSA v1.2 levels describe build and source
+ integrity. None of them proves that a service answers. Their clean run is a precondition of the effect check, not a
+ substitute for it.
+ - **`plan -detailed-exitcode` is the drift gate** (0 clean, 1 error, 2 drift); a plain plan exits 0 either way.
+ - **Do not add a dead tool.** terrascan is archived (2025-11-20), tfsec is superseded by Trivy, driftctl is in
+ maintenance with no successor.
- ## Documentation Standards
+ Read [`${CLAUDE_SKILL_DIR}/references/verification.md`] when choosing, running, or trusting a scanner, a policy engine,
+ a host assertion, a signing tool, or a native test framework — it carries what each proves, what it cannot see, its
+ status, and the Terraform and OpenTofu split.
- Infrastructure documentation is part of the deliverable. Undocumented infrastructure is a liability — it becomes opaque
- to everyone except the person who built it, and eventually to them too.
+ ## Recovery
- **Every infrastructure component must have:**
+ - **A backup is proven by a timed restore whose result was opened.** The deliverable for a stateful change records the
+ backup identifier, the restore target, the elapsed time, and the application-level check that passed.
+ - **A filesystem copy of a running database is not a backup.** The engine's own tool, or an atomic snapshot of the whole
+ data directory, is. A VM snapshot is crash-consistent; the guest agent freezes the filesystem, not the application.
+ - **The keys to the backup and the credential that can delete it are part of the backup.** Store them under a different
+ failure domain and a different credential than the data; keep one copy immutable.
+ - **Parity is not a backup**, and a reconciliation controller does not restore data.
- - Topology diagram — what exists, how it connects, what depends on what
- - Access information — how to reach it, who has access, what credentials are needed (not the credentials themselves)
- - Dependencies — upstream, downstream, external services
- - Runbooks — step-by-step operational procedures with copy-pasteable commands
- - Monitoring — what's watched, where dashboards are, what alerts exist and what they mean
- - Recovery procedures — per-component DR procedures with RTO/RPO targets
+ Read [`${CLAUDE_SKILL_DIR}/references/recovery.md`] before a destructive change and when designing or verifying a backup
+ — it carries the consistency rules per engine, what restic, Borg, vzdump, and proxmox-backup-client capture and skip,
+ state and key custody, and recovery order. Read [`${CLAUDE_SKILL_DIR}/references/gitops.md`] when Argo CD or Flux is in
+ the path — it carries what each controller reconciles, ignores, prunes, and what the Argo CD 3.0 defaults changed.
- **Runbooks are specific commands, not prose.** Every step must be executable without interpretation:
+ ## Deliverable
- - Bad: "Restart the service and check that it's working"
- - Good: `systemctl restart caddy && curl -f http://localhost:80/health`
- - Bad: "Check the database is healthy"
- - Good: `psql -h localhost -U app -c "SELECT 1" && echo "DB OK" || echo "DB FAIL"`
+ A change is done when its record carries all of these, and states any it lacks as missing rather than omitting it:
- **Runbook automation maturity:** Start with manual runbooks. Once a procedure is validated through repeated execution,
- convert to semi-automated (scripts with human oversight at decision points), then fully automated (event-driven, no
- human intervention) for rote responses to known alerts. Every automated runbook must produce an audit trail and include
- a rollback path. See [`${CLAUDE_SKILL_DIR}/references/testing.md`] for automation patterns and success check
- requirements.
+ - the target identifiers and the credential scope used;
+ - the plan or diff, its generation time, and the objects it touches, with destroy and replace counts;
+ - the effect check, its command, and its output on the consumer's path;
+ - the second-run result;
+ - the reverse, exercised, with its duration, or the captured reverse for a class a revert cannot restore;
+ - for a stateful change, the backup and its restore time;
+ - the monitor that pages on regression and what it showed after the change.
## Application
- **When writing infrastructure:** Apply all principles silently. Use the deliverable checklist as a completion gate — do
- not declare work done until every item is checked. Flag security, observability, or recoverability gaps as blocking
- issues, not suggestions.
+ When **making** a change, apply these rules silently; do not narrate a rule while following it. Where the target's
+ existing practice contradicts one, follow the practice for the change at hand and flag the divergence once. Stop and
+ report rather than proceed when a destructive path is guarded by prose alone, when the plan shows an unexplained
+ destroy, or when the state does not match the target.
- **When reviewing infrastructure:** Cite the specific principle violated. Show the fix inline — not just what's wrong,
- but what correct looks like. Prioritize: security gaps first, then observability gaps, then drift/parity issues, then
- documentation gaps.
+ When **reviewing** a change, cite the missing evidence and name the check that would supply it. Do not lecture.
## Integration
- This skill defines the discipline that all infrastructure tool skills assume.
-
- **Workflow:**
-
- 1. **DevOps skill** — establish principles, quality standards, security posture
- 2. **Tool skill** — apply tool-specific conventions and patterns:
- - `ansible` — playbook design, role structure, inventory, vault
- - `containers` — Docker/Podman, Compose, image optimization
- - `proxmox` — VM/LXC provisioning, storage, networking, clustering
- - `unraid` — arrays, Docker, VMs, shares, plugins
- - `networking` — VLANs, firewalls, DNS, reverse proxies, VPN, TLS
- 3. **DevOps skill** — verify deliverable checklist, documentation, security review
-
- The tool skills handle _how_ to implement. This skill handles _what good looks like_.
-
- IMPORTANT: Before declaring any infrastructure task complete, return to the deliverable checklist above and verify every
- item. An unchecked item is an unfinished deliverable.
+ This skill governs what a change must carry, how it is applied, and the evidence that closes it. The **ansible**,
+ **containers**, **proxmox**, **unraid**, and **networking** skills govern how their artifact is written and what it
+ does, and each wins on that question. The **coding** skill governs the shape of the work. All are active at once; this
+ skill runs once, across the whole change, not before and after the tool skill.