senior-devops · diff
v2.1.0 to v2.2.0
31 added, 264 removed. Audit A to A.
---
name: senior-devops
description: >
DevOps for CI/CD, containers, Kubernetes, and Terraform. Use when building pipelines,
containerizing apps, managing clusters, provisioning cloud infra, deploying with blue-
green/canary, or handling infrastructure incidents.
license: MIT + Commons Clause
metadata:
- version: 2.1.0
+ version: 2.2.0
author: borghei
category: engineering
domain: infrastructure
- updated: 2026-04-02
+ updated: 2026-06-17
tags: [docker, kubernetes, terraform, ci-cd, monitoring]
python-tools: pipeline_generator.py, terraform_scaffolder.py, deployment_manager.py
tech-stack: python, docker, kubernetes, terraform, prometheus
---
# Senior DevOps Engineer
The agent generates CI/CD pipelines, scaffolds Terraform infrastructure, and manages deployments with strategy selection, health checks, and rollback support.
- ---
-
- ## Quick Start
-
- ```bash
- # Generate CI/CD pipeline from project analysis
- python scripts/pipeline_generator.py <project-path> --platform github-actions --verbose
-
- # Scaffold Terraform infrastructure
- python scripts/terraform_scaffolder.py <target-path> --provider aws --env production --verbose
-
- # Manage deployment with canary strategy
- python scripts/deployment_manager.py <target-path> --strategy canary --verbose
- ```
-
- ## Tools Overview
-
- | Tool | Input | Output |
- |------|-------|--------|
- | `pipeline_generator.py` | Project path | CI/CD pipeline config (GitHub Actions, GitLab CI, Jenkins, CircleCI) |
- | `terraform_scaffolder.py` | Target path + provider | Terraform module structure with state config |
- | `deployment_manager.py` | Target path + strategy | Deployment plan with health checks and rollback |
-
- All tools support `--json` for machine-readable output and `--output` / `-o` for file writing.
-
- ---
-
- ## Workflow 1: Containerize and Deploy
-
- **Step 1 -- Build a production Dockerfile.**
-
- The agent generates multi-stage Dockerfiles following this pattern:
-
- ```dockerfile
- # Stage 1: Build
- FROM node:20-alpine AS builder
- WORKDIR /app
- COPY package.json package-lock.json ./
- RUN npm ci --only=production && npm cache clean --force
- COPY . .
- RUN npm run build
-
- # Stage 2: Production
- FROM node:20-alpine AS production
- WORKDIR /app
- RUN addgroup -g 1001 appgroup && \
- adduser -u 1001 -G appgroup -s /bin/sh -D appuser
- COPY --from=builder --chown=appuser:appgroup /app/dist ./dist
- COPY --from=builder --chown=appuser:appgroup /app/node_modules ./node_modules
- COPY --from=builder --chown=appuser:appgroup /app/package.json ./
- USER appuser
- EXPOSE 3000
- HEALTHCHECK --interval=30s --timeout=3s --start-period=10s --retries=3 \
- CMD wget --no-verbose --tries=1 --spider http://localhost:3000/healthz || exit 1
- CMD ["node", "dist/server.js"]
- ```
-
- **Validation checkpoint:** Image builds with `docker build -t app:test .` and `docker run --rm app:test` returns healthy.
-
- **Step 2 -- Deploy to Kubernetes.**
-
- The agent creates a Deployment with probes, resource limits, and security context:
-
- ```yaml
- spec:
- containers:
- - name: app
- image: myapp:1.2.3
- resources:
- requests: { cpu: 250m, memory: 256Mi }
- limits: { cpu: "1", memory: 512Mi }
- livenessProbe:
- httpGet: { path: /healthz, port: 3000 }
- initialDelaySeconds: 15
- periodSeconds: 20
- readinessProbe:
- httpGet: { path: /ready, port: 3000 }
- initialDelaySeconds: 5
- periodSeconds: 10
- startupProbe:
- httpGet: { path: /healthz, port: 3000 }
- failureThreshold: 30
- periodSeconds: 10
- ```
-
- **Probe decision:**
- - **startupProbe**: Slow-starting apps (JVM, model loading). Prevents liveness from killing during startup.
- - **livenessProbe**: Detects deadlocks. Keep simple -- do not check downstream dependencies.
- - **readinessProbe**: Controls traffic routing. Include dependency checks here.
-
- **Validation checkpoint:** `kubectl get pods -l app=myapp` shows all pods Running and Ready.
-
- ---
-
- ## Workflow 2: Infrastructure as Code with Terraform
-
- **Step 1 -- Scaffold the module structure.**
-
- ```bash
- python scripts/terraform_scaffolder.py ./infrastructure --provider aws --env production --verbose
- ```
-
- The agent produces:
- ```
- infrastructure/
- modules/
- vpc/ # main.tf, variables.tf, outputs.tf
- eks/
- rds/
- environments/
- staging/ # main.tf, terraform.tfvars, backend.tf
- production/
- ```
-
- **Step 2 -- Configure remote state.**
-
- ```hcl
- terraform {
- backend "s3" {
- bucket = "mycompany-terraform-state"
- key = "production/infrastructure.tfstate"
- region = "us-east-1"
- dynamodb_table = "terraform-locks"
- encrypt = true
- }
- }
- ```
-
- **Step 3 -- Run drift detection in CI.**
-
- ```bash
- terraform plan -detailed-exitcode -out=plan.tfplan
- # Exit 0 = clean, Exit 1 = error, Exit 2 = drift detected
- ```
-
- **Validation checkpoint:** `terraform plan` shows no unexpected changes. Drift alerts fire within 24 hours.
-
- **Key rules:**
- - One state file per environment per component (blast radius control)
- - Never store state locally or in git
- - Run `terraform plan` in CI, `terraform apply` only after approval
- - Use directories for environment separation, modules for shared logic
-
- ---
-
- ## Workflow 3: CI/CD Pipeline Design
-
- ```bash
- python scripts/pipeline_generator.py /path/to/project --platform github-actions --json
- ```
-
- The agent generates pipelines following these principles:
-
- 1. **Fail fast** -- lint and unit tests before expensive integration tests
- 2. **Cache aggressively** -- node_modules, Docker layers, pip packages
- 3. **Immutable artifacts** -- build once, deploy the same artifact everywhere
- 4. **Gate promotions** -- manual approval or smoke tests before production
- 5. **Parallel execution** -- independent test suites and security scans run concurrently
-
- **Example: GitHub Actions with matrix testing and deployment gates**
-
- ```yaml
- jobs:
- test:
- strategy:
- matrix:
- node-version: [18, 20]
- steps:
- - uses: actions/checkout@v4
- - uses: actions/setup-node@v4
- with: { node-version: "${{ matrix.node-version }}", cache: npm }
- - run: npm ci && npm run lint && npm test -- --coverage
-
- build:
- needs: [test, security]
- if: github.ref == 'refs/heads/main'
- steps:
- - uses: docker/build-push-action@v5
- with:
- push: true
- tags: "${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.sha }}"
- cache-from: type=gha
- cache-to: type=gha,mode=max
-
- deploy-staging:
- needs: build
- environment: staging
- steps:
- - run: helm upgrade --install app charts/myapp --set image.tag=${{ github.sha }} --wait
-
- deploy-production:
- needs: deploy-staging
- environment: production # requires manual approval
- ```
-
- **Validation checkpoint:** Pipeline runs in under 15 minutes. All stages produce exit code 0.
-
- ---
-
- ## Deployment Strategy Selection
-
- | Strategy | Risk | Rollback Speed | Infra Cost | Best For |
- |----------|------|----------------|------------|----------|
- | **Rolling** | Medium | Minutes | 1x | Stateless services, internal APIs |
- | **Blue-Green** | Low | Seconds | 2x | Mission-critical, zero-downtime |
- | **Canary** | Low | Seconds | 1.1x | User-facing, gradual validation |
- | **Feature Flags** | Lowest | Instant | 1x | Granular control, A/B testing |
-
- **Canary promotion ladder:**
- 1. Deploy at 5% traffic. Monitor error rate and latency for 10 min.
- 2. Promote to 25%. Monitor 10 min.
- 3. Promote to 50%. Monitor 15 min.
- 4. Promote to 100%.
- 5. Automated rollback if error rate exceeds baseline by 2x at any step.
-
- ---
-
- ## Monitoring Essentials
-
- Every service dashboard includes the **Four Golden Signals**:
-
- 1. **Latency** -- P50, P90, P99 response times
- 2. **Traffic** -- Requests per second by endpoint and status code
- 3. **Errors** -- 5xx rate, 4xx rate, application error codes
- 4. **Saturation** -- CPU, memory, connection pool, queue depth
-
- **SLO targets (example):**
-
- | Service | SLI | SLO | Error Budget |
- |---------|-----|-----|--------------|
- | API Gateway | Successful requests / Total | 99.9% (43.8 min/month downtime) | 0.1% |
- | API Latency | Requests < 500ms / Total | P99 < 500ms | 1% |
-
- When the error budget is exhausted, the agent recommends freezing feature deployments until the budget recovers.
-
- ---
+ ## Core Capabilities
- ## Anti-Patterns
+ - **CI/CD pipeline generation** — fail-fast, cached, immutable-artifact pipelines for GitHub Actions, GitLab CI, Jenkins, and CircleCI with matrix testing and promotion gates.
+ - **Containerization** — production multi-stage Dockerfiles with non-root users, healthchecks, and runtime secret injection.
+ - **Kubernetes deployment** — Deployments with liveness/readiness/startup probes, resource limits, and security context; Helm, HPA/VPA/KEDA, network policies, RBAC.
+ - **Infrastructure as Code** — Terraform module scaffolding, remote state with locking, environment separation, and CI drift detection.
+ - **Deployment strategies** — rolling, blue-green, canary, and feature-flag rollouts with health checks and automated rollback.
+ - **Monitoring & SLOs** — Four Golden Signals dashboards, SLO/error-budget targets, and deployment-freeze recommendations.
- 1. **Monolithic state** -- one Terraform state for everything. Split by component and environment.
- 2. **`latest` tag in production** -- always use specific image tags.
- 3. **Secrets in image layers** -- inject at runtime via environment or mounted secrets. Verify with `docker history --no-trunc`.
- 4. **No resource limits** -- every container needs CPU/memory limits to prevent noisy-neighbor attacks.
- 5. **Manual deployments** -- automate with approval gates instead.
+ ## When to Use
- ---
+ - Building or optimizing a CI/CD pipeline.
+ - Containerizing an app or deploying to Kubernetes.
+ - Provisioning cloud infrastructure with Terraform.
+ - Choosing and executing a deployment strategy (blue-green/canary).
+ - Handling an infrastructure incident or rollback.
- ## Troubleshooting
+ ## Tools
- | Problem | Cause | Solution |
- |---------|-------|----------|
- | Terraform state lock stuck | Interrupted `terraform apply` left DynamoDB lock | `terraform force-unlock <LOCK_ID>` after confirming no apply running |
- | Pods in `CrashLoopBackOff` | Failing health checks or missing config/secrets | `kubectl logs <pod>`, verify ConfigMaps/Secrets, increase `startupProbe.failureThreshold` |
- | Docker builds slow (10+ min) | Layer cache invalidated by early COPY of changing files | Copy dependency manifests before source; use BuildKit cache mounts |
- | Helm upgrade fails "another operation in progress" | Previous release in pending/failed state | `helm history <release>`, then `helm rollback <release> <last-good>` |
- | Canary shows healthy but users report errors | Metrics aggregated across all pods mask canary errors | Use per-revision metric labels; configure Istio/Nginx to tag canary traffic |
+ | Tool | Purpose | Command |
+ |------|---------|---------|
+ | `pipeline_generator.py` | Analyze a project and generate a CI/CD pipeline config (GitHub Actions, GitLab CI, Jenkins, CircleCI) | `python scripts/pipeline_generator.py <project-path> --json` |
+ | `terraform_scaffolder.py` | Scaffold a Terraform module structure with state config | `python scripts/terraform_scaffolder.py <target-path> --json` |
+ | `deployment_manager.py` | Produce a deployment plan with health checks and rollback | `python scripts/deployment_manager.py <target-path> --json` |
- ---
+ All tools support `--verbose`/`-v`, `--json` for machine-readable output, and `--output`/`-o` for file writing.
## References
- | Guide | Path | Content |
- |-------|------|---------|
- | CI/CD Pipeline Guide | `references/cicd_pipeline_guide.md` | Pipeline patterns, platform comparisons, optimization |
- | Infrastructure as Code | `references/infrastructure_as_code.md` | Terraform patterns, module design, state management |
- | Deployment Strategies | `references/deployment_strategies.md` | Strategy details, rollback procedures, traffic management |
-
- See also: `references/kubernetes_patterns.md` for Helm charts, HPA/VPA/KEDA decisions, network policies, and RBAC patterns. `references/cloud_platform_guide.md` for AWS/GCP/Azure service comparison, multi-cloud strategy, and cost optimization.
+ Load the reference that matches the task — keep this file lean and pull detail on demand:
- ---
+ - **[references/devops-workflows-and-operations.md](references/devops-workflows-and-operations.md)** — the three end-to-end workflows (containerize & deploy, Terraform IaC, CI/CD design) with worked Dockerfile/YAML/HCL examples, the deployment-strategy selection matrix and canary ladder, monitoring essentials (Four Golden Signals, SLO targets), anti-patterns, and the troubleshooting table. Read when executing any workflow or diagnosing an incident.
+ - **[references/cicd_pipeline_guide.md](references/cicd_pipeline_guide.md)** — pipeline patterns, platform comparisons, optimization.
+ - **[references/infrastructure_as_code.md](references/infrastructure_as_code.md)** — Terraform patterns, module design, state management.
+ - **[references/deployment_strategies.md](references/deployment_strategies.md)** — strategy details, rollback procedures, traffic management.
+ - **[references/kubernetes_patterns.md](references/kubernetes_patterns.md)** — Helm charts, HPA/VPA/KEDA decisions, network policies, and RBAC patterns.
+ - **[references/cloud_platform_guide.md](references/cloud_platform_guide.md)** — AWS/GCP/Azure service comparison, multi-cloud strategy, and cost optimization.
## Integration Points
| Skill | Integration |
|-------|-------------|
| `senior-secops` | Security scanning in CI/CD, container image scanning, compliance checks |
| `senior-architect` | Infrastructure design decisions, service topology |
| `senior-backend` | Application containerization, health endpoints, config management |
| `code-reviewer` | Terraform plan review, pipeline config review |
| `incident-commander` | Incident escalation, postmortem, rollback procedures |
---
- **Last Updated:** April 2026
- **Version:** 2.1.0
+ **Last Updated:** June 2026
+ **Version:** 2.2.0