deployment-pipeline-design · diff
git:20260307.47a5dbc to git:20260326.5e3e83d
77 added, 1 removed. Audit A to A.
---
name: deployment-pipeline-design
- description: Design multi-stage CI/CD pipelines with approval gates, security checks, and deployment orchestration. Use when architecting deployment workflows, setting up continuous delivery, or implementing GitOps practices.
+ description: Design multi-stage CI/CD pipelines with approval gates, security checks, and deployment orchestration. Use this skill when building a zero-downtime deployment strategy for a high-traffic API; implementing progressive canary rollouts with automated metric-based promotion; setting up a multi-environment pipeline with mandatory security scanning between staging and production; or debugging a pipeline where deployments succeed but health checks consistently fail post-release.
---
# Deployment Pipeline Design
Architecture patterns for multi-stage CI/CD pipelines with approval gates and deployment strategies.
## Purpose
Design robust, secure deployment pipelines that balance speed with safety through proper stage organization and approval workflows.
## When to Use
- Design CI/CD architecture
- Implement deployment gates
- Configure multi-environment pipelines
- Establish deployment best practices
- Implement progressive delivery
## Pipeline Stages
### Standard Pipeline Flow
```
┌─────────┐ ┌──────┐ ┌─────────┐ ┌────────┐ ┌──────────┐
│ Build │ → │ Test │ → │ Staging │ → │ Approve│ → │Production│
└─────────┘ └──────┘ └─────────┘ └────────┘ └──────────┘
```
### Detailed Stage Breakdown
1. **Source** - Code checkout
2. **Build** - Compile, package, containerize
3. **Test** - Unit, integration, security scans
4. **Staging Deploy** - Deploy to staging environment
5. **Integration Tests** - E2E, smoke tests
6. **Approval Gate** - Manual approval required
7. **Production Deploy** - Canary, blue-green, rolling
8. **Verification** - Health checks, monitoring
9. **Rollback** - Automated rollback on failure
## Approval Gate Patterns
### Pattern 1: Manual Approval
```yaml
# GitHub Actions
production-deploy:
needs: staging-deploy
environment:
name: production
url: https://app.example.com
runs-on: ubuntu-latest
steps:
- name: Deploy to production
run: |
# Deployment commands
```
### Pattern 2: Time-Based Approval
```yaml
# GitLab CI
deploy:production:
stage: deploy
script:
- deploy.sh production
environment:
name: production
when: delayed
start_in: 30 minutes
only:
- main
```
### Pattern 3: Multi-Approver
```yaml
# Azure Pipelines
stages:
- stage: Production
dependsOn: Staging
jobs:
- deployment: Deploy
environment:
name: production
resourceType: Kubernetes
strategy:
runOnce:
preDeploy:
steps:
- task: ManualValidation@0
inputs:
notifyUsers: "team-leads@example.com"
instructions: "Review staging metrics before approving"
```
**Reference:** See `assets/approval-gate-template.yml`
## Deployment Strategies
### 1. Rolling Deployment
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-app
spec:
replicas: 10
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 2
maxUnavailable: 1
```
**Characteristics:**
- Gradual rollout
- Zero downtime
- Easy rollback
- Best for most applications
### 2. Blue-Green Deployment
```yaml
# Blue (current)
kubectl apply -f blue-deployment.yaml
kubectl label service my-app version=blue
# Green (new)
kubectl apply -f green-deployment.yaml
# Test green environment
kubectl label service my-app version=green
# Rollback if needed
kubectl label service my-app version=blue
```
**Characteristics:**
- Instant switchover
- Easy rollback
- Doubles infrastructure cost temporarily
- Good for high-risk deployments
### 3. Canary Deployment
```yaml
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
name: my-app
spec:
replicas: 10
strategy:
canary:
steps:
- setWeight: 10
- pause: { duration: 5m }
- setWeight: 25
- pause: { duration: 5m }
- setWeight: 50
- pause: { duration: 5m }
- setWeight: 100
```
**Characteristics:**
- Gradual traffic shift
- Risk mitigation
- Real user testing
- Requires service mesh or similar
### 4. Feature Flags
```python
from flagsmith import Flagsmith
flagsmith = Flagsmith(environment_key="API_KEY")
if flagsmith.has_feature("new_checkout_flow"):
# New code path
process_checkout_v2()
else:
# Existing code path
process_checkout_v1()
```
**Characteristics:**
- Deploy without releasing
- A/B testing
- Instant rollback
- Granular control
## Pipeline Orchestration
### Multi-Stage Pipeline Example
```yaml
name: Production Pipeline
on:
push:
branches: [main]
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Build application
run: make build
- name: Build Docker image
run: docker build -t myapp:${{ github.sha }} .
- name: Push to registry
run: docker push myapp:${{ github.sha }}
test:
needs: build
runs-on: ubuntu-latest
steps:
- name: Unit tests
run: make test
- name: Security scan
run: trivy image myapp:${{ github.sha }}
deploy-staging:
needs: test
runs-on: ubuntu-latest
environment:
name: staging
steps:
- name: Deploy to staging
run: kubectl apply -f k8s/staging/
integration-test:
needs: deploy-staging
runs-on: ubuntu-latest
steps:
- name: Run E2E tests
run: npm run test:e2e
deploy-production:
needs: integration-test
runs-on: ubuntu-latest
environment:
name: production
steps:
- name: Canary deployment
run: |
kubectl apply -f k8s/production/
kubectl argo rollouts promote my-app
verify:
needs: deploy-production
runs-on: ubuntu-latest
steps:
- name: Health check
run: curl -f https://app.example.com/health
- name: Notify team
run: |
curl -X POST ${{ secrets.SLACK_WEBHOOK }} \
-d '{"text":"Production deployment successful!"}'
```
## Pipeline Best Practices
1. **Fail fast** - Run quick tests first
2. **Parallel execution** - Run independent jobs concurrently
3. **Caching** - Cache dependencies between runs
4. **Artifact management** - Store build artifacts
5. **Environment parity** - Keep environments consistent
6. **Secrets management** - Use secret stores (Vault, etc.)
7. **Deployment windows** - Schedule deployments appropriately
8. **Monitoring integration** - Track deployment metrics
9. **Rollback automation** - Auto-rollback on failures
10. **Documentation** - Document pipeline stages
## Rollback Strategies
### Automated Rollback
```yaml
deploy-and-verify:
steps:
- name: Deploy new version
run: kubectl apply -f k8s/
- name: Wait for rollout
run: kubectl rollout status deployment/my-app
- name: Health check
id: health
run: |
for i in {1..10}; do
if curl -sf https://app.example.com/health; then
exit 0
fi
sleep 10
done
exit 1
- name: Rollback on failure
if: failure()
run: kubectl rollout undo deployment/my-app
```
### Manual Rollback
```bash
# List revision history
kubectl rollout history deployment/my-app
# Rollback to previous version
kubectl rollout undo deployment/my-app
# Rollback to specific revision
kubectl rollout undo deployment/my-app --to-revision=3
```
## Monitoring and Metrics
### Key Pipeline Metrics
- **Deployment Frequency** - How often deployments occur
- **Lead Time** - Time from commit to production
- **Change Failure Rate** - Percentage of failed deployments
- **Mean Time to Recovery (MTTR)** - Time to recover from failure
- **Pipeline Success Rate** - Percentage of successful runs
- **Average Pipeline Duration** - Time to complete pipeline
### Integration with Monitoring
```yaml
- name: Post-deployment verification
run: |
# Wait for metrics stabilization
sleep 60
# Check error rate
ERROR_RATE=$(curl -s "$PROMETHEUS_URL/api/v1/query?query=rate(http_errors_total[5m])" | jq '.data.result[0].value[1]')
if (( $(echo "$ERROR_RATE > 0.01" | bc -l) )); then
echo "Error rate too high: $ERROR_RATE"
exit 1
fi
```
+
+ ## Troubleshooting
+
+ ### Health check passes in pipeline but service is unhealthy in production
+
+ The pipeline health check is hitting a shallow `/ping` endpoint that returns 200 even when the database is unreachable. Use a deep health check that verifies actual dependencies:
+
+ ```yaml
+ - name: Deep health check
+ run: |
+ HEALTH=$(curl -sf https://app.example.com/health/ready | jq -r '.status')
+ if [ "$HEALTH" != "ok" ]; then
+ echo "Deep health check failed: $HEALTH"
+ exit 1
+ fi
+ ```
+
+ ```python
+ # /health/ready endpoint — checks real dependencies
+ @app.get("/health/ready")
+ async def readiness():
+ checks = {
+ "database": await check_db_connection(),
+ "cache": await check_redis_connection(),
+ "queue": await check_queue_connection(),
+ }
+ status = "ok" if all(checks.values()) else "degraded"
+ return {"status": status, "checks": checks}
+ ```
+
+ ### Canary deployment never promotes to 100%
+
+ Argo Rollouts requires a valid `AnalysisTemplate` to auto-promote. If the Prometheus query in the analysis template returns no data (e.g., metric name changed), the analysis stays inconclusive and promotion stalls. Add a `failureCondition` and `inconclusiveLimit` so the rollout fails fast rather than hanging:
+
+ ```yaml
+ spec:
+ metrics:
+ - name: error-rate
+ failureCondition: "result[0] > 0.05"
+ inconclusiveLimit: 2 # fail after 2 inconclusive results
+ provider:
+ prometheus:
+ query: |
+ sum(rate(http_requests_total{status=~"5.."}[2m]))
+ / sum(rate(http_requests_total[2m]))
+ ```
+
+ ### Staging deploy succeeds but production job never starts
+
+ Check that the production environment protection rules are configured correctly — a missing reviewer assignment means the approval gate waits indefinitely with no notification. In GitHub Actions, add `reviewers` to the environment settings and verify the user or team exists:
+
+ ```yaml
+ # In GitHub repo settings → Environments → production:
+ # Required reviewers: @your-org/release-managers
+ ```
+
+ ### Docker layer cache busted on every run causing slow builds
+
+ If `COPY . .` appears before dependency installation, any source file change invalidates the dependency layer. Reorder to copy dependency manifests first:
+
+ ```dockerfile
+ # Good: dependencies cached separately from source
+ COPY package*.json ./
+ RUN npm ci
+ COPY . .
+ RUN npm run build
+ ```
+
+ ### Rollback leaves database migrations applied to old code
+
+ A service rollback without a migration rollback causes schema/code mismatch errors. Always make migrations backward-compatible (additive only) for at least one release cycle, and keep rollback scripts versioned alongside the migration:
+
+ ```bash
+ # migrations/V20240315__add_nullable_column.sql (forward)
+ # migrations/V20240315__add_nullable_column.undo.sql (backward)
+ ```
## Related Skills
- `github-actions-templates` - For GitHub Actions implementation
- `gitlab-ci-patterns` - For GitLab CI implementation
- `secrets-management` - For secrets handling