git:20260326.840d219 to git:20260522.be57c0b

2 added, 263 removed. Audit A to A.

---

name: on-call-handoff-patterns

description: Master on-call shift handoffs with context transfer, escalation procedures, and documentation. Use this skill when transitioning on-call responsibilities between engineers and ensuring the incoming responder has full situational awareness, when writing a shift summary that captures active incidents, ongoing investigations, and recent changes, when handing off mid-incident so a fresh engineer can take over the incident commander role without losing context, when onboarding a new engineer to the on-call rotation for the first time, or when auditing and improving the quality of existing handoff processes across teams.

---

# On-Call Handoff Patterns

Effective patterns for on-call shift transitions, ensuring continuity, context transfer, and reliable incident response across shifts.

## When to Use This Skill

- Transitioning on-call responsibilities

- Writing shift handoff summaries

- Documenting ongoing investigations

- Establishing on-call rotation procedures

- Improving handoff quality

- Onboarding new on-call engineers

## Core Concepts

### 1. Handoff Components

| Component | Purpose |

| -------------------------- | ----------------------- |

| **Active Incidents** | What's currently broken |

| **Ongoing Investigations** | Issues being debugged |

| **Recent Changes** | Deployments, configs |

| **Known Issues** | Workarounds in place |

| **Upcoming Events** | Maintenance, releases |

### 2. Handoff Timing

```

Recommended: 30 min overlap between shifts

Outgoing:

├── 15 min: Write handoff document

└── 15 min: Sync call with incoming

Incoming:

├── 15 min: Review handoff document

├── 15 min: Sync call with outgoing

└── 5 min: Verify alerting setup

```

- ## Templates

- ### Template 1: Shift Handoff Document

- ````markdown

- # On-Call Handoff: Platform Team

- **Outgoing**: @alice (2024-01-15 to 2024-01-22)

- **Incoming**: @bob (2024-01-22 to 2024-01-29)

- **Handoff Time**: 2024-01-22 09:00 UTC

- ---

- ## 🔴 Active Incidents

- ### None currently active

- No active incidents at handoff time.

- ---

- ## 🟡 Ongoing Investigations

- ### 1. Intermittent API Timeouts (ENG-1234)

- **Status**: Investigating

- **Started**: 2024-01-20

- **Impact**: ~0.1% of requests timing out

- **Context**:

- - Timeouts correlate with database backup window (02:00-03:00 UTC)

- - Suspect backup process causing lock contention

- - Added extra logging in PR #567 (deployed 01/21)

- **Next Steps**:

- - [ ] Review new logs after tonight's backup

- - [ ] Consider moving backup window if confirmed

- **Resources**:

- - Dashboard: [API Latency](https://grafana/d/api-latency)

- - Thread: #platform-eng (01/20, 14:32)

- ---

- ### 2. Memory Growth in Auth Service (ENG-1235)

- **Status**: Monitoring

- **Started**: 2024-01-18

- **Impact**: None yet (proactive)

- **Context**:

- - Memory usage growing ~5% per day

- - No memory leak found in profiling

- - Suspect connection pool not releasing properly

- **Next Steps**:

- - [ ] Review heap dump from 01/21

- - [ ] Consider restart if usage > 80%

- **Resources**:

- - Dashboard: [Auth Service Memory](https://grafana/d/auth-memory)

- - Analysis doc: [Memory Investigation](https://docs/eng-1235)

- ---

- ## 🟢 Resolved This Shift

- ### Payment Service Outage (2024-01-19)

- - **Duration**: 23 minutes

- - **Root Cause**: Database connection exhaustion

- - **Resolution**: Rolled back v2.3.4, increased pool size

- - **Postmortem**: [POSTMORTEM-89](https://docs/postmortem-89)

- - **Follow-up tickets**: ENG-1230, ENG-1231

- ---

- ## 📋 Recent Changes

- ### Deployments

- | ------------ | ------- | ----------- | -------------------------- |

- | api-gateway | v3.2.1 | 01/21 14:00 | Bug fix for header parsing |

- | user-service | v2.8.0 | 01/20 10:00 | New profile features |

- | auth-service | v4.1.2 | 01/19 16:00 | Security patch |

- ### Configuration Changes

- - 01/21: Increased API rate limit from 1000 to 1500 RPS

- - 01/20: Updated database connection pool max from 50 to 75

- ### Infrastructure

- - 01/20: Added 2 nodes to Kubernetes cluster

- - 01/19: Upgraded Redis from 6.2 to 7.0

- ---

- ## ⚠️ Known Issues & Workarounds

- ### 1. Slow Dashboard Loading

- **Issue**: Grafana dashboards slow on Monday mornings

- **Workaround**: Wait 5 min after 08:00 UTC for cache warm-up

- **Ticket**: OPS-456 (P3)

- ### 2. Flaky Integration Test

- **Issue**: `test_payment_flow` fails intermittently in CI

- **Workaround**: Re-run failed job (usually passes on retry)

- **Ticket**: ENG-1200 (P2)

- ---

- ## 📅 Upcoming Events

- | ----------- | -------------------- | ------------------- | ------------- |

- ---

- ## 📞 Escalation Reminders

- | Issue Type | First Escalation | Second Escalation |

- | --------------- | -------------------- | ----------------- |

- | Payment issues | @payments-oncall | @payments-manager |

- | Auth issues | @auth-oncall | @security-team |

- | Database issues | @dba-team | @infra-manager |

- | Unknown/severe | @engineering-manager | @vp-engineering |

- ---

- ## 🔧 Quick Reference

- ### Common Commands

- ```bash

- # Check service health

- kubectl get pods -A | grep -v Running

- # Recent deployments

- kubectl get events --sort-by='.lastTimestamp' | tail -20

- # Database connections

- psql -c "SELECT count(*) FROM pg_stat_activity;"

- # Clear cache (emergency only)

- redis-cli FLUSHDB

- ```

- ````

- ### Important Links

- - [Runbooks](https://wiki/runbooks)

- - [Service Catalog](https://wiki/services)

- - [Incident Slack](https://slack.com/incidents)

- - [PagerDuty](https://pagerduty.com/schedules)

- ---

- ## Handoff Checklist

- ### Outgoing Engineer

- - [x] Document active incidents

- - [x] Document ongoing investigations

- - [x] List recent changes

- - [x] Note known issues

- - [x] Add upcoming events

- - [x] Sync with incoming engineer

- ### Incoming Engineer

- - [ ] Read this document

- - [ ] Join sync call

- - [ ] Verify PagerDuty is routing to you

- - [ ] Verify Slack notifications working

- - [ ] Check VPN/access working

- - [ ] Review critical dashboards

- ````

- ### Template 2: Quick Handoff (Async)

- ```markdown

- # Quick Handoff: @alice → @bob

- ## TL;DR

- - No active incidents

- - 1 investigation ongoing (API timeouts, see ENG-1234)

- - Major release tomorrow (01/24) - be ready for issues

- ## Watch List

- 1. API latency around 02:00-03:00 UTC (backup window)

- 2. Auth service memory (restart if > 80%)

- ## Recent

- - Deployed api-gateway v3.2.1 yesterday (stable)

- - Increased rate limits to 1500 RPS

- ## Coming Up

- - 01/23 02:00 - DB maintenance (5 min read-only)

- - 01/24 14:00 - v5.0 release

- ## Questions?

- I'll be available on Slack until 17:00 today.

- ````

- ### Template 3: Incident Handoff (Mid-Incident)

- ```markdown

- # INCIDENT HANDOFF: Payment Service Degradation

- **Incident Start**: 2024-01-22 08:15 UTC

- **Current Status**: Mitigating

- **Severity**: SEV2

- ---

- ## Current State

- - Error rate: 15% (down from 40%)

- - Mitigation in progress: scaling up pods

- - ETA to resolution: ~30 min

- ## What We Know

- 1. Root cause: Memory pressure on payment-service pods

- 2. Triggered by: Unusual traffic spike (3x normal)

- 3. Contributing: Inefficient query in checkout flow

- ## What We've Done

- - Scaled payment-service from 5 → 15 pods

- - Enabled rate limiting on checkout endpoint

- - Disabled non-critical features

- ## What Needs to Happen

- 1. Monitor error rate - should reach <1% in ~15 min

- 2. If not improving, escalate to @payments-manager

- 3. Once stable, begin root cause investigation

- ## Key People

- - Incident Commander: @alice (handing off)

- - Comms Lead: @charlie

- - Technical Lead: @bob (incoming)

- ## Communication

+ ## Templates and detailed worked examples

- - Status page: Updated at 08:45

- - Customer support: Notified

- - Exec team: Aware

+ Full template library and detailed worked examples live in `references/details.md`. Read that file when you need the concrete templates.

## Troubleshooting

**Incoming engineer misses a critical issue because the handoff document was incomplete.**

Use the outgoing checklist as a gate: do not mark handoff complete until every section has at least one entry (or an explicit "none"). Make incomplete handoffs a blameless postmortem action item.

**A 30-minute sync call is not possible due to timezone gaps.**

Fall back to the async quick handoff template (Template 2). Supplement with a short Loom or voice memo walking through the watch list. Ensure the incoming engineer has a direct contact method if they have follow-up questions.

**The incoming engineer inherits a mid-incident and is immediately overwhelmed.**

Use the incident handoff template (Template 3) specifically. The outgoing engineer should remain available on Slack for 15 minutes after handoff, even if off-call, to answer clarifying questions.

**On-call handoff documents are inconsistently formatted across teams.**

Adopt the shift handoff template organization-wide and store completed handoffs in a shared location (wiki, Notion, Confluence). Link each handoff from the on-call schedule entry in PagerDuty.

**Incoming engineer cannot verify their alerting is working before the outgoing engineer logs off.**

Add a standard step: outgoing engineer fires a test alert and confirms incoming engineer receives it in PagerDuty and Slack before ending the overlap window.

## Related Skills

- [incident-classification](../../skills/incident-classification/SKILL.md) — Classify and prioritize incidents that need to be included in the handoff document

- [postmortem-facilitation](../../skills/postmortem-facilitation/SKILL.md) — Turn resolved incidents from the shift into structured postmortems