message-queues git:20260920.ebf9a3dA

Immutable. This exact content is served forever at /api/v1/blob/bfbce11c02a0b2e5.

---
name: message-queues
description: >
  Design, diagnose, and safely operate brokered delivery with Kafka, RabbitMQ, and compatible queues: acknowledgements, retries, dead letters, ordering, lag, and replay.
license: MIT
metadata:
  source: iuliandita/skills
  date_added: "2026-09-20"
  effort: high
  argument_hint: "[broker-or-symptom]"
---

# Message Queues

Design and diagnose asynchronous delivery without inventing guarantees the broker or application
does not provide. Make the delivery contract explicit: who owns a message, when an effect is safe
to repeat, how poison messages stop blocking progress, and how recovery avoids replaying more than
intended.

## When to use

- Designing Kafka, RabbitMQ, or compatible producer and consumer delivery behavior
- Investigating consumer lag, growing queues, duplicate effects, redeliveries, or stuck messages
- Choosing acknowledgement, retry, dead-letter, ordering, backpressure, retention, or replay rules
- Reviewing an event-driven workflow for idempotency, observability, and failure recovery

## When NOT to use

- The failing layer is unknown during a live outage; use **debug-triage** to localize it first
- Building generic telemetry pipelines, alerts, SLOs, or dashboards; use **observability**
- Modeling a database transaction, schema, outbox table, or data-store recovery; use **databases**
- Designing HTTP, GraphQL, or gRPC request contracts; use **backend-api**
- Editing Kubernetes, Terraform, or broker deployment configuration; use the matching infrastructure skill

## Workflow

### 1. State the contract before changing settings

Record producer success, broker durability, consumer acknowledgement point, retry limit, ordering
scope, retention, and the permitted duplicate/loss window. "Exactly once" only applies to a stated
broker-to-consumer boundary; an external side effect still needs an idempotency key or a transactional
outbox/inbox protocol.

### 2. Trace one message through normal and failure paths

For a representative message, trace publish -> broker acceptance -> delivery -> durable business
effect -> acknowledgement/offset commit. Then trace process crash after the effect, timeout, malformed
payload, dependency outage, and retry exhaustion. Place the acknowledgement only after the durable
effect, and make duplicate delivery harmless.

### 3. Bound pressure and failure handling

- Limit in-flight work to consumer capacity. RabbitMQ prefetch bounds unacknowledged deliveries;
  Kafka consumers need bounded poll/worker handoff and processing that stays within the group timeout.
- Retry only transient, classified failures with capped attempts and delayed or scheduled backoff.
  Do not immediate-requeue every failure: it can create a hot redelivery loop.
- Send exhausted, malformed, or permanently rejected messages to a dead-letter destination with the
  original payload, stable message ID, failure reason, attempt count, and source location. A DLQ is
  a recovery queue, not a discard bin: assign ownership, alerting, and a repair/replay procedure.
- Preserve order only where the broker can: Kafka orders within a partition and a RabbitMQ queue has
  delivery order constraints that competing consumers, priorities, requeues, and retries can disturb.
  Choose a key/partition or a single ordered consumer deliberately; never claim global order by default.

### 4. Diagnose from read-only evidence

Compare ingress rate, egress/ack rate, consumer count, in-flight messages, retry/redelivery rate,
oldest-message age, DLQ rate, and Kafka consumer-group lag. A rising backlog with steady consumer
throughput suggests insufficient capacity or increased input; high in-flight work plus slow acks
suggests downstream saturation; repeating message IDs suggest a retry or idempotency defect.
Inspect one correlation/message ID across producer, broker, consumer, and effect records before
changing prefetch, scaling consumers, resetting offsets, purging queues, or replaying data.

### 5. Recover with a scoped, measurable plan

Choose a finite message range, target consumer group/queue, idempotency protection, expected
side effects, stop condition, and before/after lag or backlog measurement. Test replay on a
non-production copy or a quarantined message first. Offset reset, purge, requeue, or broad replay
changes delivery state: obtain explicit authorization after presenting the exact scope and rollback
or containment plan.

## AI Self-Check

- [ ] Delivery guarantee names match the chosen producer, broker, consumer, and side-effect boundary
- [ ] Acknowledgement/offset commit follows the durable effect and a crash can safely redeliver
- [ ] Retryable and terminal errors are classified; attempts, backoff, and DLQ ownership are explicit
- [ ] Message identity and idempotency cover producer retry, redelivery, and replay
- [ ] Ordering claim is limited to the actual partition, queue, key, and consumer topology
- [ ] Backpressure limits in-flight work rather than hiding saturation with unbounded buffers
- [ ] Lag/backlog diagnosis includes ingress, egress, age, retries, and downstream latency
- [ ] Any destructive recovery action has an exact scope, expected effects, measurement, and approval

## Output Contract

See `references/output-contract.md` for the full contract.

- **Skill name:** MESSAGE-QUEUES
- **Deliverable bucket:** `audits`
- **Mode:** conditional. When invoked to audit or review queue-related repository content, emit the
  full contract and write the deliverable to `docs/local/audits/message-queues/<YYYY-MM-DD>-<slug>.md`.
  Design, diagnosis, and recovery guidance remains conversational.
- **Severity scale:** `P0 | P1 | P2 | P3 | info`

## Sources

- [Apache Kafka documentation](https://kafka.apache.org/documentation/) - consumer groups, ordering,
  offsets, idempotent producers, and transactions
- [RabbitMQ acknowledgements and confirms](https://www.rabbitmq.com/docs/confirms) - manual acks,
  redelivery, publisher confirms, prefetch, and requeue behavior
- [RabbitMQ quorum queues](https://www.rabbitmq.com/docs/quorum-queues) - dead-lettering guarantees

## Rules

1. **Assume at-least-once delivery unless the complete boundary proves otherwise.** Consumers must
   tolerate duplicates; producer retries need stable message IDs.
2. **Acknowledge after the durable effect.** Auto-ack or early commit trades reliability for loss.
3. **Make retries bounded and observable.** Include jitter/delay where supported; route poison
   messages away from the hot path.
4. **Treat replay as a new execution.** It can recreate effects, so use idempotency and scope it.
5. **Measure age as well as count.** A small backlog containing old messages can be more urgent than
   a large fresh burst.
6. **Do not alter live broker state during diagnosis.** Read metrics, logs, and configuration first;
   request approval for offset resets, purges, requeues, topology edits, or production replay.