git:20261003.496420b to git:20261105.65c9c02

- Design, diagnose, and safely operate brokered delivery with Kafka, RabbitMQ, and compatible queues: acknowledgements, retries, dead letters, ordering, lag, and replay.

- compatibility: "Optional, broker-dependent: kafka-consumer-groups.sh (Kafka) or rabbitmqctl (RabbitMQ) for live diagnosis and recovery"

- effort: high

- argument_hint: "[broker-or-symptom]"

- # Message Queues

-

- Design and diagnose asynchronous delivery without inventing guarantees the broker or application

- does not provide. Make the delivery contract explicit: who owns a message, when an effect is safe

- to repeat, how poison messages stop blocking progress, and how recovery avoids replaying more than

- intended.

- **Target versions** (October 2026):

- - Apache Kafka **4.3.1** (2026-06-25; 4.x is KRaft-only, ZooKeeper removed since 4.0.0 released 2025-03-18)

- - RabbitMQ **4.3.6** current, **4.2.10** previous lane (4.2 community support ended 2026-07-31; https://www.rabbitmq.com/release-information)

- - Designing Kafka, RabbitMQ, or compatible producer and consumer delivery behavior

- - Investigating consumer lag, growing queues, duplicate effects, redeliveries, or stuck messages

- - Choosing acknowledgement, retry, dead-letter, ordering, backpressure, retention, or replay rules

- - Reviewing an event-driven workflow for idempotency, observability, and failure recovery

- - The failing layer is unknown during a live outage; use **debug-triage** to localize it first

- - Building generic telemetry pipelines, alerts, SLOs, or dashboards; use **observability**

- - Modeling a database transaction, schema, outbox table, or data-store recovery; use **databases**

- - Designing HTTP, GraphQL, or gRPC request contracts; use **backend-api**

- - Editing Kubernetes, Terraform, or broker deployment configuration; use the matching infrastructure skill

- - CPU, heap, allocation, or lock profiling of a producer or consumer process; use **performance-debugging**

- ### 1. State the contract before changing settings

-

- Record producer success, broker durability, consumer acknowledgement point, retry limit, ordering

- scope, retention, and the permitted duplicate/loss window. "Exactly once" only applies to a stated

- broker-to-consumer boundary; an external side effect still needs an idempotency key or a transactional

- outbox/inbox protocol.

-

- ### 2. Trace one message through normal and failure paths

-

- For a representative message, trace publish -> broker acceptance -> delivery -> durable business

- effect -> acknowledgement/offset commit. Then trace process crash after the effect, timeout, malformed

- payload, dependency outage, and retry exhaustion. Place the acknowledgement only after the durable

- effect (auto-ack or early commit trades reliability for loss), and make duplicate delivery harmless.

-

- ### 3. Bound pressure and failure handling

-

- - Limit in-flight work to consumer capacity. RabbitMQ prefetch bounds unacknowledged deliveries;

- Kafka consumers need bounded poll/worker handoff and processing that stays within the group timeout.

- - Retry only transient, classified failures with capped attempts and delayed or scheduled backoff.

- Do not immediate-requeue every failure: it can create a hot redelivery loop.

- - Send exhausted, malformed, or permanently rejected messages to a dead-letter destination with the

- original payload, stable message ID, failure reason, attempt count, and source location. A DLQ is

- a recovery queue, not a discard bin: assign ownership, alerting, and a repair/replay procedure.

- - Preserve order only where the broker can: Kafka orders within a partition and a RabbitMQ queue has

- delivery order constraints that competing consumers, priorities, requeues, and retries can disturb.

- Choose a key/partition or a single ordered consumer deliberately; never claim global order by default.

-

- ### 4. Diagnose from read-only evidence

-

- Compare ingress rate, egress/ack rate, consumer count, in-flight messages, retry/redelivery rate,

- oldest-message age, DLQ rate, and Kafka consumer-group lag. A rising backlog with steady consumer

- throughput suggests insufficient capacity or increased input; high in-flight work plus slow acks

- suggests downstream saturation; repeating message IDs suggest a retry or idempotency defect.

- Inspect one correlation/message ID across producer, broker, consumer, and effect records before

- changing prefetch, scaling consumers, resetting offsets, purging queues, or replaying data. Read-only

- starting points (the CLIs ship with the broker; detect with `command -v kafka-consumer-groups.sh rabbitmqctl`):

- `kafka-consumer-groups.sh --bootstrap-server <host:port> --describe --group <group>` and

- `rabbitmqctl list_queues name messages_ready messages_unacknowledged consumers`.

-

- ### 5. Recover with a scoped, measurable plan

-

- Offset reset, purge, requeue, and broad replay change delivery state. Copy and track:

-

- - [ ] Scoped: finite message range, target group/queue, idempotency protection, side effects, stop condition

- - [ ] Baselined: lag or backlog recorded; for Kafka, current offsets captured as the rollback record

- - [ ] Rehearsed: replay on a non-production copy or a quarantined message; Kafka reset run with `--dry-run`

- - [ ] Approved: exact command, scope, and rollback or containment plan explicitly authorized

- - [ ] Executed unchanged: Kafka swaps only `--dry-run` for `--execute`; the group must have no active members

- - [ ] Verified: lag/backlog re-measured; if it diverges from the stop condition, halt and return to Scoped

-

- ```bash

- kafka-consumer-groups.sh --bootstrap-server <host:port> --describe --group <group> > offsets-before.txt

- kafka-consumer-groups.sh --bootstrap-server <host:port> --group <group> \

- --topic <topic>:<partitions> --reset-offsets --to-offset <offset> --dry-run

- ```

-

- ## AI Self-Check

-

- - [ ] Delivery guarantee names match the chosen producer, broker, consumer, and side-effect boundary

- - [ ] Acknowledgement/offset commit follows the durable effect and a crash can safely redeliver

- - [ ] Retryable and terminal errors are classified; attempts, backoff, and DLQ ownership are explicit

- - [ ] Message identity and idempotency cover producer retry, redelivery, and replay

- - [ ] Ordering claim is limited to the actual partition, queue, key, and consumer topology

- - [ ] Backpressure limits in-flight work rather than hiding saturation with unbounded buffers

- - [ ] Lag/backlog diagnosis includes ingress, egress, age, retries, and downstream latency

- - [ ] Any destructive recovery action has an exact scope, expected effects, measurement, and approval

- - [ ] Cross-cutting agent hygiene applied - see `references/agent-hygiene.md`

-

- ## Output Contract

-

- See `references/output-contract.md` for the full contract.

-

- - **Skill name:** MESSAGE-QUEUES

- - **Deliverable bucket:** `audits`

- - **Mode:** conditional. When invoked to audit or review queue-related repository content, emit the

- full contract and write the deliverable to `docs/local/audits/message-queues/<YYYY-MM-DD>-<slug>.md`.

- Design, diagnosis, and recovery guidance remains conversational.

- - **Severity scale:** `P0 | P1 | P2 | P3 | info`

-

- ## Sources

- - [Apache Kafka documentation](https://kafka.apache.org/documentation/) - consumer groups, ordering,

- offsets, idempotent producers, and transactions

- - [RabbitMQ acknowledgements and confirms](https://www.rabbitmq.com/docs/confirms) - manual acks,

- redelivery, publisher confirms, prefetch, and requeue behavior

- - [RabbitMQ quorum queues](https://www.rabbitmq.com/docs/quorum-queues) - dead-lettering guarantees

- 1. **Assume at-least-once delivery unless the complete boundary proves otherwise.** Consumers must

- tolerate duplicates; producer retries need stable message IDs.

- 2. **Make retries bounded and observable.** Include jitter/delay where supported; route poison

- messages away from the hot path.

- 3. **Measure age as well as count.** A small backlog containing old messages can be more urgent than

- a large fresh burst.

- 4. **Do not alter live broker state during diagnosis.** Read metrics, logs, and configuration first;

- request approval for offset resets, purges, requeues, topology edits, or production replay.