advanced
Dead-letter queues
Send exhausted or invalid messages to a dead-letter queue with enough context for diagnosis, replay, quarantine, or manual repair.
Dead-letter queues (DLQ) hold messages that cannot be processed after retries or fail validation immediately. A useful DLQ stores the original payload, failure reason, stack trace or error code, consumer identity, and timestamps so operators can triage without reproducing production traffic.
Operational workflows include inspect, fix code or data, replay in controlled batches, and quarantine permanently bad messages. Access-control DLQs because they often contain sensitive business data. Alert on DLQ growth rate, not just absolute depth.
On interviews: describe DLQ triage runbook steps, how replay differs from automatic retry, and what metadata you require in every dead-lettered message.
Common pitfalls: DLQ as a graveyard with no owner; replaying all messages after deploy without sampling; missing correlation IDs in DLQ payloads; no retention policy so DLQ becomes unbounded storage.
The trade-off is flexibility versus complexity—know when the simpler path is enough.
Checklist:
- Capture failure reason and correlation IDs.
- Alert on DLQ ingress rate.
- Define replay and quarantine runbooks.
- Restrict DLQ access and set retention.