advanced
Retry queues
Use bounded retry queues or delayed retries with backoff, attempt metadata, observability, and a final failure path.
Retry queues delay failed messages before re-attempting processing. Implement delays with TTL plus DLX hop, dedicated delay queues, or broker plugins — avoid tight spin loops that hammer downstream systems. Store attempt count, first-failure time, and last error in headers or a side table.
Cap maximum attempts and route exhausted messages to a dead-letter queue for human review. Exponential backoff with jitter spreads retry storms after outages. Make retry policy explicit per message type: payment retries differ from email notification retries.
On interviews: sketch a three-queue retry ladder, explain why jitter matters, and define when a failure should not retry.
Common pitfalls: unbounded retries on poison messages; retrying non-idempotent side effects without deduplication; fixed delay that synchronizes thundering herds; no visibility into retry depth.
The trade-off is flexibility versus complexity—know when the simpler path is enough.
Checklist:
- Bound attempt count per message type.
- Use backoff with jitter.
- Persist attempt metadata.
- Route exhausted messages to DLQ with context.