Poison messages

Poison messages fail on every delivery attempt because of corrupt payloads, schema mismatch, missing referenced data, or logic bugs — not transient infrastructure faults. Without detection they consume CPU, block partitions, and trigger retry storms that amplify outages.

Detect poison via attempt counters, error classification, and circuit breaking on repeated identical failures. Move them quickly to DLQ after a small fixed attempt budget. Fix forward with schema versioning and validation at ingress before enqueue.

On interviews: distinguish transient versus poison failures, explain how one bad message can stall a queue, and describe alerting you would add.

Common pitfalls: retrying validation errors indefinitely; no schema compatibility checks between producer and consumer versions; shared retry policy for all error types; operators unaware because errors look like normal retries.

The trade-off is flexibility versus complexity—know when the simpler path is enough.

Checklist:

• Classify errors as transient versus permanent.
• Cap retries and route poison to DLQ fast.
• Validate payloads before enqueue.
• Alert on repeated identical failure signatures.