advanced
Backpressure
Control producer rate, queue depth, consumer concurrency, prefetch, lag, and load shedding so messaging buffers do not hide overload.
Backpressure signals that downstream cannot keep up — rising queue depth, consumer lag, or broker disk use. Responses include slowing producers, limiting prefetch, scaling consumers, shedding load, or rejecting publishes when buffers exceed thresholds.
Trade-off: buffering smooths spikes but hides overload until catastrophic failure; early shedding protects the system but drops work.
On interviews: diagnose lag growing linearly; choose between scale consumers and throttle producers.
Common pitfalls: infinite buffer pretending capacity is unlimited; scaling consumers past partition count; no alerts on lag SLO.
Checklist:
- Define lag and depth SLOs with alerts.
- Cap prefetch and in-flight messages per consumer.
- Implement producer throttling or circuit breaking.
- Load test with sustained overload scenarios.