Alert fatigue

Alert fatigue happens when responders receive too many noisy, duplicate, or unactionable pages. Trust erodes; real incidents get missed. Good alerting is a product: symptoms, owners, runbooks, and tuning loops.

| Anti-pattern | Fix | |--------------|-----| | Page on every error log | Aggregate; alert on SLO burn | | Duplicate per replica | Group by service/region | | Flapping threshold | Add `for` duration, hysteresis | | No runbook | Link dashboard + first steps |

					symptom-based alert → user can't checkout (SLO)
cause-based alert  → pod restarted (ticket, not page)
				

Measure alert quality: pages per week, ack-without-action rate, MTTA vs MTTR. Delete alerts nobody acted on in 90 days.

On interviews: paging vs ticketing, on-call rotation sustainability, and distinguishing customer pain from internal noise.

Common pitfalls: alerting on dependencies teams cannot fix at 3am; copy-paste thresholds from blog posts; no ownership per alert.

The trade-off is missing rare failures versus burning out on-call engineers.

Checklist:

• Page on user-impacting symptoms.
• Every page has owner and runbook.
• Review and prune alerts quarterly.
• Track fatigue metrics openly.