Alerts

Alerts turn telemetry into human action. Good alerts page on user-impacting symptoms, include severity and ownership, and link to dashboards or runbooks that support fast mitigation.

| Good alert | Bad alert | |------------|-----------| | Elevated checkout error rate for 5m | Any 500 in logs | | SLO burn rate exceeded | CPU > 50% on one pod | | Queue age above threshold | Disk 60% full with no trend |

Use multi-window burn-rate alerts for SLOs. Route by severity and time zone with escalation policies.

On interviews: alert fatigue, burn-rate alerts, actionable thresholds, routing, escalation, maintenance windows, and paging discipline.

Common pitfalls: paging on causes instead of symptoms; alerts without runbooks; never reviewing noisy rules.

The trade-off is sensitivity (catch everything early) versus on-call sustainability.

Checklist:

• Alert on user impact or SLO threat.
• Provide runbook context in notifications.
• Review and tune noise regularly.
• Test alert paths in game days.