Effective alerting wakes someone up only when human action is required. Alert fatigue — too many low-signal alerts — causes on-call engineers to ignore pages. Principles: alert on symptoms not causes (high error rate, not high CPU), alert on SLO burn rate not point-in-time values, require every alert to have a runbook. AlertManager routes, groups, and deduplicates alerts. PagerDuty/OpsGenie manage on-call schedules and escalation policies.
Multi-window burn rate alerting for SLO violations.
high_cpu_alert fires 20 times per week and is never actionable. high_error_rate fires twice per week, always requires action. Alert on what users experience, not internal system metrics.
If you can't write a runbook for an alert, you don't understand it well enough to page someone for it. Every alert must have runbook_url annotation with documented triage and remediation steps.
Alerts accumulate over time. Schedule a 30-minute weekly review: which alerts fired most often? Were they actionable? Prune or increase thresholds for noisy alerts. Target <2 alerts per week per service.
Sign in to share your feedback and join the discussion.