Alerting: Signal vs Noise

Problem Context

Bad alerting is worse than no alerting. Pages at 3 a.m. for transient blips train the on-call to mute everything; missed alerts on real incidents bleed users. Good alerting is a small number of rules, each tied to user-visible symptoms, with a runbook and an owner. Everything else is a dashboard.

Alert on Symptoms, Not Causes

Alert when the user is unhappy: error-rate spike, latency SLO burn, broken checkout. Cause-based alerts (high CPU, queue depth) belong on dashboards and as auto-remediationtriggers, not as pages. A high CPU that doesn't affect users is not an incident.

flowchart LR
    M[Metrics + logs + traces] --> S[Symptom rules<br/>SLO burn, error rate, saturation]
    M --> C[Cause rules<br/>CPU, queue, retries]
    S --> P[Page]
    C --> AR[Auto-remediate / dashboard]
    P --> RB[Runbook]
    RB --> OC[On-call]
      

SLOs and Burn-Rate Alerts

The Google SRE multi-window burn-rate pattern is the modern default. Two pages, two dashboards:

• Fast burn — 2% of monthly error budget in 1 hour (14.4× rate). Page immediately.
• Slow burn — 10% in 6 hours (6× rate). Page during business hours.

Burn-rate alerts catch real degradations without flapping on single-minute blips. Required: a defined SLI (e.g. successful HTTP 200/non-429 requests < 500 ms) and an SLO (99.9% per 30 days).

The Five Things Every Alert Must Have

1. Owner — a team alias, not a person.
2. Runbook link — "what to check, in what order". No runbook = delete the alert.
3. Severity — page (wake up) vs ticket (next business day). One scale, ruthlessly applied.
4. Dashboard link — direct to the chart that proves/disproves the symptom.
5. Recent deploys — auto-attached. 70% of incidents correlate with a deploy in the last 30 minutes.

Alert Hygiene Metrics

Track and review monthly:

• Pages per shift — > 2 means you have noise to cull.
• False-positive rate — > 30% means thresholds need tuning.
• Time-to-acknowledge — rising trend means alert fatigue.
• Mean time to detect (MTTD) — the value alerting actually delivers.

Multi-Window, Multi-Burn-Rate in PromQL

# Fast burn: > 14.4× over 1h AND > 14.4× over 5m
(
  (sum(rate(http_requests_total{code=~"5.."}[1h])) / sum(rate(http_requests_total[1h]))) > 0.0144
  and
  (sum(rate(http_requests_total{code=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))) > 0.0144
)
labels:  severity=page
annotations:
  runbook: https://wiki/runbooks/api-error-rate
  dashboard: https://grafana/d/api-errors

Inhibition and Grouping

• Inhibition — if "cluster down" fires, don't also page for every individual service in that cluster.
• Grouping — bundle alerts by service/cluster into one page with N items, not N pages.
• Routing — service A pages team A, service B pages team B; shared infra pages SRE. Define in code, review quarterly.

Notification Channels

• Page — phone call/push (PagerDuty, Opsgenie, Squadcast). Reserve for "wake up now".
• Ticket — Slack/Teams + Jira/Linear issue. For things that can wait until morning.
• Status page — auto-updated from a small set of customer-impact alerts. Subscribers love this.

Failure Modes

• CPU/memory pages — alert on saturation symptoms (queue grows, latency rises), not the raw resource.
• Threshold drift — "always 500 errors/min" becomes the new normal; tighten thresholds quarterly.
• One giant rule file — no ownership, no review. Split by service, store in the service's repo.
• Alerts without runbooks — on-call wastes 10 minutes per page guessing the right dashboard.

Production Checklist

• Burn-rate alerts on every user-facing SLO; fast + slow window pair.
• Owner, runbook, severity, dashboard, recent-deploys on every alert.
• Inhibition rules so one cluster-down doesn't produce 200 pages.
• Monthly alert-hygiene review: cull silent, flapping, ownerless rules.

Key Takeaways

• Alert on user-visible symptoms; everything else is a dashboard or an auto-remediation hook.
• Burn-rate alerts are the modern default — they catch real degradations without flapping.
• An alert without an owner and a runbook is noise; delete or fix it.