Too many alerts are ignored by on-call staff. What should you change?
Why C is the answer
Alert on user-facing symptoms
Alerting on user-facing symptoms gives fewer, more meaningful alerts that people act on.
In detail
- Alert on what users feel.
- Each alert should need action.
Use case
An illustrative scenario, not a real incident.
On-call staff ignore 200 alerts a day. Alerting on slow checkouts gives fewer, useful alerts.
See it in different ways
mindmap
root((Operations))
Tests
Load
Integration
Chaos
Penetration
Observability
Cloud Logging
Cloud Profiler
Learning
Blameless reviewflowchart TD
Q{"Do users feel it?"}
Q -->|"Yes"| A0["Alert"]
Q -->|"No"| A1["Dashboard or log"]Key point: Alert on symptoms.
Why the other options are wrong
- A. Send alerts to a larger group
- That spreads the noise.
- B. Turn all alerts off
- That hides real failures.
- D. Add more alerts for each metric
- That adds noise.