Monitoring provides visibility into system health over time. The Four Golden Signals (Google SRE): Latency, Traffic, Errors, Saturation — measure these for every service. Prometheus scrapes metrics (pull model) and stores time-series data. Grafana visualises Prometheus data with dashboards. Recording rules pre-compute expensive queries. SLIs (Service Level Indicators) are metrics; SLOs are targets; error budget tracks how much unreliability you have left.
Each stage in order — click any step to read what it does.
Prometheus monitoring architecture with Grafana.
The trade-offs worth knowing before you build this.
Don't try to monitor everything. Start with Latency, Traffic, Errors, and Saturation for each service. These four metrics cover 95% of production incidents.
histogram_quantile(0.99, rate(bucket[5m])) on 30 days of data is slow. Pre-compute with recording rule: record: job:request_latency_p99:rate5m. Dashboards load instantly from pre-computed metrics.
An error budget transforms "is it safe to deploy?" into "do we have budget for risk?". When budget is full, teams can move fast. When it's exhausted, reliability work takes priority.
Sign in to share your feedback and join the discussion.
