Production Monitoring: The Four Golden Signals and the SLO Stack

Alerts That Mean Something

Most monitoring setups are reactive: CPU alert fires, you look at dashboards, you guess what broke. Good production monitoring flips this — it tells you what users are experiencing before they file tickets, alerts on symptoms not causes, and gives you the context to diagnose fast. The Four Golden Signals and SLOs are the framework.

The Four Golden Signals

Google SRE defined four signals to watch for any service. They're sufficient for detecting most production problems:

flowchart TD
    signals["Four Golden Signals"] --> latency
    signals --> traffic
    signals --> errors
    signals --> saturation

    latency["Latency
How long requests take
(p50, p95, p99 — not average)"]
    traffic["Traffic
How much demand
(requests/sec, events/sec)"]
    errors["Errors
Failed requests rate
(5xx, timeouts, business errors)"]
    saturation["Saturation
How full the service is
(CPU, memory, connection pool, queue depth)"]

    style signals fill:#0078D4,color:#fff,stroke:#005a9e

SLI / SLO / SLA

// SLI — Service Level Indicator
// A quantitative measure of service behaviour
// Example: proportion of requests served in < 200ms

const sli = {
  name: 'API latency',
  query: 'sum(rate(http_request_duration_seconds_bucket{le="0.2"}[5m])) / sum(rate(http_request_duration_seconds_count[5m]))',
  // Value: 0.0 to 1.0
};

// SLO — Service Level Objective
// The target value for an SLI
// Example: 99.5% of requests in < 200ms over a 30-day rolling window

const slo = {
  sli: 'API latency',
  target: 0.995,    // 99.5%
  window: '30d',
};

// SLA — Service Level Agreement
// The contractual commitment to customers (usually weaker than SLO)
// Example: 99.9% uptime (weaker than your 99.5% latency SLO)

// Error budget = time/requests you can "spend" on incidents
// Error budget = (1 - SLO target) * window
// = (1 - 0.995) * 30 days
// = 0.005 * 43,200 minutes
// = 216 minutes per month of allowed degraded latency

Error Budget Alerts

Alert on error budget burn rate, not on individual metric thresholds. A 5xx spike at 3am that uses 2% of your monthly error budget is not worth waking someone up. A sustained 5x burn rate that will exhaust the budget in 6 hours is.

# Prometheus alerting rules — SLO burn rate approach
groups:
  - name: api_slo_burn_rate
    rules:
      # Fast burn: high severity — page immediately
      # 14.4x burn rate = budget exhausted in 2 hours
      - alert: APILatencySLOHighBurn
        expr: |
          sum(rate(http_request_duration_seconds_bucket{le="0.2"}[1h]))
          / sum(rate(http_request_duration_seconds_count[1h])) < 0.931
          and
          sum(rate(http_request_duration_seconds_bucket{le="0.2"}[5m]))
          / sum(rate(http_request_duration_seconds_count[5m])) < 0.931
        labels:
          severity: critical
        annotations:
          summary: "API latency SLO burning fast — page on-call"
          description: "14.4x burn rate. Budget exhausted in ~2h if sustained."

      # Slow burn: ticket only — budget exhausted in 3 days
      - alert: APILatencySLOSlowBurn
        expr: |
          sum(rate(http_request_duration_seconds_bucket{le="0.2"}[6h]))
          / sum(rate(http_request_duration_seconds_count[6h])) < 0.985
        for: 60m
        labels:
          severity: warning
        annotations:
          summary: "API latency SLO degraded — create ticket"
          description: "Budget burning at elevated rate. No immediate action needed."

      # Don't do this:
      # - alert: HighCPU
      #   expr: cpu_usage > 0.9
      # This is a cause, not a symptom. CPU at 90% may be fine.

Structured Logging

import pino from 'pino';

// ✅ Structured — queryable, filterable, correlatable
const logger = pino({
  level: process.env.LOG_LEVEL ?? 'info',
  base: {
    service: 'api-server',
    version: process.env.APP_VERSION,
    env: process.env.NODE_ENV,
  },
});

// Log with context that enables downstream queries
export function logRequest(req: Request, res: Response, durationMs: number) {
  logger.info({
    msg: 'HTTP request',
    method: req.method,
    path: req.path,
    statusCode: res.statusCode,
    durationMs,
    requestId: req.headers['x-request-id'],
    userId: req.user?.id,    // correlate errors to users
    traceId: req.traceId,    // correlate to distributed traces
  });
}

// Error logging with full context
export function logError(error: Error, context: Record<string, unknown>) {
  logger.error({
    msg: 'Unhandled error',
    err: {
      message: error.message,
      stack: error.stack,
      name: error.name,
    },
    ...context,
  });
}

// ❌ Unstructured — impossible to query at scale
// console.log('User ' + userId + ' failed to checkout: ' + error.message);

Distributed Tracing

// OpenTelemetry — vendor-neutral tracing
import { NodeSDK } from '@opentelemetry/sdk-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
import { HttpInstrumentation } from '@opentelemetry/instrumentation-http';
import { ExpressInstrumentation } from '@opentelemetry/instrumentation-express';
import { PgInstrumentation } from '@opentelemetry/instrumentation-pg';

// src/instrumentation.ts — loaded before everything else
const sdk = new NodeSDK({
  serviceName: 'api-server',
  traceExporter: new OTLPTraceExporter({
    url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT,
  }),
  instrumentations: [
    new HttpInstrumentation(),        // auto-instrument all HTTP calls
    new ExpressInstrumentation(),     // auto-instrument Express routes
    new PgInstrumentation(),          // auto-instrument PostgreSQL queries
  ],
});

sdk.start();

// Manual span for custom operations
import { trace } from '@opentelemetry/api';

const tracer = trace.getTracer('api-server');

async function processPayment(orderId: string) {
  return tracer.startActiveSpan('process-payment', async (span) => {
    try {
      span.setAttribute('order.id', orderId);
      const result = await paymentProvider.charge(orderId);
      span.setAttribute('payment.provider', 'stripe');
      return result;
    } catch (error) {
      span.recordException(error as Error);
      span.setStatus({ code: SpanStatusCode.ERROR });
      throw error;
    } finally {
      span.end();
    }
  });
}

Pitfalls

Alerting on averages

Average latency hides tail latency problems. If 99% of your requests complete in 50ms but 1% take 10 seconds, your average might look fine at 150ms. Always use percentiles (p95, p99) for latency SLOs. Your slowest users are the ones who churn.

Too many dashboards, not enough context

A 50-panel Grafana dashboard that shows everything doesn't help you during an incident. Create a single “service health” dashboard that shows the four golden signals and nothing else. Link from alert to dashboard to runbook — make the path automatic.

No runbooks

Every alert should have a runbook: what to check first, what actions to take, when to escalate. A runbook doesn't need to be long — three bullet points written by whoever created the alert is 10x better than nothing. The goal is any on-call engineer can handle it, not just the one who wrote the code.