Deployments That Don't Wake You Up at 3am

Kubernetes Deployments manage rollouts, rollbacks, and replica sets — but the defaults are wrong for production. Zero-downtime rolling updates require correct readinessProbe config. Resource limits prevent one misbehaving pod from killing a node. HPA scales your workload without manual intervention. None of this works without deliberate configuration.

🤔 Sound familiar?
  • Your rolling updates drop requests because pods are marked ready before the app actually handles traffic
  • A memory leak in one pod takes down other services on the same node
  • You scale manually by editing replicas in YAML and reapplying
  • You don't know the difference between requests and limits

Properly configured readiness probes, resource limits, and HPA are the difference between “it usually works” and a production-grade deployment.

Rolling Updates Done Right


sequenceDiagram
    participant K as Kubernetes
    participant P1 as Pod v1
    participant P2 as Pod v2 (new)
    participant LB as Load Balancer

    K->>P2: Create new pod
    P2->>P2: App starting...
    K->>P2: readinessProbe fails
    Note over LB,P2: No traffic sent yet
    P2->>P2: App ready
    K->>P2: readinessProbe passes
    LB->>P2: Traffic starts routing
    K->>P1: Terminate old pod

# k8s/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-server
  labels:
    app: api-server
spec:
  replicas: 3
  
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1         # Allow 1 extra pod during update (max 4 total)
      maxUnavailable: 0   # Never take pods offline before replacement is ready
  
  selector:
    matchLabels:
      app: api-server
  
  template:
    metadata:
      labels:
        app: api-server
    spec:
      containers:
        - name: api
          image: ghcr.io/myorg/api:sha-abc123
          ports:
            - containerPort: 3000
          
          # Resource requests: what the scheduler uses for placement
          # Resource limits: hard ceiling — OOMKill at memory limit
          resources:
            requests:
              memory: "128Mi"
              cpu: "100m"      # 100 millicores = 0.1 CPU core
            limits:
              memory: "512Mi"
              cpu: "500m"      # 0.5 CPU core
          
          # Readiness: is this pod ready to receive traffic?
          # Kubernetes won't route traffic until this passes
          readinessProbe:
            httpGet:
              path: /health/ready
              port: 3000
            initialDelaySeconds: 10
            periodSeconds: 5
            failureThreshold: 3
          
          # Liveness: is this pod still alive?
          # Kubernetes will restart the pod if this fails
          livenessProbe:
            httpGet:
              path: /health/live
              port: 3000
            initialDelaySeconds: 30
            periodSeconds: 15
            failureThreshold: 3
          
          # Startup: only checked during startup (prevents liveness killing slow-starting pods)
          startupProbe:
            httpGet:
              path: /health/live
              port: 3000
            failureThreshold: 30  # 30 * 10s = 5 minutes for startup
            periodSeconds: 10
          
          # Graceful shutdown — send SIGTERM and wait for in-flight requests
          lifecycle:
            preStop:
              exec:
                command: ["/bin/sleep", "5"]  # Give LB time to stop routing
          
          envFrom:
            - secretRef:
                name: api-secrets
            - configMapRef:
                name: api-config
      
      terminationGracePeriodSeconds: 30

Health Endpoints

// src/routes/health.ts — Express example
import { Router } from 'express';
import { db } from '../db';

const router = Router();

// Liveness: is the process healthy? No external checks.
// Fast — Kubernetes calls this constantly
router.get('/health/live', (req, res) => {
  res.status(200).json({ status: 'ok' });
});

// Readiness: can this pod serve traffic?
// Checks dependencies — fails if they're unreachable
router.get('/health/ready', async (req, res) => {
  try {
    await db.raw('SELECT 1'); // Check DB connection
    res.status(200).json({ status: 'ok', db: 'connected' });
  } catch (error) {
    // Return 503 — Kubernetes won't route traffic to this pod
    res.status(503).json({ status: 'unavailable', db: 'disconnected' });
  }
});

export { router as healthRouter };

Horizontal Pod Autoscaler

# k8s/hpa.yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api-server-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api-server
  
  minReplicas: 2   # Never scale below 2 (for availability)
  maxReplicas: 20  # Never scale above 20
  
  metrics:
    # Scale on CPU utilization
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70  # Scale up when average CPU > 70% of requests
    
    # Scale on memory utilization
    - type: Resource
      resource:
        name: memory
        target:
          type: Utilization
          averageUtilization: 80
  
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 60  # Wait 60s before scaling up again
      policies:
        - type: Pods
          value: 2
          periodSeconds: 60  # Add max 2 pods per 60s
    
    scaleDown:
      stabilizationWindowSeconds: 300  # Wait 5min before scaling down
      policies:
        - type: Percent
          value: 25
          periodSeconds: 60  # Remove max 25% of pods per 60s

Rollbacks

# Check rollout status
kubectl rollout status deployment/api-server

# View rollout history (requires --record or annotated deployments)
kubectl rollout history deployment/api-server

# Rollback to previous version
kubectl rollout undo deployment/api-server

# Rollback to specific revision
kubectl rollout undo deployment/api-server --to-revision=3

# Pause a rollout (useful for canary-style manual checks)
kubectl rollout pause deployment/api-server
# ... check metrics ...
kubectl rollout resume deployment/api-server

Pitfalls

No CPU limits

CPU limits are controversial — some teams remove them to avoid CPU throttling. The tradeoff: without limits, a noisy neighbour pod can starve others on the node. Set limits at 3-5x your request value as a safety ceiling. Always set memory limits.

readinessProbe too aggressive

A readiness probe with failureThreshold: 1 and periodSeconds: 5 will temporarily remove pods from load balancing on a single slow health check response. Set failureThreshold to 3+ to tolerate transient slowness.

Forgetting PodDisruptionBudget for rollouts

HPA and node maintenance can both remove pods simultaneously. A PodDisruptionBudgetsets a floor: always keep at least N pods available, regardless of what's happening. Without it, you can briefly hit zero replicas during a node drain + rollout.