Problem Context

A single server can't serve a popular product. The cure is N servers behind a load balancer that distributes requests so no single machine drowns. The mechanics get interesting fast: which algorithm? Layer 4 or layer 7? How do you handle sticky sessions, slow backends, sudden failures, blue/green deploys, and TLS termination?

Modern load balancers do far more than "round-robin" — health checks, outlier ejection, weighted traffic, header-based routing, request hedging, retry budgets, and circuit breaking are all the load balancer's job in 2026. This guide is the layer choices, the algorithms, and the operational behaviour that decides whether one bad backend takes your service down.

🤔 Sound familiar?
  • You picked round-robin and one backend with a slow GC took the fleet down via timeouts
  • You can't explain when to use Azure Front Door vs Application Gateway vs Load Balancer
  • Your sticky sessions broke a rolling deploy
  • You don't have outlier ejection and a half-dead instance is still in rotation

This is the load-balancer mental model — L4 vs L7, the algorithms that matter, health checks that work, and the Azure service that fits each tier.

Concept Explanation

Two layers, very different jobs:

  • L4 (TCP/UDP) — Azure Load Balancer, AWS NLB. Routes packets by 5-tuple (src ip, src port, dst ip, dst port, protocol). Cheap, fast, no app awareness. Best for raw TCP/UDP, gRPC at scale, and intra-zone traffic.
  • L7 (HTTP/HTTPS) — Azure Front Door (global), Application Gateway (regional), nginx, Envoy. Understands HTTP — can route on host, path, headers; terminate TLS; rewrite; cache; WAF.

flowchart LR
    USR["Users (global)"] --> FD["Azure Front Door<br/>L7 + WAF + CDN<br/>(global anycast)"]
    FD -->|host: api.x.com| AG["App Gateway<br/>L7 (regional)"]
    FD -->|host: cdn.x.com| BLOB["Storage Static<br/>(direct origin)"]
    AG --> POOL["Backend Pool"]
    POOL --> B1["Backend 1<br/>(healthy)"]
    POOL --> B2["Backend 2<br/>(healthy)"]
    POOL -.->|ejected| B3["Backend 3<br/>(5xx > 50%)"]

    style FD fill:#0078D4,color:#fff,stroke:#005a9e
    style AG fill:#0078D4,color:#fff,stroke:#005a9e
    style B3 fill:#dc2626,color:#fff,stroke:#b91c1c

Implementation

Step 1: Algorithm — pick by latency variance, not feel

Round-robin                — simple, fair when backends are identical and fast
Weighted round-robin       — different SKUs (canary at 5%, prod at 95%)
Least connections          — long-lived connections (websockets, gRPC)
Least response time / EWMA — heterogeneous backends with variable latency
Power-of-two-choices       — pick 2 random backends, send to the less loaded one
                             (closest to optimal, no global state)
Consistent hash            — sticky-by-key (cache-friendly, session affinity)

Modern proxies (Envoy, YARP, Front Door) default to round-robin but expose least-request and P2C— switch to P2C the moment your backends have variable latency. It's a one-line change and prevents most "why is one server hot?" incidents.

Step 2: Health checks that catch real failures

// /healthz that actually verifies dependencies
app.MapHealthChecks("/healthz", new HealthCheckOptions
{
    ResultStatusCodes =
    {
        [HealthStatus.Healthy]   = 200,
        [HealthStatus.Degraded]  = 200,   // serve traffic, but warn
        [HealthStatus.Unhealthy] = 503    // remove from rotation
    }
});

builder.Services.AddHealthChecks()
    .AddSqlServer(connStr, name: "sql", failureStatus: HealthStatus.Unhealthy)
    .AddRedis(redisConn,  name: "redis", failureStatus: HealthStatus.Degraded)
    .AddCheck("self", () => HealthCheckResult.Healthy());
Health-check rules:
  • Probe every 5–10s, mark unhealthy after 2 consecutive failures
  • Drain (stop new requests, finish in-flight) before marking down
  • Liveness  → "is the process responsive?" (cheap)
  • Readiness → "should it get traffic?" (checks deps; may flap)
  • Don't fail readiness on transient downstream blips — use a sliding window

Step 3: Outlier ejection — the "sick instance" defence

# Envoy / YARP / App Gateway equivalent: eject backends with elevated 5xx rate
outlierDetection:
  consecutive5xx: 5
  interval: 10s
  baseEjectionTime: 30s     # remove for 30s, then probe back in
  maxEjectionPercent: 50    # never eject more than half the pool

Step 4: Sticky sessions — only when you must

Stateless services don't need stickiness — every request is independent.

If you DO need session affinity (legacy in-memory sessions, websocket reconnects):
  • Cookie-based affinity (App Gateway: ARRAffinity / SessionAffinityEnabled)
  • TTL the cookie short (1 hour) so deploys actually rotate users off old pods
  • Have a fallback: when the affine instance disappears, fail over cleanly

Better long-term: externalize session state (Redis) and lose stickiness entirely.

Step 5: Azure: pick the right service

Azure Front Door (Standard/Premium)
  • Global L7 + CDN + WAF; anycast routing to the nearest PoP
  • Use as the public entry for any internet-facing app
  • Premium: managed WAF rules, Private Link to origin

Application Gateway (v2)
  • Regional L7; URL/path/header routing; WAF; mTLS; autoscale
  • Use behind Front Door for regional fan-out, or solo for internal apps

Azure Load Balancer (Standard)
  • Regional L4; the only choice for non-HTTP TCP/UDP
  • Free with VMs (Basic) — but Basic SKU is being retired; use Standard

Traffic Manager
  • DNS-based global routing (priority/weighted/geographic)
  • Cheap, but the resolution latency + TTL caching make it slower than Front Door

Step 6: Client-side resilience — the LB isn't magic

// Polly v8 — retries with budget, circuit breaker, hedging
builder.Services.AddHttpClient("api")
  .AddResilienceHandler("std", builder =>
  {
      builder.AddRetry(new HttpRetryStrategyOptions
      {
          MaxRetryAttempts = 2,
          BackoffType = DelayBackoffType.Exponential,
          UseJitter = true,
          ShouldHandle = new PredicateBuilder<HttpResponseMessage>()
              .Handle<HttpRequestException>()
              .HandleResult(r => (int)r.StatusCode >= 500)
      });
      builder.AddCircuitBreaker(new HttpCircuitBreakerStrategyOptions
      {
          FailureRatio = 0.5, MinimumThroughput = 20,
          BreakDuration = TimeSpan.FromSeconds(30)
      });
      builder.AddTimeout(TimeSpan.FromSeconds(2));
  });

Pitfalls

1. Round-robin with heterogeneous backends. One slow GC pause and round-robin keeps sending requests into the abyss. Use P2C or least-request and the slow node sheds load automatically.

2. Health checks that are too lenient. A liveness probe that just returns 200 from /health never catches a stuck DB connection pool. Verify dependencies in readiness; keep liveness cheap.

3. Health checks that are too strict. Failing readiness on a 50ms Redis blip ejects the whole fleet. Use a sliding window (3-of-5 failures) before marking down.

4. Sticky sessions that survive deploys. Cookie-affine to a pod that just got rolled — the user gets a 502. Short TTLs and graceful drains.

5. No retry budget. Polly retries on every 5xx; downstream dies; retries amplify load 3x; downstream takes longer to recover. Cap retries (2 attempts), require jitter, and pair with a circuit breaker.

6. TLS terminated only at the edge.Plain HTTP from the LB to backends is fine in a private VNet; on the public internet or untrusted network it's not. Use end-to-end TLS or service mesh mTLS.

Practical Takeaways

  • Public internet entry → Azure Front Door. Regional fan-out → Application Gateway. TCP/UDP → Standard Load Balancer.
  • Default algorithm: P2C or least-request the moment latency varies. Round-robin is the demo answer.
  • Liveness = process responsive. Readiness = dependencies OK. Drain before marking down.
  • Always enable outlier ejection. Cap maxEjectionPercentso a flap doesn't empty the pool.
  • Avoid sticky sessions if you can. Externalize session state instead.
  • Pair the LB with client-side retries (capped + jittered) and circuit breakers — the LB alone isn't enough.
  • End-to-end TLS over any network you don't fully control.