Problem
When an LLM provider degrades, your application hammers the failing endpoint, exhausting connection pools, inflating costs, and degrading all users — not just those on the affected model.
Solution
Apply the Circuit Breaker pattern: track provider health; when failure rate exceeds a threshold, open the circuit (stop sending requests) for a cooldown period; then probe to test recovery.
States
- Closed (normal): requests flow through; track failure rate
- Open (tripped): immediately return fallback response; no provider calls
- Half-Open (probing): allow a small % of requests through to test recovery
Implementation
- Track rolling 60-second window: request count + failure count per provider
- Open circuit when error rate > 50% over at least 10 requests
- Open circuit for 30 seconds (configurable)
- In half-open state, allow 10% of requests; close if success rate normalises
Trade-offs
- Pro: Prevents cascading failures and connection pool exhaustion
- Pro: Fast-fail for users instead of hanging requests
- Con: False trips during brief transient errors
- Con: You must decide where breaker state lives: per-process is simplest, while a shared store such as Redis lets every instance trip together
When to use
Any system making more than ~100 LLM requests/day that needs high availability.
When to avoid
Low-traffic internal tools where simple retry is sufficient.