In this article
Problem: When an LLM provider degrades, your application hammers the failing endpoint, exhausting connection pools, inflating costs, and degrading all users — not just those on the affected model.
Solution: Apply the Circuit Breaker pattern: track provider health; when failure rate exceeds a threshold, open the circuit (stop sending requests) for a cooldown period; then probe to test recovery.
States:
- Closed (normal): requests flow through; track failure rate
- Open (tripped): immediately return fallback response; no provider calls
- Half-Open (probing): allow a small % of requests through to test recovery
Implementation:
- Track rolling 60-second window: request count + failure count per provider
- Open circuit when error rate > 50% over at least 10 requests
- Open circuit for 30 seconds (configurable)
- In half-open state, allow 10% of requests; close if success rate normalises
Trade-Offs:
- ✔ Pro: Prevents cascading failures and connection pool exhaustion
- ✔ Pro: Fast-fail for users instead of hanging requests
- ✖ Con: False trips during brief transient errors
- ✖ Con: Requires tracking state in Redis or a shared cache (not per-process)
When To Use: Any system making more than ~100 LLM requests/day that needs high availability. When to avoid: Low-traffic internal tools where simple retry is sufficient.