In this article
Problem: LLM providers experience outages, rate limits, and high latency. Depending on a single provider creates single points of failure in production.
Solution: Define a prioritised provider chain and automatically route to the next provider on failure.
Implementation:
- Define chain: Primary (GPT-4o via Azure OpenAI) → Fallback 1 (GPT-4o via OpenAI direct) → Fallback 2 (Claude 3.5 via Anthropic)
- Trigger conditions: HTTP 429 (rate limited), HTTP 5xx (provider error), timeout > SLO (e.g., 30s)
- Circuit state: Track failure rate per provider; skip degraded providers proactively (circuit breaker)
- Prompt adaptation: If switching model families (OpenAI → Claude), apply per-model prompt templates
LiteLLM pattern:
from litellm import completion
response = completion(
model="gpt-4o",
fallbacks=["claude-3-5-sonnet-20241022", "gemini/gemini-1.5-pro"],
messages=[{"role": "user", "content": prompt}]
)
Trade-Offs:
- ✔ Pro: High availability — provider outages become seamless
- ✔ Pro: Cost optimisation — route to cheaper models during low-priority periods
- ✖ Con: Different model families produce inconsistent outputs — test each fallback path
- ✖ Con: Cost unpredictability if fallbacks are more expensive
When To Use: Any production system with SLA requirements and more than 1000 RPD. When to avoid: Internal tools where brief downtime is acceptable.
Related Articles
the ai gateway pattern