In this article
Problem: Sending every request to GPT-4-class models is wasteful. Most queries (70-80%) can be handled by cheaper, faster models.
Solution: Use a lightweight classifier (fine-tuned small model, embedding similarity, or even regex/rules) to determine user intent, then route to the appropriate handler: simple FAQ → cached response, moderate → GPT-4o-mini, complex reasoning → GPT-4o or Claude Opus.
Implementation:
- Define intent categories (FAQ, summarisation, reasoning, code generation, chitchat)
- Build classifier — options: fine-tuned DistilBERT, embedding + kNN, or LLM-as-judge with a cheap model
- Route to handler with appropriate model, temperature, and system prompt
- Monitor intent distribution and accuracy; retrain classifier periodically
Trade-Offs:
- ✔ Pro: 50-70% cost reduction with maintained quality
- ✔ Pro: Lower average latency (simple queries resolved faster)
- ✖ Con: Classification errors can degrade experience
- ✖ Con: Additional complexity in routing logic and monitoring
When To Use: Multi-purpose chatbots, customer-facing assistants, high-volume APIs. When to avoid: Single-purpose tools where all queries need the same model.
Related Articles
the ai gateway pattern