Problem
Sending every request to a frontier model is wasteful when many queries can be handled by cheaper, faster models.
Solution
Use a lightweight classifier (fine-tuned small model, embedding similarity, or even regex/rules) to determine user intent, then route to the appropriate handler: simple FAQ → cached response, moderate → GPT-6 Luna, complex reasoning → GPT-6.1 Sol or Claude Opus 5.5.
Implementation
- Define intent categories (FAQ, summarisation, reasoning, code generation, chitchat)
- Build classifier — options: fine-tuned DistilBERT, embedding + kNN, or LLM-as-judge with a cheap model
- Route to handler with appropriate model, temperature, and system prompt
- Monitor intent distribution and accuracy; retrain classifier periodically
Trade-offs
- Pro: Lower cost per request, in proportion to how much traffic the cheaper tiers handle at acceptable quality (measure it with evals)
- Pro: Lower average latency (simple queries resolved faster)
- Con: Classification errors can degrade experience
- Con: Additional complexity in routing logic and monitoring
When to use
Multi-purpose chatbots, customer-facing assistants, high-volume APIs.
When to avoid
Single-purpose tools where all queries need the same model.