Problem: Sending every request to GPT-4-class models is wasteful. Most queries (70-80%) can be handled by cheaper, faster models.
Solution: Use a lightweight classifier (fine-tuned small model, embedding similarity, or even regex/rules) to determine user intent, then route to the appropriate handler: simple FAQ → cached response, moderate → GPT-4o-mini, complex reasoning → GPT-4o or Claude Opus 5.
Implementation:
- Define intent categories (FAQ, summarisation, reasoning, code generation, chitchat)
- Build classifier — options: fine-tuned DistilBERT, embedding + kNN, or LLM-as-judge with a cheap model
- Route to handler with appropriate model, temperature, and system prompt
- Monitor intent distribution and accuracy; retrain classifier periodically
Trade-Offs:
- Pro: 50-70% cost reduction with maintained quality
- Pro: Lower average latency (simple queries resolved faster)
- Con: Classification errors can degrade experience
- Con: Additional complexity in routing logic and monitoring
When To Use: Multi-purpose chatbots, customer-facing assistants, high-volume APIs. When to avoid: Single-purpose tools where all queries need the same model.