Problem

Sending every request to a frontier model is wasteful when many queries can be handled by cheaper, faster models.

Solution

Use a lightweight classifier (fine-tuned small model, embedding similarity, or even regex/rules) to determine user intent, then route to the appropriate handler: simple FAQ → cached response, moderate → GPT-6 Luna, complex reasoning → GPT-6.1 Sol or Claude Opus 5.5.

Implementation

  1. Define intent categories (FAQ, summarisation, reasoning, code generation, chitchat)
  2. Build classifier — options: fine-tuned DistilBERT, embedding + kNN, or LLM-as-judge with a cheap model
  3. Route to handler with appropriate model, temperature, and system prompt
  4. Monitor intent distribution and accuracy; retrain classifier periodically

Trade-offs

  • Pro: Lower cost per request, in proportion to how much traffic the cheaper tiers handle at acceptable quality (measure it with evals)
  • Pro: Lower average latency (simple queries resolved faster)
  • Con: Classification errors can degrade experience
  • Con: Additional complexity in routing logic and monitoring

When to use

Multi-purpose chatbots, customer-facing assistants, high-volume APIs.

When to avoid

Single-purpose tools where all queries need the same model.