Costs are high, and most requests are simple. What helps?
Why A is the answer
Routing simple requests to a smaller model
Routing keeps the expensive model for hard work. Check quality with evaluations.
In detail
- Simple requests rarely need the largest model.
- Routing them to a smaller one saves money.
- Measure quality on each route.
Use case
An illustrative scenario, not a real incident.
Most requests are simple lookups. Routing them to a smaller model cuts cost, and hard ones still reach the larger model.
See it in different ways
flowchart TD
A{"Traffic pattern"} -->|"Variable or low"| B["Serverless API or standard"]
A -->|"Steady and high"| C["Provisioned throughput"]stateDiagram-v2 [*] --> CheckRetrieval CheckRetrieval --> TuneChunks: wrong chunks CheckRetrieval --> TunePrompt: right chunks, poor answers TunePrompt --> FineTune: style still not met TuneChunks --> [*] FineTune --> [*]
flowchart TD
Q{"How hard is the request?"}
Q -->|"Simple"| A0["Smaller model"]
Q -->|"Complex"| A1["Larger model"]Key point: Route simple requests to smaller models.
Why the other options are wrong
- B. Sending each request to two models at once
- Two models double the cost.
- C. Raising the max tokens on every single request
- That allows longer, costlier replies.
- D. Adding more context to each of the prompts
- More context costs more.