You classify 2 million support tickets into 12 categories, and Haiku scores the same as Opus on your labeled test set. What should you ship?
Why D is the answer
Haiku, because the eval shows no quality gain from paying more
Let your own eval decide. If the cheap tier matches on the labeled set, the larger model only adds cost and latency for this task.
In detail
- Your own labeled eval is the best evidence for your task.
- If Haiku matches Opus there, the larger model only adds cost and latency.
- At 2 million tickets, that difference is large.
- Rerun the eval when the data or the task changes.
Use case
An illustrative scenario, not a real incident.
A support platform tests three tiers on 2,000 labeled tickets. Haiku matches Opus, so the team ships Haiku and saves on every ticket, with a weekly sample still checked by hand.
See it in different ways
flowchart LR
A["Labeled test set"] --> B["Run each model"]
B --> C{"Same quality?"}
C -->|Yes| D["Ship the cheaper model"]
C -->|No| E["Ship the one that passes"]flowchart TD
Q{"Does the cheaper tier match on your eval?"}
Q -->|"Yes"| A0["Ship the cheaper tier"]
Q -->|"No"| A1["Move up a tier and measure again"]Key point: Your eval decides, not the model's rank.
Why the other options are wrong
- A. Opus, because the strongest model is always the safest choice
- The strongest model is not automatically right when the eval shows no gain.
- B. Sonnet, because the middle tier is the standard default
- There is no default middle tier. The eval should decide.
- C. Whichever model has the largest context window
- Ticket classification does not need a large context window.