You need an eval set for a support assistant before launch. Which source gives you the most trustworthy results?
Why B is the answer
Real historical tickets labeled by reviewers, including edge cases
Evaluate on data that looks like production, and include the hard cases. Public benchmarks measure general ability, not your task.
In detail
- An eval set should look like production.
- Real tickets labeled by reviewers include the hard cases that matter most.
- Refresh the set as products and policies change.
Use case
An illustrative scenario, not a real incident.
A telco builds an eval from 500 past tickets, labeled by senior agents, including angry and ambiguous ones. It exposes failures that the team's own examples never showed.
See it in different ways
flowchart LR A["Real labeled cases"] --> D["Eval suite"] B["Edge and adversarial cases"] --> D C["Human review samples"] --> D D --> E["Release decision"]
mindmap
root((Eval set))
Real traffic
Historical cases
Edge cases
Known failures
Human review
Labeled samples
Slices
Segment
Task type
Riskflowchart TD
Q{"Where did the test cases come from?"}
Q -->|"Real tickets, reviewed"| A0["Trustworthy"]
Q -->|"The builders"| A1["Flattering"]
Q -->|"A public benchmark"| A2["Not your task"]Key point: Evaluate on data that looks like production.
Why the other options are wrong
- A. A handful of happy-path examples written by the team that built it
- Builders write the cases they already expect to pass.
- C. A public benchmark for general reasoning, plus a few questions from the team
- A public benchmark measures general ability, not your task.
- D. Synthetic questions the model generates about its own product knowledge
- Questions generated by the model tend to flatter the model.