Which eval set best predicts production quality?
Why D is the answer
Real cases, plus edge cases and human-reviewed samples
A mix covers common traffic, known risks and human judgment. A set made by the builders or the model tends to flatter the system.
In detail
- A mix covers common traffic, known risks and human judgment.
- Sets made by the builders or by the model tend to flatter the system.
- Weight the set to match real traffic.
Use case
An illustrative scenario, not a real incident.
A team mixes real tickets, known edge cases and samples reviewed by experts. The set predicts production far better than its old list of written questions.
See it in different ways
flowchart LR A["Real labeled cases"] --> D["Eval suite"] B["Edge and adversarial cases"] --> D C["Human review samples"] --> D D --> E["Release decision"]
mindmap
root((Eval set))
Real traffic
Historical cases
Edge cases
Known failures
Human review
Labeled samples
Slices
Segment
Task type
Riskmindmap
root(("Eval set"))
n1["Real cases"]
n2["Common traffic"]
n3["Edge cases"]
n4["Known risks"]
n5["Human review"]
n6["Judgment on quality"]Key point: Mix real cases, edge cases and human review.
Why the other options are wrong
- A. Questions written by the team who built the assistant
- Builders share the system's blind spots.
- B. Examples generated in bulk by the model under test itself
- A model generating its own test favors itself.
- C. A public benchmark of general knowledge questions
- General knowledge does not measure your task.