Average quality looks fine, but users report failures on one topic. What should you do?
Why B is the answer
Slice error analysis by topic
Averages hide failures in a segment.
In detail
- Averages hide failures in a segment.
- Slice errors by topic.
- Averages hide segments.
Use case
An illustrative scenario, not a real incident.
Overall scores look fine, but slicing by topic shows that billing answers fail.
See it in different ways
mindmap
root((Evaluation))
Translation
BLEU
Summaries
ROUGE
RAG
Faithfulness
Errors
Slice by topicmindmap
root(("Quality by topic"))
n1["Good"]
n2["Shipping"]
n3["Returns"]
n4["Failing"]
n5["Billing"]Key point: Slice errors by topic.
Why the other options are wrong
- A. Increase the batch size in the serving layer
- Batch size is unrelated.
- C. Report the average, since it passed the target
- Passing on average ignores those users.
- D. Remove that topic from the evaluation set
- Removing the topic hides it.