A change improves average quality but badly hurts one customer segment. What should the review catch?
Why A is the answer
Results broken down by segment, not just the overall average
Averages hide regressions. Slice results by segment, task type and risk before a release decision.
In detail
- Averages hide regressions.
- Slice results by segment, task type and risk before you decide.
- Choose your segments before you look at the results.
Use case
An illustrative scenario, not a real incident.
A change lifts average quality, but answers for one region get far worse. Results split by segment show this before release.
See it in different ways
mindmap
root((Eval set))
Real traffic
Historical cases
Edge cases
Known failures
Human review
Labeled samples
Slices
Segment
Task type
Riskmindmap
root(("Release review"))
n1["Overall"]
n2["Average quality"]
n3["By segment"]
n4["Customer type"]
n5["Region"]
n6["By risk"]
n7["High-impact tasks"]Key point: Averages hide regressions.
Why the other options are wrong
- B. Nothing, because the average is the measure that the team agreed on
- An agreed average does not make the harm acceptable.
- C. Whether the segment is small enough to leave out of future reports
- Hiding a segment leaves the harm in place.
- D. Whether the segment's users could be moved to a different product
- Moving users avoids the problem without fixing it.