Domain experts rate the same answer very differently. What should you do first?
Why C is the answer
Define rubrics and calibrate the experts
Fix the source of disagreement. Averaging or dropping cases hides it.
In detail
- Disagreement means the standard is unclear.
- Define rubrics and calibrate before trusting the scores.
- Measure agreement again after calibration.
Use case
An illustrative scenario, not a real incident.
Three clinicians score the same answer 2, 3 and 5. A rubric with examples, and a calibration session, bring their scores together, so the data becomes usable.
See it in different ways
flowchart LR A["Define a rubric"] --> B["Calibrate experts"] B --> C["Label sampled answers"] C --> D["Evaluate in MLflow"] D --> E["Improve the agent"]
sequenceDiagram participant U as Caller participant G as AI Gateway participant E as Endpoint participant T as Inference tables U->>G: Request G->>G: Apply rate limits G->>E: Forward E-->>U: Response E->>T: Record the request and response
flowchart LR S0["Write a rubric"] --> S1["Calibrate on samples"] --> S2["Check agreement"] --> S3["Label at scale"]
Key point: Align experts before you trust their scores.
Why the other options are wrong
- A. Keep only the cases where all agree
- Dropping cases removes the hard ones you most need.
- B. Replace the experts with an LLM judge
- A judge needs the same clear standard.
- D. Average all of the scores and carry on with the release
- Averaging hides the disagreement and gives you an unreliable score.