Some evaluation judges compare answers with expected ones. What do they need?
Why B is the answer
Ground truth
A judge that compares answers needs expected answers to compare against, which is what ground truth provides.
In detail
- Some judges require ground truth, and others do not.
- A judge cannot compare with an expected answer unless one exists.
- Other judges can score things such as safety with no expected answer.
Use case
An illustrative scenario, not a real incident.
A team wants to score answers against known correct ones, so it builds a set of questions with expected answers. Without it, that judge has nothing to compare with.
See it in different ways
flowchart LR A["Evaluate on a test set"] --> B["Deploy"] B --> C["Monitor live traffic"] C --> D["Add failures to the test set"] D --> A
sequenceDiagram participant U as Caller participant G as AI Gateway participant E as Endpoint participant T as Inference tables U->>G: Request G->>G: Apply rate limits G->>E: Forward E-->>U: Response E->>T: Record the request and response
flowchart TD
Q{"What does the judge do?"}
Q -->|"Compares with an expected answer"| A0["Needs ground truth"]
Q -->|"Checks safety or tone"| A1["No ground truth needed"]Key point: A comparison needs an expected answer.
Why the other options are wrong
- A. A model with a larger context
- Model size does not supply expected answers.
- C. A custom scorer with no labels
- A scorer with no labels has nothing to compare against.
- D. Inference tables alone
- Logged traffic has no expected answers.