Two models solve the same task. How do you compare their loss fairly?
Why A is the answer
Same loss function, on the same held-out data
Comparisons only hold when the metric and data match.
In detail
- Use the same loss on the same held-out data.
- Keep metric and data the same.
Use case
An illustrative scenario, not a real incident.
Two models use different data, so their scores cannot be compared until both use the same set.
See it in different ways
flowchart LR
A["Baseline"] --> B["Change one thing"]
B --> C["Run the same test set"]
C --> D{"Better?"}
D -->|Yes| E["New baseline"]
D -->|No| Amindmap
root((Data analysis))
Metrics
Loss
R-squared
Charts
Line: over time
Scatter: two variables
Care
Same data
Confounding factorsflowchart LR S0["Same loss function"] --> S1["Same held-out data"] --> S2["Compare"]
Key point: Fair comparisons need the same metric and data.
Why the other options are wrong
- B. Each model's own loss on its own training data
- Training data flatters each model.
- C. Any loss function, as long as both are low numbers
- Different losses cannot be compared.
- D. Whichever split gives the newer model a better score
- That is cherry picking.