Why compare a new model against the previous version on the same test?
Why A is the answer
To detect regressions before users do
Comparing with the previous version on the same test detects regressions before users do.
In detail
- The same test shows regressions before users do.
- Keep the test fixed across versions.
Use case
An illustrative scenario, not a real incident.
A new version beats a public benchmark, but it scores lower than the old one on a team's test.
See it in different ways
flowchart LR
A["New version"] --> B["Canary traffic"]
B --> C{"Metrics good?"}
C -->|Yes| D["Full rollout"]
C -->|No| E["Roll back"]stateDiagram-v2 [*] --> Production Production --> Drift: inputs change Drift --> Retrain Retrain --> Compare: same test as the old version Compare --> Canary Canary --> Production: metrics hold Canary --> Rollback: metrics fall
flowchart TD
Q{"Is it at least as good on the same test?"}
Q -->|"Yes"| A0["Continue"]
Q -->|"No"| A1["Regression, investigate"]Key point: Compare with the previous version.
Why the other options are wrong
- B. To reduce the size of the model's weights
- It does not change weights.
- C. To remove the need for logs in production
- Logs are still needed.
- D. To change the context window of the old model
- It does not change the old model.