You change the prompt and want to know whether quality truly improved. What is the sound method?
Why D is the answer
Compare both versions on the same labeled test set
A fixed test set gives a fair comparison. Single examples mislead.
In detail
- A fair test uses the same labeled set for both.
- One good example proves little.
Use case
An illustrative scenario, not a real incident.
A team compares two prompts on 200 labeled questions, and the new one wins by a clear margin.
See it in different ways
flowchart LR
A["Baseline"] --> B["Change one thing"]
B --> C["Run the same test set"]
C --> D{"Better?"}
D -->|Yes| E["New baseline"]
D -->|No| Amindmap
root((Data analysis))
Metrics
Loss
R-squared
Charts
Line: over time
Scatter: two variables
Care
Same data
Confounding factorsflowchart LR S0["Fixed test set"] --> S1["Run version A"] --> S2["Run version B"] --> S3["Compare scores"]
Key point: Compare on one fixed test set.
Why the other options are wrong
- A. Try one difficult example and judge both outputs by eye
- One example proves little.
- B. Ask the model which of the two versions it prefers
- A model's preference is not evidence.
- C. Measure which version produces the shorter response
- Shorter does not mean better.