Offline evals show a small gain for a new prompt. How do you decide whether to ship?
Why B is the answer
Run a limited A/B test against a real outcome metric
Offline evals narrow the options, and live tests measure the outcome. A limited rollout also keeps the risk small.
In detail
- Offline evals narrow the options.
- A live test measures the real outcome.
- A limited rollout keeps the risk small.
- Decide the success metric before the test starts.
Use case
An illustrative scenario, not a real incident.
A new prompt shows a small gain offline. A limited A/B test finds it also lifts resolved tickets, so it rolls out with evidence.
See it in different ways
flowchart LR
A["Offline eval passes"] --> B["Small live A/B test"]
B --> C{"Outcome metric better?"}
C -->|Yes| D["Wider rollout"]
C -->|No| E["Roll back"]sequenceDiagram participant E as Eval suite participant T as Limited test participant P as Production E->>T: Promising version T->>T: Compare a real outcome T->>P: Roll out, or keep the pinned version
flowchart LR S0["Offline eval"] --> S1["Limited A/B test"] --> S2["Measure the outcome"] --> S3["Roll out or drop"]
Key point: Offline evals narrow options, and live tests measure outcomes.
Why the other options are wrong
- A. Ship it to everyone, because the offline score has improved
- A small offline gain may not hold in production.
- C. Ask three colleagues which of the two versions they prefer
- Three opinions are not a measurement.
- D. Reject it, because small gains are never worth the risk involved
- Small gains can matter, so test before rejecting.