Answer quality fell after a model upgrade, and retrieval scores did not move. What is the sound response?
Why A is the answer
Rerun the eval suite on both versions and pin the previous version while you tune prompts
Measure the difference, protect production with a version pin, then tune. Retrieval is ruled out, so changing it would not help.
In detail
- Retrieval is ruled out, so the change came from the model or how it reads your prompts.
- Measure both versions, and pin the previous one to protect production.
- Then tune the prompts against the eval.
- Change one thing at a time while tuning.
Use case
An illustrative scenario, not a real incident.
Quality drops after an upgrade, while retrieval is unchanged. The team pins the old version for production, compares both on the eval suite, and tunes prompts before moving over.
See it in different ways
flowchart TD
A["Bad answer"] --> B{"Was the right passage retrieved?"}
B -->|No| C["Retrieval problem"]
B -->|Yes| D{"Does a clearer prompt fix it?"}
D -->|Yes| E["Prompt problem"]
D -->|No| F["Model mismatch"]sequenceDiagram participant E as Eval suite participant T as Limited test participant P as Production E->>T: Promising version T->>T: Compare a real outcome T->>P: Roll out, or keep the pinned version
stateDiagram-v2 state "Quality falls" as s0 state "Pin previous version" as s1 state "Compare on evals" as s2 state "Tune prompts" as s3 state "Move to new version" as s4 [*] --> s0 s0 --> s1: protect users s1 --> s2 s2 --> s3 s3 --> s4: evals pass s4 --> [*]
Key point: Protect production, then measure and tune.
Why the other options are wrong
- B. Roll out to everyone and watch customer complaints to see which prompts need work
- This uses customers as the test while quality is known to be worse.
- C. Double the retrieved chunks and see whether more context brings quality back
- Retrieval scores did not change, so more chunks would not help.
- D. Rewrite all prompts from scratch immediately, then ship them with the new model
- A full rewrite changes everything at once, with no evidence of what helped.