Users report wrong answers, yet the model and prompt are unchanged. How do you start troubleshooting?
Why B is the answer
Inspect logged requests and retrieved context to see what the model received
Find out what the model actually saw. If retrieval returned the wrong context, prompt and model changes will not help.
In detail
- Look at what the model actually received.
- Logs and retrieved context show that.
- Log prompts and retrieved context, with care for private data.
Use case
An illustrative scenario, not a real incident.
Answers worsen with no code change. Logs show that a recent document sync broke, so retrieval returns old content.
See it in different ways
flowchart TD A["GenAI evaluation"] --> B["Retrieval: did it find the right passages?"] A --> C["Model: accuracy and relevance"] A --> D["Agent: task completion"] A --> E["Users: feedback"]
stateDiagram-v2 [*] --> InspectLogs InspectLogs --> Retrieval: wrong context InspectLogs --> Prompt: right context, wrong answer Retrieval --> Evaluate Prompt --> Evaluate Evaluate --> [*]
flowchart TD
Q{"Was the retrieved context right?"}
Q -->|"No"| A0["Fix retrieval"]
Q -->|"Yes"| A1["Look at prompt or model"]Key point: See what the model received.
Why the other options are wrong
- A. Rewrite the prompt with stricter grounding rules and redeploy it
- Changing the prompt is a guess.
- C. Scale up the endpoint in case load is affecting quality
- Load rarely causes wrong content.
- D. Switch to a different model immediately and compare the two
- Changing the model is a guess.