Answers are wrong, and the right document exists in the knowledge base. What do you check first?
Why B is the answer
Whether the retrieved context actually held the right passage
Retrieval is the first layer. If the passage never arrived, no prompt or model change will fix it.
In detail
- Retrieval is the first layer to check.
- If the passage never arrived, no prompt or model change can fix it.
- Log retrieved passages with each answer.
Use case
An illustrative scenario, not a real incident.
A support bot gets a refund rule wrong, although the rule is in the knowledge base. Logs show the passage was never retrieved, so the fix is in search, not the prompt.
See it in different ways
flowchart TD
A["Bad answer"] --> B{"Was the right passage retrieved?"}
B -->|No| C["Retrieval problem"]
B -->|Yes| D{"Does a clearer prompt fix it?"}
D -->|Yes| E["Prompt problem"]
D -->|No| F["Model mismatch"]stateDiagram-v2 [*] --> Retrieval Retrieval --> FixRetrieval: passage missing Retrieval --> Prompt: passage present Prompt --> Model: prompt is clear Model --> Labels: still wrong Labels --> [*]: check the test itself
flowchart TD
Q{"Was the right passage retrieved?"}
Q -->|"No"| A0["Fix retrieval"]
Q -->|"Yes"| A1["Look at the prompt and model"]Key point: Check retrieval first.
Why the other options are wrong
- A. Whether users are phrasing their questions in the expected way
- Phrasing may matter, but the retrieved context shows that directly.
- C. Whether a more capable model would produce a better answer here
- A stronger model cannot read a passage it never received.
- D. Whether the system prompt should be rewritten with stricter wording
- Prompt changes should wait until retrieval is confirmed.