The right passage was retrieved, the prompt is clear, and a stronger model still gets it wrong. What is the likely issue?
Why A is the answer
The task is ambiguous or the source data is wrong, so check the labels
When every layer checks out, look at the test itself. Ambiguous cases and bad labels are common causes of stubborn failures.
In detail
- Every layer checks out, so look at the test itself.
- Ambiguous cases and bad labels are common causes of stubborn failures.
- Have two people review stubborn failures.
Use case
An illustrative scenario, not a real incident.
The passage is present, the prompt is clear, and a stronger model still fails. Reviewers find the test label is wrong, so the model was right.
See it in different ways
flowchart TD
A["Bad answer"] --> B{"Was the right passage retrieved?"}
B -->|No| C["Retrieval problem"]
B -->|Yes| D{"Does a clearer prompt fix it?"}
D -->|Yes| E["Prompt problem"]
D -->|No| F["Model mismatch"]stateDiagram-v2 [*] --> Retrieval Retrieval --> FixRetrieval: passage missing Retrieval --> Prompt: passage present Prompt --> Model: prompt is clear Model --> Labels: still wrong Labels --> [*]: check the test itself
flowchart LR S0["Retrieval"] --> S1["Prompt"] --> S2["Model"] --> S3["Test data and labels"]
Key point: When every layer checks out, check the test.
Why the other options are wrong
- B. The temperature is too low for this type of reasoning task
- Temperature is rarely why a clear reasoning task fails.
- C. The vector index needs rebuilding because retrieval has stopped working
- The passage was retrieved, so retrieval is working.
- D. The prompt needs one more example to reach the required accuracy
- The prompt is already clear, and one more example is a guess.