Why is final-answer accuracy not enough for agents?
Why B is the answer
An agent can reach it through wasteful or unsafe steps
Tool use, safety and cost all matter as well.
In detail
- Tool use, safety and cost all matter.
- Check the path as well as the answer.
Use case
An illustrative scenario, not a real incident.
An agent gets the right answer after 40 needless calls and one unsafe action.
See it in different ways
flowchart LR A["Scenario set"] --> B["Run agent"] B --> C["Score task success"] B --> D["Score tool choices"] B --> E["Measure cost and latency"] C --> F["Compare with baseline"] D --> F E --> F
mindmap
root((Agent evaluation))
Outcome
Task success
Path
Tool accuracy
Loops
Cost
Tokens
Latency
Safety
Regression setmindmap
root(("Agent quality"))
n1["Result"]
n2["Final answer"]
n3["Path"]
n4["Tool use"]
n5["Safety"]
n6["Cost"]Key point: The path matters as well as the answer.
Why the other options are wrong
- A. Agents do not produce a final answer at any stage
- Agents do give answers.
- C. Final answers can never be scored by any method
- Answers can be scored.
- D. Accuracy is measured only during model pretraining
- Accuracy is measured after training too.