How do you judge whether an agent did its job?
Why C is the answer
Task completion rates and efficiency measures
Task completion rates and efficiency measures show if an agent did its job, and at what effort.
In detail
- Task completion and efficiency measure what the agent achieved.
- Track failure cases too.
Use case
An illustrative scenario, not a real incident.
A booking agent is measured on trips booked correctly, and on the steps it took.
See it in different ways
flowchart TD A["GenAI evaluation"] --> B["Retrieval: did it find the right passages?"] A --> C["Model: accuracy and relevance"] A --> D["Agent: task completion"] A --> E["Users: feedback"]
stateDiagram-v2 [*] --> InspectLogs InspectLogs --> Retrieval: wrong context InspectLogs --> Prompt: right context, wrong answer Retrieval --> Evaluate Prompt --> Evaluate Evaluate --> [*]
mindmap
root(("Agent evaluation"))
n1["Outcome"]
n2["Task completion rate"]
n3["Effort"]
n4["Efficiency measures"]Key point: Judge agents on outcomes.
Why the other options are wrong
- A. The number of tools it has been given
- Tool count is not a result.
- B. Whether it reports that it is confident
- Its confidence is not evidence.
- D. The length of its final message
- Length does not measure success.