You want to know whether a new agent version is better. What should you measure?
Why D is the answer
Task success on a fixed scenario set, plus tool accuracy and cost
Agents need task-level evaluation across a fixed set. Single metrics hide trade-offs.
In detail
- Measure task success on fixed scenarios, plus tool accuracy and cost.
- Track safety as well.
Use case
An illustrative scenario, not a real incident.
A new version solves more tasks but costs twice as much. A fixed scenario set shows both.
See it in different ways
flowchart LR A["Scenario set"] --> B["Run agent"] B --> C["Score task success"] B --> D["Score tool choices"] B --> E["Measure cost and latency"] C --> F["Compare with baseline"] D --> F E --> F
mindmap
root((Agent evaluation))
Outcome
Task success
Path
Tool accuracy
Loops
Cost
Tokens
Latency
Safety
Regression setmindmap
root(("Agent evaluation"))
n1["Outcome"]
n2["Task success"]
n3["Process"]
n4["Tool accuracy"]
n5["Spend"]
n6["Cost"]Key point: Measure task success, tools and cost.
Why the other options are wrong
- A. The number of tokens used per task, compared across the two versions
- Tokens alone do not show quality.
- B. Developer opinion after a demo of each agent version
- Opinion is not a measurement.
- C. The size of the system prompt and the number of tools
- Size shows nothing about outcomes.