Evaluating agents is harder than evaluating single LLM calls because agents take multiple steps, use tools, and can succeed with different approaches. Key metrics: task success rate (did it achieve the goal?), trajectory correctness (did it take the right steps?), efficiency (tokens and steps used), and reliability (success rate variance). Evaluation frameworks: AgentBench, WebArena, GAIA, and custom evals. LLM-as-judge evaluates intermediate steps and final output quality.
Metrics for comprehensive agent performance monitoring.
If you cannot define what success looks like for a task, you cannot evaluate it, and you cannot know if a code change improved or degraded agent performance.
A vague prompt ("rate this response 1-10") produces noisy scores. A rubric with specific dimensions (correctness, efficiency, safety, relevance) and anchor examples produces reliable, actionable scores.
An agent that succeeds in 20 steps when 5 would suffice is expensive and slow. Trajectory efficiency (optimal_steps / actual_steps) is a key metric alongside success rate.
Sign in to share your feedback and join the discussion.