Results vary between runs of the same task. How do you judge a tuning change?
Why B is the answer
Compare it across several runs and tasks against fixed criteria
Agent output varies, so one run proves little. Repeat against the same criteria.
In detail
- One run can mislead when results vary.
- Compare several runs and tasks against fixed criteria.
- Keep the criteria fixed while you compare.
Use case
An illustrative scenario, not a real incident.
A change looks better on one run and worse on the next. Comparing ten runs across several tasks shows whether it truly helped.
See it in different ways
sequenceDiagram participant T as Team participant A as Agent participant R as Results T->>T: Set success criteria A->>R: Several runs, across tasks R-->>T: Tests, scans and review signals T->>A: Keep or drop the change
flowchart TD
Q{"How much evidence?"}
Q -->|"One run"| A0["May be luck"]
Q -->|"Several runs and tasks"| A1["Reliable comparison"]Key point: Judge changes across many runs.
Why the other options are wrong
- A. Ask the agent whether it believes that the change has helped
- The agent's opinion is not evidence.
- C. Keep the change if the output of the new run is longer than before
- Longer output does not mean better output.
- D. Run it once, and keep the change if that one run looks better
- One run may be luck.