Which is a quantitative signal for evaluating agent work?
Why C is the answer
The pass rate of tests on the agent's pull requests
Test pass rate is measured. Reviewer opinion is a useful qualitative signal, and self-reported confidence is weak.
In detail
- A test pass rate is a number you can track.
- Track the number over time.
Use case
An illustrative scenario, not a real incident.
A team tracks how often tests pass on agent pull requests, and it sees a clear rise after improving instructions. Opinions could not show that.
See it in different ways
flowchart LR A["Agent output"] --> B["Tests and code scanning"] A --> C["Reviewer feedback"] B --> D["Evaluation"] C --> D D --> E["Tune the agent"]
sequenceDiagram participant T as Team participant A as Agent participant R as Results T->>T: Set success criteria A->>R: Several runs, across tasks R-->>T: Tests, scans and review signals T->>A: Keep or drop the change
flowchart TD
Q{"What kind of signal?"}
Q -->|"Test pass rate"| A0["Quantitative"]
Q -->|"It feels clean"| A1["Subjective"]Key point: Test pass rate is a number you can track.
Why the other options are wrong
- A. Reviewers say that the code feels clean to them
- That is an opinion.
- B. A teammate describes the pull request as pleasant to read
- That is also an opinion.
- D. The agent reports that it is confident in its work
- Self-reported confidence is not a measurement.