How should success criteria relate to evaluation?
Why C is the answer
They should reflect the intended outcome and be set in advance
Criteria set in advance keep evaluation honest and aligned with development intent.
In detail
- Criteria should reflect the intended outcome.
- Setting them in advance keeps the judgment honest.
- Tailor the criteria to each kind of task.
Use case
An illustrative scenario, not a real incident.
A team sets success criteria before a run, including tests and limits. Judging afterward against those criteria keeps the result honest.
See it in different ways
flowchart LR A["Agent output"] --> B["Tests and code scanning"] A --> C["Reviewer feedback"] B --> D["Evaluation"] C --> D D --> E["Tune the agent"]
sequenceDiagram participant T as Team participant A as Agent participant R as Results T->>T: Set success criteria A->>R: Several runs, across tasks R-->>T: Tests, scans and review signals T->>A: Keep or drop the change
flowchart LR S0["Set criteria"] --> S1["Run the agent"] --> S2["Compare results"] --> S3["Decide"]
Key point: Set criteria in advance.
Why the other options are wrong
- A. They should be kept private from the people reviewing the work
- Reviewers need to see them.
- B. They should be written after the results, to match what happened
- Writing them afterward fits them to the result.
- D. They should be the same for every kind of task the team has
- Different tasks need different criteria.