What should a production failure turn into?
Why A is the answer
A new evaluation case
This stops the same failure from returning.
In detail
- A new evaluation case stops the same failure returning.
- Keep the real input.
Use case
An illustrative scenario, not a real incident.
A production failure becomes an evaluation case, so it cannot quietly return.
See it in different ways
flowchart LR A["Agent run"] --> B["Trace of each step"] B --> C["Dashboards and alerts"] C --> D["Root cause"] D --> E["Add a test and fix"]
stateDiagram-v2 [*] --> Running Running --> Incident Incident --> ReadTrace ReadTrace --> Fix Fix --> AddEvalCase AddEvalCase --> StagedRollout StagedRollout --> Running
flowchart LR S0["Failure"] --> S1["Evaluation case"] --> S2["Fix"] --> S3["Case passes"]
Key point: Turn failures into tests.
Why the other options are wrong
- B. A larger system prompt
- Length is not a fix.
- C. A closed ticket with no notes
- No notes means no learning.
- D. A new model name
- A name changes nothing.