After launch, quality slowly declines as products and policies change. What should be in place?
Why C is the answer
A monitoring loop that feeds failing cases back into the eval set
Production changes under you. Feeding real failures into evals turns drift into a routine fix.
In detail
- Production changes under you.
- Feeding real failures into evals turns drift into a routine fix.
- Sample production answers on a regular schedule.
Use case
An illustrative scenario, not a real incident.
Policies change, and quality slips over months. Failing cases flow into the eval set from monitoring, so each drift becomes a test and a fix.
See it in different ways
flowchart LR A["Monitor production"] --> B["Find failing cases"] B --> C["Add them to the eval set"] C --> D["Fix and retest"] D --> A
sequenceDiagram participant E as Eval suite participant T as Limited test participant P as Production E->>T: Promising version T->>T: Compare a real outcome T->>P: Roll out, or keep the pinned version
stateDiagram-v2 state "Production" as s0 state "Monitoring" as s1 state "Eval set" as s2 state "Fix and release" as s3 [*] --> s0 s0 --> s1: sample answers s1 --> s2: add failing cases s2 --> s3 s3 --> s0
Key point: Turn real failures into evals.
Why the other options are wrong
- A. A plan to rebuild the whole system from scratch every year
- A yearly rebuild is costly, and it reacts slowly.
- B. A yearly review carried out by a team outside the project
- A yearly review misses months of decline.
- D. A larger model, used to keep the error rate from growing
- A larger model does not know that a policy changed.