Quality falls gradually in production, though tests still pass. What is missing?
Why D is the answer
Evaluation of production traces
Foundry can evaluate production traces. Test sets do not show how real traffic changes.
In detail
- Static tests cannot see changing traffic.
- Evaluate production traces too.
- Add failing traces to the test set.
Use case
An illustrative scenario, not a real incident.
Tests pass, but real quality falls as questions change. Evaluating production traces shows what the static tests cannot.
See it in different ways
flowchart LR A["Request"] --> B["Retrieval span"] B --> C["Model span"] C --> D["Tool span"] D --> E["Response"]
sequenceDiagram participant C as Change participant P as CI/CD pipeline participant R as Production participant T as Traces C->>P: Runs evaluators on a test dataset P->>R: Releases if it passes R->>T: Records real traffic T->>P: New failing cases
stateDiagram-v2 state "Production traces" as s0 state "Evaluate samples" as s1 state "Update test set" as s2 [*] --> s0 s0 --> s1 s1 --> s2 s2 --> s0
Key point: Evaluate production traces too.
Why the other options are wrong
- A. A longer system prompt for each request
- A longer prompt is a guess.
- B. A second copy of the test set
- A copy of the same test finds nothing new.
- C. A new name for the deployment
- A name has no effect.