Where should evaluation run so that regressions are caught before release?
Why B is the answer
In the CI/CD pipeline, on each change
Running evaluations in the pipeline makes quality a release gate.
In detail
- Running on each change catches regressions before release.
- Set thresholds that fail the pipeline.
Use case
An illustrative scenario, not a real incident.
Each change runs evaluators in CI/CD, so a drop in quality blocks the release before users see it.
See it in different ways
flowchart LR A["Test dataset"] --> B["Map columns to evaluator inputs"] B --> C["Quality and safety evaluators"] C --> D["Compare with the baseline"] D --> E["Release decision"]
sequenceDiagram participant C as Change participant P as CI/CD pipeline participant R as Production participant T as Traces C->>P: Runs evaluators on a test dataset P->>R: Releases if it passes R->>T: Records real traffic T->>P: New failing cases
flowchart TD
Q{"Do scores pass?"}
Q -->|"Yes"| A0["Release"]
Q -->|"No"| A1["Block it"]Key point: Evaluate on every change.
Why the other options are wrong
- A. On the production database directly
- Evaluation does not belong on a production database.
- C. Once a year, in a manual review
- Yearly reviews are too slow.
- D. Only after users complain
- That finds problems too late.