What do you need before running evaluators on an app?
Why C is the answer
A test dataset, mapped to the evaluator inputs
Evaluators read named inputs, so you need a test dataset with its columns mapped to those inputs.
In detail
- Evaluators need a dataset, with fields mapped to their inputs.
- Include hard cases.
Use case
An illustrative scenario, not a real incident.
A team prepares a test dataset and maps its columns to evaluator inputs such as query and response.
See it in different ways
flowchart LR A["Test dataset"] --> B["Map columns to evaluator inputs"] B --> C["Quality and safety evaluators"] C --> D["Compare with the baseline"] D --> E["Release decision"]
sequenceDiagram participant C as Change participant P as CI/CD pipeline participant R as Production participant T as Traces C->>P: Runs evaluators on a test dataset P->>R: Releases if it passes R->>T: Records real traffic T->>P: New failing cases
flowchart LR S0["Test dataset"] --> S1["Map columns"] --> S2["Run evaluators"] --> S3["Review scores"]
Key point: Evaluators need a mapped dataset.
Why the other options are wrong
- A. A provisioned deployment in each of your regions
- Provisioned capacity is not required.
- B. A private network link to the Foundry portal
- A private link is not required.
- D. A new model trained on your production traffic
- No new model is required.