How do you keep quality from regressing as prompts change?
Why D is the answer
Continuous evaluation against a test set
Continuous evaluation against a test set catches quality regressions whenever prompts change.
In detail
- Continuous evaluation catches regressions on each change.
- Add real failures to the set.
Use case
An illustrative scenario, not a real incident.
Each prompt change runs against a test set, and a score drop blocks the release.
See it in different ways
flowchart TD A["GenAI evaluation"] --> B["Retrieval: did it find the right passages?"] A --> C["Model: accuracy and relevance"] A --> D["Agent: task completion"] A --> E["Users: feedback"]
stateDiagram-v2 [*] --> InspectLogs InspectLogs --> Retrieval: wrong context InspectLogs --> Prompt: right context, wrong answer Retrieval --> Evaluate Prompt --> Evaluate Evaluate --> [*]
stateDiagram-v2 state "Change prompt" as s0 state "Run test set" as s1 state "Release" as s2 state "Fix" as s3 [*] --> s0 s0 --> s1 s1 --> s2: scores hold s1 --> s3: scores fall s3 --> s1 s2 --> [*]
Key point: Evaluate continuously.
Why the other options are wrong
- A. A review once a year
- Yearly is too slow.
- B. A rule against changing prompts
- A ban stops all improvement.
- C. A larger knowledge base
- More data does not check prompts.