On this page
The problem
Traditional unit and integration tests are insufficient for AI features because LLM outputs are non-deterministic. We need a testing strategy that gives us confidence in AI feature quality without requiring expensive human review of every change.
What we weighed
Manual Testing Only
Pros
- +Simple — no test infrastructure required
- +Captures nuanced quality issues
- +Appropriate for early prototypes
Cons
- −Does not scale — every prompt change requires manual re-review
- −No regression detection for prompt or model changes
- −Quality is subjective and inconsistent across reviewers
Automated Evals with LLM-as-Judge
Pros
- +Scalable — evaluate hundreds of test cases in minutes
- +Runs in CI/CD — blocks regressions automatically
- +LLM judge scores quality dimensions (faithfulness, relevance, safety) systematically
Cons
- −LLM judge has its own biases and is not perfectly calibrated to human preferences
- −Requires investing time in building good eval datasets
- −Cost: eval suite run = N test cases × evaluator LLM cost
Snapshot Testing for Prompts
Pros
- +Catches unintended prompt changes
- +Fast — no LLM calls required for snapshot verification
- +Familiar pattern from UI testing
Cons
- −Frozen snapshots do not test quality — only that output is unchanged
- −Brittle — non-determinism causes snapshot failures
Hybrid: Unit + Evals + Production Sampling
Pros
- +Unit tests for deterministic components (prompt templates, parsers, validators)
- +Evals for output quality dimensions
- +Production sampling for real-world quality monitoring
Cons
- −Most complex to set up — three test types to maintain
- −Requires eval dataset curation investment upfront
What we chose
We adopt the Hybrid approach. (1) Standard unit tests for all non-LLM components — prompt template rendering, response parsers, guardrail logic, schema validation. (2) LLM-as-judge eval suites (RAGAS + DeepEval) run in CI for every prompt or model change, with quality thresholds enforced. (3) 5% production sampling with Phoenix online evals for continuous quality monitoring.
What it committed us to
- •Create eval dataset for each AI feature — minimum 20 test cases per feature
- •Integrate DeepEval or RAGAS into CI pipeline with automated quality gates
- •Define per-feature quality thresholds (e.g., RAG faithfulness > 0.85, relevance > 0.80)
- •Log 5% of production LLM calls to Phoenix for online evaluation
- •Set up weekly AI quality review process to review flagged responses