Online evaluation measures LLM quality on real production traffic using implicit signals (user engagement, thumbs up/down, follow-up queries, task completion) and explicit automated probes (shadow LLM judges scoring live responses). A/B experiments compare model, prompt, or retrieval variants on a traffic split. Online evals detect distribution shift, quality degradation, and unexpected failure modes that offline datasets cannot simulate.
Each stage in order — click any step to read what it does.
Production LLM evaluation pipeline.
The trade-offs worth knowing before you build this.
Production traffic has infinite diversity. Your 200-sample golden dataset can't represent all real user inputs. Online eval scores real traffic — including the long tail of unexpected inputs that offline datasets miss.
A prompt variant that looks better after 50 requests might be statistical noise. Run until you have 500+ samples per variant and p < 0.05. Premature graduation based on small samples causes regressions.
When production traffic discovers a failure mode, capture it as a golden dataset sample. Over time, your offline dataset grows to cover real failure patterns. Self-improving evaluation system.
Sign in to share your feedback and join the discussion.
