Online evaluation measures LLM quality on real production traffic using implicit signals (user engagement, thumbs up/down, follow-up queries, task completion) and explicit automated probes (shadow LLM judges scoring live responses). A/B experiments compare model, prompt, or retrieval variants on a traffic split. Online evals detect distribution shift, quality degradation, and unexpected failure modes that offline datasets cannot simulate.
Traffic split for comparing LLM pipeline variants.
Production traffic has infinite diversity. Your 200-sample golden dataset can't represent all real user inputs. Online eval scores real traffic — including the long tail of unexpected inputs that offline datasets miss.
A prompt variant that looks better after 50 requests might be statistical noise. Run until you have 500+ samples per variant and p < 0.05. Premature graduation based on small samples causes regressions.
When production traffic discovers a failure mode, capture it as a golden dataset sample. Over time, your offline dataset grows to cover real failure patterns. Self-improving evaluation system.
Sign in to share your feedback and join the discussion.