Production-time evaluation — user feedback, A/B testing, and real-time quality monitoring for LLM outputs.
Five passes over the same idea, each from a different angle. Do them in order, or jump to whichever you need.
Online evaluations assess LLM quality in production. Methods: implicit feedback (thumbs up/down, regeneration rate), A/B testing between model versions, human evaluation sampling, real-time LLM-as-judge on production traffic, and quality score dashboards. Essential for continuous improvement.
