Free-form generation is hard to measure. Production LLM systems combine deterministic checks (schema, regex, exact match), LLM-as-judge for quality, online proxy metrics (CSAT, retention), and component-level tests (retrieval recall, faithfulness) to detect quality regressions early.
Eval-as-CI pipeline: every prompt or model change runs goldens before merge.
Every production failure becomes a new golden. Over time your eval becomes uniquely valuable IP.
Validate LLM-judge agreement with human raters on a sample (Cohen's κ ≥ 0.6). Otherwise you optimize for the judge's biases.
Aggregate metrics hide regressions in minority segments. Always slice by intent, locale, user tier, doc type.
CI gates with hard thresholds prevent quiet quality erosion. Treat eval like unit tests.
Offline goldens catch regressions early; online metrics (CSAT, completion rate) catch what goldens missed. You need both.
Quality metrics alone are incomplete. Track tokens/req, P95 latency, $/req per slice — quality wins that 10x cost may not ship.
Sign in to share your feedback and join the discussion.