Start With Offline Evals
Before launch there's no traffic to evaluate, and a fixed test set in CI is the right tool. Keep it. Online evals are what you add once real users arrive, because they ask things your test set never imagined.
Offline evals (a fixed test set in CI) catch the regressions you can imagine. Online evals catch the regressions only your real users generate: surprise topic mixes, new languages, abusive inputs, and slow degradations from prompt drift. Without online evals, your first signal of quality decay is a customer complaint — and by then the bad responses may be weeks old.
- You ship a prompt change, CI is green, users complain three days later
- You have no number for "is the answer actually good?" in production
- Your only quality signal is thumbs-down clicks, which lag by hours and bias toward angry users
The Three Online Eval Modes
- Reference-free heuristics — schema valid, language detect, length plausible, citation present. Fast, cheap, run on 100%.
- LLM-as-judge — a separate model scores faithfulness, relevance, harmfulness against a rubric. Run on 1–5% of traffic.
- Implicit user signals — click-through, copy event, follow-up question, abandonment. Free, noisy, lagging.
flowchart LR
P[Production request] --> R[Response]
R --> H[Heuristics: schema/language/length]
H --> S{Sample 1–5%}
S -- yes --> J[LLM-as-judge:<br/>faithfulness, relevance, harm]
J --> M[Metric: eval.score]
R --> UI[User UI]
UI --> IM[Implicit signals:<br/>copy, follow-up, abandon]
IM --> M
Check it: List what you could check on every response with code alone: a valid schema, the expected language, a plausible length, a citation present. Nothing on that list should be costing you a judge call.
LLM-as-Judge: The Three Rubrics That Matter
- Faithfulness — does the answer only state things supported by the retrieved context? Critical for RAG.
- Relevance — does the answer address the question that was asked? Catches off-topic drift.
- Safety/policy — does the response violate your content rules? Complements your guardrails.
Use a smaller, cheaper model as the judge (gpt-6-luna, claude-haiku) with a strict rubric and a 1–5 score. Calibrate the judge against a human-labelled set before trusting it — judges drift too.
Rubric: Faithfulness (1–5)
5 Every claim is directly supported by the context.
4 All key claims supported; minor unsupported detail.
3 Mix of supported and unsupported claims.
2 Mostly unsupported; one or two grounded claims.
1 Hallucinated; nothing in the context.
Return JSON: { "score": int, "unsupported_claims": [string] }Check it: Have a person score thirty sampled answers with the same rubric, without seeing the judge's scores, and compare the two. Where they disagree, read the rubric before you blame the judge.
Implicit Signals: Treat as Hints, Not Truth
Implicit signals are great as early-warning indicators but terrible as ground truth (users abandon for many reasons unrelated to quality). Useful pairings:
- Drop in "copy answer" events on the support widget → answers got less actionable.
- Spike in "ask again" rate → answers got less clear.
- Increase in average follow-up turn count → first-turn answer is missing context.
Sampling Strategy
- Run heuristic checks on 100% — cost is essentially zero.
- LLM-as-judge on 1–5% steady-state, ramped to 100% on a deploy for 24h.
- Always evaluate all negative-feedback responses (free training set for offline evals).
- Always evaluate a stratified sample (per tenant, per language, per endpoint) so small segments aren't invisible.
Check it: Count last week's judged answers for your smallest tenant and your least common language. If either is zero, the sample isn't stratified.
Closing the Loop
Online evals only matter if they trigger action. Wire them into:
- An alert when 7-day faithfulness drops > 5%.
- A nightly export of low-scoring responses into your offline eval set (after PII review).
- A deploy gate that runs a 30-minute canary at 100% eval sampling on each release.
Check it: Follow one low-scoring answer from last week to what happened next. It should have raised an alert or become a test case. If it did neither, the scores are only a chart.
Failure Modes
- Judge model upgrade silently shifts scores. Pin the judge model version and the rubric version.
- Sampling bias — sampling only successful requests hides the failures you most need to evaluate.
- Eval drift — the rubric stops matching what the product cares about. Re-validate against human labels quarterly.
- Privacy — judge prompts include user inputs. Send them to the same vendor/region as your primary LLM, not a third party.
Try It on Your Own System This Week
Three checks on a live feature, about an hour in total:
- Read twenty answers that got a thumbs-down. They're free test cases. Add the real failures to your offline set.
- Check which model the judge is. If it isn't pinned, your quality chart can move without your product changing.
- Find where judged data goes. Judge prompts contain user input. Confirm it stays with a vendor and a region your terms cover.
Production Checklist
- 100% heuristic checks; 1–5% LLM-as-judge; 100% on negative-feedback samples.
- Pinned judge model + versioned rubric + human-calibrated thresholds.
- Eval scores emitted as metrics with the same dimensions as cost/latency.
- Alerts and deploy gates wired to eval drift.
Catching Silent Regressions
Offline evals protect you from what you can predict. Online evals catch what you can't, and the failure they exist for is a quiet one. A vendor updates a model behind an alias, a document store goes stale, or the mix of user questions shifts. Nothing errors, and quality just slides.
Pin your model versions where you can, so changes are at least deliberate. Then watch the score distribution, not only the average. An average can hold steady while one category of questions collapses, so break it down by intent, language and customer tier. Alert on a sustained drop against a rolling baseline, since single-day swings are often noise. A useful alert fires on something like "faithfulness down five points for three days on billing questions".
Judges drift too
An LLM judge is a model, so it has all the same problems. Upgrade the judge and your scores can shift overnight with no real change in quality. Keep the judge pinned, and version it alongside its prompt. When you do change it, score the same sample with the old and new judge, and record the offset, or your trend charts will show a step nobody can explain.
Calibrate judges against people. Each month, have a person label 50 to 100 sampled responses, and compare. If agreement falls below your bar, the judge is measuring something else. Keep that human sample going even when the dashboards look healthy, because a judge can only flag what it was written to look for.
Evaluating agents online
For agents, score the run as well as the reply. Did the task finish? How many turns and how much did it cost? Did a person have to step in? Did the user retry or abandon? Task completion and escalation rate are often more honest than any quality score. Keep an eye on tool failures per run, too, because agents can hide tool problems by working around them, and the answer still looks fine while costs climb.
Privacy and cost
Sending production traffic to a judge means sending user data to another model, possibly with another vendor, so check that your data processing terms cover it. Redact personal data first where you can. On cost, a small sample is often enough to detect drift on busy routes. Run judges on the batch API at half price if you can accept a delay of up to a day, since a daily score doesn't need to be real time.
Key Takeaways
- Online evals are the quality counterpart to cost telemetry — without them you ship blind.
- Heuristics + LLM-as-judge + implicit signals form a layered, affordable quality pulse.
- Wire evals into alerts and deploy gates or they are just dashboards.