Offline Evals: The Regression Suite for LLM Features

The Test Suite for Behaviour You Cannot Unit Test

An offline evaluation runs a fixed set of inputs through your system before release and scores the outputs. It is the nearest thing an LLM feature has to a regression suite: the check that tells you whether today's prompt edit, model upgrade or retrieval change made things better or worse, before users find out. Online evaluation watches real traffic after release. The two complement each other, but only the offline suite can stop a bad change from shipping.

Build the Dataset From Real Use

The dataset decides what the evaluation can see. Examples invented at a desk test what you imagined users would ask. Examples taken from production test what they do ask.

Start with fifty to a hundred examples. That is enough to see large regressions, and small enough that you will read every failure. Add a case each time production shows you a new kind of mistake. Tag each example with a category, because the breakdown by category is where the useful signal is.

Use the Cheapest Grader That Works

1. Code checks. Exact match, a required field, valid JSON, a number within a range, a forbidden phrase. These are fast, free and repeatable. Use them wherever the criterion can be written as code.
2. Model graders. For qualities that code cannot check, such as whether an answer is supported by the source or whether the tone is appropriate. Give the grader a specific rubric and ask for a short reason before the verdict.
3. Human review. For calibrating the model grader and for cases where the criteria are still being worked out. Review a sample regularly, not everything.

A model grader is a classifier you have built, and it needs its own test. Label a few dozen outputs yourself, run the grader over them, and measure how often it agrees. If agreement is poor, improve the rubric before you trust the scores.

A Minimal Harness

import json
from collections import defaultdict

def run_suite(system, cases, graders):
    results = defaultdict(list)
    for case in cases:
        output = system(case["input"])
        passed = all(grade(case, output) for grade in graders[case["category"]])
        results[case["category"]].append(passed)
    return {cat: sum(r) / len(r) for cat, r in results.items()}

cases = [json.loads(line) for line in open("evals/support.jsonl")]
scores = run_suite(candidate_system, cases, GRADERS)
baseline = json.load(open("evals/baseline.json"))

regressions = {
    cat: (baseline[cat], score)
    for cat, score in scores.items()
    if score < baseline[cat] - 0.03          # tolerance for run-to-run noise
}
if regressions:
    raise SystemExit(f"Regression in: {regressions}")

Noise: Know How Much a Difference Means

LLM outputs vary between runs, and current reasoning models cannot be pinned with a temperature of zero. Model graders vary too. A two-point change on a hundred examples may be nothing. Before you rely on the suite, measure its noise.

• Run the unchanged system several times. The spread between those runs is your noise floor. A difference smaller than that is not a result.
• Compare case by case. Looking at which examples flipped from pass to fail tells you more than two averages do.
• Repeat unstable cases. Run each example three or five times and score the pass rate.
• Fix what you can. Pin the model version, the grader model and the reasoning effort, and record them with every run.

Make It a Gate

An evaluation that is run by hand is run when someone remembers. Put it in the pipeline: run the suite on every change to a prompt, a model setting or the retrieval code, compare each category with the stored baseline, and block the merge when a category falls by more than the noise tolerance. Update the baseline only as a deliberate, reviewed step.

Cost is a practical limit. Run a small fast subset on every pull request and the full suite before release and on a schedule. Use batch pricing and prompt caching for the full run, since it does not need an immediate answer.

Key Takeaways