Evals for Choosing a Model

The Only Benchmark That Matters Is Yours

In September 2026 OpenAI, Anthropic and Google each released new models within the same four weeks. Every release came with charts showing it ahead of the others. Those charts cannot all describe your workload, and none of them was measured on it. An eval is how you answer the question the leaderboards cannot: which model, at which settings, does your task best for the least money.

Why Public Benchmarks Do Not Transfer

• Saturation. Once the leading models all score above 90% on a benchmark, the remaining differences are noise. Several well-known benchmarks are in this state.
• Contamination. Public test questions end up in training data. A high score can reflect memory as much as ability.
• Distribution.A benchmark measures a model on someone else's tasks, prompts and grading. Your documents, your instructions and your definition of a good answer are different.
• Settings. Published scores are usually taken at high reasoning effort with long outputs. You may be running at low effort with a latency budget.

Benchmarks are still useful as a filter. They tell you which three or four models are worth testing. They do not tell you which one to ship.

Building a Model-Selection Eval

1. Collect representative tasks. Fifty to two hundred real inputs from your application, covering the common cases and the difficult ones.
2. Define success for each task. Use a code check where possible, such as correct label, valid schema or passing test. Use a rubric scored by a model where it is not.
3. Adapt the prompt for each model. A prompt tuned for one family often underperforms on another. Give each candidate a fair prompt, then compare best against best.
4. Sweep the settings. Run each candidate at two or three effort levels. A cheaper model at higher effort, or a stronger model at lower effort, frequently beats the obvious configuration.
5. Record more than accuracy. Capture tokens, cost and latency for every run, so the comparison is about the whole trade-off.

Measure Cost per Successful Task

Price per token is the wrong unit for comparing models. A cheaper model that needs a retry, a longer prompt or a second pass can cost more for each finished task than an expensive model that succeeds first time.

def summarise(runs):
    """runs: one dict per attempt, with keys passed, cost_usd, latency_s."""
    successes = sum(r["passed"] for r in runs)
    total_cost = sum(r["cost_usd"] for r in runs)
    latencies = sorted(r["latency_s"] for r in runs)
    return {
        "pass_rate": successes / len(runs),
        "cost_per_success": total_cost / max(successes, 1),
        "p95_latency_s": latencies[int(len(latencies) * 0.95) - 1],
    }

Keep the Harness Ready

The value of a selection eval is that you can run it again. With a harness that takes a model id and a settings file, assessing a new release is an afternoon and not a project. That matters because models are retired as well as released: when a provider announces an end date for the model you use, the eval tells you which replacement is safe and what the prompt needs.

Re-run it on a schedule as well, about once a quarter and whenever a provider you depend on ships something new. Keep the results in a table over time. The trend shows whether a change is worth the migration effort.

Key Takeaways