Start With One Sensible Default
For one low-stakes use, pick a sensible mid-tier model and get on with it. Comparison work pays for itself when a pipeline runs at volume, when quality matters, or when someone wants to switch because of a leaderboard.
"Which model should we use?" comes up in every team building with LLMs, and the usual answer is a screenshot of a leaderboard. The leaderboard measures someone else's tasks. Your code reviews, debugging sessions and architecture questions are not on it.
Claude, GPT and Gemini differ, but the differences show up task by task, not as one overall ranking. Below are the facts the vendors publish (context windows, prices, documented features and their own benchmark numbers), then my starting routing policy, labelled as opinion, and a harness for checking it on your own tasks before you commit.
- Your team defaults to GPT-6.1 Sol or Claude Sonnet 5.5 for everything without evaluating alternatives
- You've tried Claude once, got a different result, and aren't sure which was better
- You need to pick a model for a production pipeline and want a data-driven decision
- You're paying for a frontier model on tasks where a cheaper model would work just as well
What Benchmarks Measure, and What They Miss
Comparing LLMs requires defining what you're comparing on. Benchmark scores (MMLU, HumanEval) tell you about general capability, not about performance on your specific tasks. What matters in practice:
flowchart TD
T["Your Task"] --> E["Evaluation Criteria"]
E --> A["Accuracy\n(correct output)"]
E --> F["Instruction Following\n(format compliance)"]
E --> C["Context Handling\n(long input performance)"]
E --> L["Latency\n(time to first token)"]
E --> P["Price\n(cost per million tokens)"]
style T fill:#4f46e5,color:#fff,stroke:#4338ca
style A fill:#059669,color:#fff,stroke:#047857
style F fill:#059669,color:#fff,stroke:#047857
style C fill:#059669,color:#fff,stroke:#047857
style L fill:#d97706,color:#fff,stroke:#b45309
style P fill:#d97706,color:#fff,stroke:#b45309
What the Vendors Publish
These figures come from each vendor's model and pricing pages, checked on 2026-10-31. Prices are standard (non-batch) USD per million input / output tokens.
| Model | Vendor positioning | Context | Max output | Price (in / out) |
|---|---|---|---|---|
| Claude Fable 5.2 | Most capable widely released Claude model | 1M | 128K | $8 / $40 |
| Claude Opus 5.5 | Long-running agentic coding and knowledge work | 1M | 128K | $4 / $20 |
| Claude Sonnet 5.5 | Speed and intelligence | 1M | 128K | $2 / $10 |
| Claude Haiku 5.5 | High-volume, latency-sensitive work | 1M | 128K | $0.10 / $0.50 (prompts up to 100K tokens; $0.50 / $2.50 above) |
| GPT-6 Astra | OpenAI's most capable model | 1.05M (922K input) | 128K | $10 / $50 |
| GPT-6.1 Sol | Near-Astra performance at lower cost | 1.05M (922K input) | 128K | $2 / $10 |
| GPT-6 Luna | Focused, high-volume tasks | 1.05M (922K input) | 128K | $0.10 / $0.50 |
| Gemini 4 Argon | Google's flagship reasoning model | 1,048,576 (input and output combined) | 65,536 (up to 1M with long decode) | $2 / $10 through 2026-11-30, then $4 / $20 |
| Gemini 3.8 Flash | Most intelligent Flash model | 1,048,576 input | 65,536 | See Google's pricing page |
Long prompts change the arithmetic. OpenAI bills GPT-6 prompts over 272K input tokens at 2x input and 1.5x output for the whole request, and Haiku 5.5 switches to its higher price above 100K prompt tokens. Anthropic's other current models bill the full 1M window at the standard rate.
Documented features that matter for routing: OpenAI lists structured outputs and function calling for GPT-6.1 Sol, with tool calling through the Responses API. Gemini 4 Argon lists code execution, computer use, function calling and structured outputs, and supports implicit caching only. Claude Opus 5.5 and Fable 5.2 always think (you steer depth with effort), and Fable 5.2 and the 5.5 models reject forced tool_choice.
Vendor benchmark numbers exist, but they are the vendor's own runs. In Anthropic's Claude Opus 5.5 announcement, Terminal-Bench 4.0 scores are 66.4% for Opus 5.5, 55.8% for Fable 5.1 (the Fable model at the time, since replaced by Fable 5.2) and 57.9% for GPT-6 Astra (the Astra figure as reported by OpenAI), and Anthropic's Sonnet 5.5 announcement reports 70.6% for Sonnet 5.5. Anthropic itself notes that its gap between Opus 5.5 and Fable 5.1 is narrower in use than these scores suggest. Treat them as a reason to include a model in your eval, not as the result of one.
My Starting Routing Policy (Opinion)
Everything in this section is my opinion, not a measured result. I haven't run a controlled evaluation behind it, so use it to pick what goes into your first eval run, then let your own numbers decide.
Code Generation
I'd put GPT-6.1 Sol and Claude Sonnet 5.5 side by side here first. They have the same list price ($2/$10), so the eval decides on quality and token count alone. I'd keep GPT-6 Astra and Claude Opus 5.5 for tasks where the cheaper tier fails your quality bar.
Debugging and Code Review
For debugging that means reading across a large codebase, every model in the table above accepts close to 1M tokens, so context size alone doesn't pick a winner. Price does: a 500K-token prompt costs more on GPT-6.1 Sol than on Sonnet 5.5 because of OpenAI's long-context multiplier. For focused, single-function debugging, I'd try both with a tight prompt.
Architecture and Design
Anthropic positions Claude Fable 5.2 for its most demanding reasoning and recommends starting with Opus 5.5, then moving to Fable 5.2 when your evals at higher effort still fall short. Fable costs $8/$40 against Sonnet 5.5's $2/$10 and Opus 5.5's $4/$20, so I'd use it on review passes that matter, not on every call. On the OpenAI side, GPT-6 Astra ($10/$50) is the equivalent tier, at a slightly higher price.
Documentation and Explanation
Style is a matter of taste, so judge samples from each model against your own house style. Whichever model you use, check the facts and run every example before publishing; fluent output hasn't been verified against the source.
Gemini
Gemini 4 Argon became generally available on October 13, 2026, and its model page lists built-in code execution and text, image, video, audio and PDF input. I'd include it when your inputs mix file types. Google describes Gemini 3.8 Flash (GA on September 2, 2026) as built for long-horizon software engineering and agents at Flash prices, and Flash-Lite as its budget line for high-throughput work.
Check it: Write down ten real tasks from your own workload, and what a good answer to each looks like. That list is the start of your eval.
A Note on Anthropic's Naming
Anthropic ships four widely released tiers — Haiku (fast/cheap), Sonnet (the default workhorse), Opus, and Fable above it as the most capable. Fable is not a rename of Opus: both ship in the Claude 5 generation, at different prices. There is also a Mythos line, which is not generally available, so do not plan a router around it. If you built a router against the Opus 5 or Sonnet 5 model IDs, budget time to re-run your eval suite against the 5.5 replacements before you assume the routing logic still picks the right tier. Opus 5.5 is cheaper than Opus 5, and both 5.5 models reject requests that disable thinking or force a specific tool, so price and request shape shift between releases even when the naming pattern stays the same.
Comparing Models on Real Engineering Tasks
Step 1: Define Your Evaluation Matrix
public class ModelEvaluation
{
public required string TaskCategory { get; init; }
public required string TestCase { get; init; }
public required string ModelId { get; init; }
public required double AccuracyScore { get; init; } // 0-1
public required double FormatScore { get; init; } // 0-1
public required double LatencyMs { get; init; }
public required double CostUsd { get; init; }
public required int InputTokens { get; init; }
public required int OutputTokens { get; init; }
}
Step 2: Build a Comparison Harness
// IModelClient and IScorer are your own interfaces, one adapter per provider SDK.
public class ModelComparer
{
private readonly Dictionary<string, IModelClient> _clients;
private readonly IScorer _scorer;
public async Task<List<ModelEvaluation>> CompareAsync(
string taskCategory,
List<TestCase> testCases,
params string[] modelIds)
{
var results = new List<ModelEvaluation>();
foreach (var testCase in testCases)
{
foreach (var modelId in modelIds)
{
var client = _clients[modelId];
var sw = Stopwatch.StartNew();
var response = await client.CompleteAsync(testCase.Messages);
sw.Stop();
var evaluation = await _scorer.ScoreAsync(
testCase, response.Content, taskCategory);
results.Add(new ModelEvaluation
{
TaskCategory = taskCategory,
TestCase = testCase.Name,
ModelId = modelId,
AccuracyScore = evaluation.Accuracy,
FormatScore = evaluation.FormatCompliance,
LatencyMs = sw.ElapsedMilliseconds,
CostUsd = CalculateCost(modelId, response.Usage),
InputTokens = response.Usage.InputTokens,
OutputTokens = response.Usage.OutputTokens
});
}
}
return results;
}
}
Step 3: Model Selection Decision Tree
public string SelectModel(TaskRequirements requirements)
{
// All three families accept about 1M tokens, so long context is a price question:
// GPT-6 bills prompts over 272K tokens at 2x input, Sonnet 5.5 bills 1M at the standard rate.
if (requirements.InputTokens > 272_000)
return "claude-sonnet-5-5";
// Multi-step reasoning / hard algorithmic problems → a top tier
if (requirements.RequiresDeepReasoning)
return "gpt-6-astra";
// Strict JSON extraction → GPT-6.1 Sol (structured outputs)
if (requirements.RequiresStructuredOutput)
return "gpt-6.1-sol";
// Cost-sensitive with acceptable quality → smaller models
if (requirements.MaxCostPerCall < 0.01m)
return "claude-haiku-5-5"; // or gpt-6-luna in the OpenAI ecosystem
// Architecture/deep review tasks → Claude Fable 5.2 (my choice; confirm with your eval)
if (requirements.Category is "architecture" or "code-review")
return "claude-fable-5-2";
// Default: GPT-6.1 Sol (swap for Claude Sonnet 5.5 if your eval prefers it; same list price)
return "gpt-6.1-sol";
}
Check it: Run the harness on two models, and divide total spend by the number of outputs that passed. Compare that figure and not the list price.
Mistakes in Model Comparisons
1. Evaluating on benchmarks instead of your tasks
Public benchmarks measure general capability. Your task is specific. A model that scores 5% higher on HumanEval might score 10% lower on your particular code generation task. Always evaluate on representative examples from your actual workload.
2. Comparing across different prompts
GPT and Claude respond differently to the same prompt. A prompt optimized for GPT often underperforms on Claude, and vice versa. For fair comparison, optimize the prompt for each model separately, then compare best-versus-best.
3. Ignoring cost per quality unit
A model that's 5% more accurate but 3x more expensive might not be worth it. Calculate cost per "good result" — total spend divided by the number of outputs that passed your quality threshold.
4. One-model-fits-all
Using a single model for all tasks is convenient but wasteful. Route different task types to different models. Code generation to GPT-6.1 Sol, long-context review to Claude Sonnet 5.5, complex reasoning to GPT-6 Astra or Claude Fable 5.2, simple classification to Claude Haiku 5.5 or GPT-6 Luna. The routing logic is simple. The saving is the price gap: GPT-6 Luna and Haiku 5.5 list at $0.10/$0.50, against $8/$40 (Fable 5.2) and $10/$50 (GPT-6 Astra) for the top tiers.
Try It on Your Own System This Week
Three checks on your current model choice, about an hour in total:
- Ask why each pipeline uses the model it does. If the answer is habit or a leaderboard, it hasn't been tested on your tasks.
- List the task types that go to your most expensive model. Try the simplest of them on a cheaper tier.
- Find out when you last re-ran a comparison. If a model generation has shipped since, the result is out of date.
How I Would Choose
- Evaluate on your tasks, not benchmarks. Build a test suite from your actual workload and compare models against it. What works for HumanEval may not work for your codebase.
- Context size no longer separates the families; price, features and your own eval do. All three list about 1M tokens of context. My starting split is in the routing section above, and it is opinion: for deep reasoning tasks, add the GPT-6 Astra and Claude Fable 5.2 tiers to the eval and keep them only where the quality gain pays for a 4x to 5x price step.
- Route tasks to the right model. Build a simple model router based on task type, context length, and cost budget, and measure the saving against your own bill.
- Optimize prompts per model. Same task, different prompt. Each model family has different strengths in instruction following — tune accordingly.
- Re-evaluate every quarter. All three families ship often. Between September and October 2026, OpenAI released GPT-6 Sol, GPT-6 Luna and GPT-6.1 Sol, Anthropic released Claude Opus 5.5, Sonnet 5.5 and Haiku 5.5, and Google made Gemini 3.8 Flash and Gemini 4 Argon generally available. Today's winner might not be tomorrow's. Keep your evaluation harness ready to re-run.