Start With One Sensible Default

For one low-stakes use, pick a sensible mid-tier model and get on with it. Comparison work pays for itself when a pipeline runs at volume, when quality matters, or when someone wants to switch because of a leaderboard.

"Which model should we use?" comes up in every team building with LLMs, and the usual answer is a screenshot of a leaderboard. The leaderboard measures someone else's tasks. Your code reviews, debugging sessions and architecture questions are not on it.

Claude, GPT and Gemini differ, but the differences show up task by task, not as one overall ranking. Below are the facts the vendors publish (context windows, prices, documented features and their own benchmark numbers), then my starting routing policy, labelled as opinion, and a harness for checking it on your own tasks before you commit.

If you are choosing a model right now
  • Your team defaults to GPT-6.1 Sol or Claude Sonnet 5.5 for everything without evaluating alternatives
  • You've tried Claude once, got a different result, and aren't sure which was better
  • You need to pick a model for a production pipeline and want a data-driven decision
  • You're paying for a frontier model on tasks where a cheaper model would work just as well

What Benchmarks Measure, and What They Miss

Comparing LLMs requires defining what you're comparing on. Benchmark scores (MMLU, HumanEval) tell you about general capability, not about performance on your specific tasks. What matters in practice:


      flowchart TD
          T["Your Task"] --> E["Evaluation Criteria"]
          E --> A["Accuracy\n(correct output)"]
          E --> F["Instruction Following\n(format compliance)"]
          E --> C["Context Handling\n(long input performance)"]
          E --> L["Latency\n(time to first token)"]
          E --> P["Price\n(cost per million tokens)"]
      
          style T fill:#4f46e5,color:#fff,stroke:#4338ca
          style A fill:#059669,color:#fff,stroke:#047857
          style F fill:#059669,color:#fff,stroke:#047857
          style C fill:#059669,color:#fff,stroke:#047857
          style L fill:#d97706,color:#fff,stroke:#b45309
          style P fill:#d97706,color:#fff,stroke:#b45309
      

What the Vendors Publish

These figures come from each vendor's model and pricing pages, checked on 2026-10-31. Prices are standard (non-batch) USD per million input / output tokens.

ModelVendor positioningContextMax outputPrice (in / out)
Claude Fable 5.2Most capable widely released Claude model1M128K$8 / $40
Claude Opus 5.5Long-running agentic coding and knowledge work1M128K$4 / $20
Claude Sonnet 5.5Speed and intelligence1M128K$2 / $10
Claude Haiku 5.5High-volume, latency-sensitive work1M128K$0.10 / $0.50 (prompts up to 100K tokens; $0.50 / $2.50 above)
GPT-6 AstraOpenAI's most capable model1.05M (922K input)128K$10 / $50
GPT-6.1 SolNear-Astra performance at lower cost1.05M (922K input)128K$2 / $10
GPT-6 LunaFocused, high-volume tasks1.05M (922K input)128K$0.10 / $0.50
Gemini 4 ArgonGoogle's flagship reasoning model1,048,576 (input and output combined)65,536 (up to 1M with long decode)$2 / $10 through 2026-11-30, then $4 / $20
Gemini 3.8 FlashMost intelligent Flash model1,048,576 input65,536See Google's pricing page

Long prompts change the arithmetic. OpenAI bills GPT-6 prompts over 272K input tokens at 2x input and 1.5x output for the whole request, and Haiku 5.5 switches to its higher price above 100K prompt tokens. Anthropic's other current models bill the full 1M window at the standard rate.

Documented features that matter for routing: OpenAI lists structured outputs and function calling for GPT-6.1 Sol, with tool calling through the Responses API. Gemini 4 Argon lists code execution, computer use, function calling and structured outputs, and supports implicit caching only. Claude Opus 5.5 and Fable 5.2 always think (you steer depth with effort), and Fable 5.2 and the 5.5 models reject forced tool_choice.

Vendor benchmark numbers exist, but they are the vendor's own runs. In Anthropic's Claude Opus 5.5 announcement, Terminal-Bench 4.0 scores are 66.4% for Opus 5.5, 55.8% for Fable 5.1 (the Fable model at the time, since replaced by Fable 5.2) and 57.9% for GPT-6 Astra (the Astra figure as reported by OpenAI), and Anthropic's Sonnet 5.5 announcement reports 70.6% for Sonnet 5.5. Anthropic itself notes that its gap between Opus 5.5 and Fable 5.1 is narrower in use than these scores suggest. Treat them as a reason to include a model in your eval, not as the result of one.

My Starting Routing Policy (Opinion)

Everything in this section is my opinion, not a measured result. I haven't run a controlled evaluation behind it, so use it to pick what goes into your first eval run, then let your own numbers decide.

Code Generation

I'd put GPT-6.1 Sol and Claude Sonnet 5.5 side by side here first. They have the same list price ($2/$10), so the eval decides on quality and token count alone. I'd keep GPT-6 Astra and Claude Opus 5.5 for tasks where the cheaper tier fails your quality bar.

Debugging and Code Review

For debugging that means reading across a large codebase, every model in the table above accepts close to 1M tokens, so context size alone doesn't pick a winner. Price does: a 500K-token prompt costs more on GPT-6.1 Sol than on Sonnet 5.5 because of OpenAI's long-context multiplier. For focused, single-function debugging, I'd try both with a tight prompt.

Architecture and Design

Anthropic positions Claude Fable 5.2 for its most demanding reasoning and recommends starting with Opus 5.5, then moving to Fable 5.2 when your evals at higher effort still fall short. Fable costs $8/$40 against Sonnet 5.5's $2/$10 and Opus 5.5's $4/$20, so I'd use it on review passes that matter, not on every call. On the OpenAI side, GPT-6 Astra ($10/$50) is the equivalent tier, at a slightly higher price.

Documentation and Explanation

Style is a matter of taste, so judge samples from each model against your own house style. Whichever model you use, check the facts and run every example before publishing; fluent output hasn't been verified against the source.

Gemini

Gemini 4 Argon became generally available on October 13, 2026, and its model page lists built-in code execution and text, image, video, audio and PDF input. I'd include it when your inputs mix file types. Google describes Gemini 3.8 Flash (GA on September 2, 2026) as built for long-horizon software engineering and agents at Flash prices, and Flash-Lite as its budget line for high-throughput work.

✅

Check it: Write down ten real tasks from your own workload, and what a good answer to each looks like. That list is the start of your eval.

A Note on Anthropic's Naming

Anthropic ships four widely released tiers — Haiku (fast/cheap), Sonnet (the default workhorse), Opus, and Fable above it as the most capable. Fable is not a rename of Opus: both ship in the Claude 5 generation, at different prices. There is also a Mythos line, which is not generally available, so do not plan a router around it. If you built a router against the Opus 5 or Sonnet 5 model IDs, budget time to re-run your eval suite against the 5.5 replacements before you assume the routing logic still picks the right tier. Opus 5.5 is cheaper than Opus 5, and both 5.5 models reject requests that disable thinking or force a specific tool, so price and request shape shift between releases even when the naming pattern stays the same.

Comparing Models on Real Engineering Tasks

Step 1: Define Your Evaluation Matrix

public class ModelEvaluation
{
    public required string TaskCategory { get; init; }
    public required string TestCase { get; init; }
    public required string ModelId { get; init; }
    public required double AccuracyScore { get; init; }    // 0-1
    public required double FormatScore { get; init; }       // 0-1
    public required double LatencyMs { get; init; }
    public required double CostUsd { get; init; }
    public required int InputTokens { get; init; }
    public required int OutputTokens { get; init; }
}

Step 2: Build a Comparison Harness

// IModelClient and IScorer are your own interfaces, one adapter per provider SDK.
public class ModelComparer
{
    private readonly Dictionary<string, IModelClient> _clients;
    private readonly IScorer _scorer;

    public async Task<List<ModelEvaluation>> CompareAsync(
        string taskCategory,
        List<TestCase> testCases,
        params string[] modelIds)
    {
        var results = new List<ModelEvaluation>();

        foreach (var testCase in testCases)
        {
            foreach (var modelId in modelIds)
            {
                var client = _clients[modelId];
                var sw = Stopwatch.StartNew();
                var response = await client.CompleteAsync(testCase.Messages);
                sw.Stop();

                var evaluation = await _scorer.ScoreAsync(
                    testCase, response.Content, taskCategory);

                results.Add(new ModelEvaluation
                {
                    TaskCategory = taskCategory,
                    TestCase = testCase.Name,
                    ModelId = modelId,
                    AccuracyScore = evaluation.Accuracy,
                    FormatScore = evaluation.FormatCompliance,
                    LatencyMs = sw.ElapsedMilliseconds,
                    CostUsd = CalculateCost(modelId, response.Usage),
                    InputTokens = response.Usage.InputTokens,
                    OutputTokens = response.Usage.OutputTokens
                });
            }
        }

        return results;
    }
}

Step 3: Model Selection Decision Tree

public string SelectModel(TaskRequirements requirements)
{
    // All three families accept about 1M tokens, so long context is a price question:
    // GPT-6 bills prompts over 272K tokens at 2x input, Sonnet 5.5 bills 1M at the standard rate.
    if (requirements.InputTokens > 272_000)
        return "claude-sonnet-5-5";

    // Multi-step reasoning / hard algorithmic problems → a top tier
    if (requirements.RequiresDeepReasoning)
        return "gpt-6-astra";

    // Strict JSON extraction → GPT-6.1 Sol (structured outputs)
    if (requirements.RequiresStructuredOutput)
        return "gpt-6.1-sol";

    // Cost-sensitive with acceptable quality → smaller models
    if (requirements.MaxCostPerCall < 0.01m)
        return "claude-haiku-5-5";  // or gpt-6-luna in the OpenAI ecosystem

    // Architecture/deep review tasks → Claude Fable 5.2 (my choice; confirm with your eval)
    if (requirements.Category is "architecture" or "code-review")
        return "claude-fable-5-2";

    // Default: GPT-6.1 Sol (swap for Claude Sonnet 5.5 if your eval prefers it; same list price)
    return "gpt-6.1-sol";
}
✅

Check it: Run the harness on two models, and divide total spend by the number of outputs that passed. Compare that figure and not the list price.

Mistakes in Model Comparisons

⚠️ Common Mistakes

1. Evaluating on benchmarks instead of your tasks

Public benchmarks measure general capability. Your task is specific. A model that scores 5% higher on HumanEval might score 10% lower on your particular code generation task. Always evaluate on representative examples from your actual workload.

2. Comparing across different prompts

GPT and Claude respond differently to the same prompt. A prompt optimized for GPT often underperforms on Claude, and vice versa. For fair comparison, optimize the prompt for each model separately, then compare best-versus-best.

3. Ignoring cost per quality unit

A model that's 5% more accurate but 3x more expensive might not be worth it. Calculate cost per "good result" — total spend divided by the number of outputs that passed your quality threshold.

4. One-model-fits-all

Using a single model for all tasks is convenient but wasteful. Route different task types to different models. Code generation to GPT-6.1 Sol, long-context review to Claude Sonnet 5.5, complex reasoning to GPT-6 Astra or Claude Fable 5.2, simple classification to Claude Haiku 5.5 or GPT-6 Luna. The routing logic is simple. The saving is the price gap: GPT-6 Luna and Haiku 5.5 list at $0.10/$0.50, against $8/$40 (Fable 5.2) and $10/$50 (GPT-6 Astra) for the top tiers.

Try It on Your Own System This Week

Three checks on your current model choice, about an hour in total:

  1. Ask why each pipeline uses the model it does. If the answer is habit or a leaderboard, it hasn't been tested on your tasks.
  2. List the task types that go to your most expensive model. Try the simplest of them on a cheaper tier.
  3. Find out when you last re-ran a comparison. If a model generation has shipped since, the result is out of date.

How I Would Choose

✅ Key Lessons
  • Evaluate on your tasks, not benchmarks. Build a test suite from your actual workload and compare models against it. What works for HumanEval may not work for your codebase.
  • Context size no longer separates the families; price, features and your own eval do. All three list about 1M tokens of context. My starting split is in the routing section above, and it is opinion: for deep reasoning tasks, add the GPT-6 Astra and Claude Fable 5.2 tiers to the eval and keep them only where the quality gain pays for a 4x to 5x price step.
  • Route tasks to the right model. Build a simple model router based on task type, context length, and cost budget, and measure the saving against your own bill.
  • Optimize prompts per model. Same task, different prompt. Each model family has different strengths in instruction following — tune accordingly.
  • Re-evaluate every quarter. All three families ship often. Between September and October 2026, OpenAI released GPT-6 Sol, GPT-6 Luna and GPT-6.1 Sol, Anthropic released Claude Opus 5.5, Sonnet 5.5 and Haiku 5.5, and Google made Gemini 3.8 Flash and Gemini 4 Argon generally available. Today's winner might not be tomorrow's. Keep your evaluation harness ready to re-run.