Context Windows: Size, Quality, and Trade-offs

Bigger Context Is Not the Same As Better Retrieval

Context windows grew from a few thousand tokens to over a million between 2023 and 2025, and the instinct that follows is obvious: paste everything in. Skip retrieval, skip chunking, skip the engineering — let the model see it all and figure it out.

That instinct is wrong often enough to matter. A model that can technically accept a million tokens does not attend to all of them equally, and paying for a million input tokens on every call is a cost decision someone eventually has to answer for, not a free upgrade you get for switching model names in a config file. Context window size is a ceiling. It isn't a quality guarantee, and it isn't retrieval.

What "Context Window" Actually Counts

For most providers the context window is a shared budget across input and output tokens combined, not two separate limits. A model advertised with a 128k context window doesn't give you 128k for input plus a separate allotment for output — the response eats into the same ceiling as the prompt. So a long system prompt, a long conversation history, and a request for a long structured output are all competing for the same number, and pushing input close to that ceiling can silently truncate how much room the model has left to answer. Some providers publish a separate max-output cap sitting inside the larger window; always check both numbers, because the headline context length on the marketing page is not the whole story.

Lost in the Middle

Research on long-context retrieval — the "lost in the middle" work that ran needle-in-a-haystack tests across long inputs — found a consistent U-shaped accuracy curve. Models retrieve information placed near the start or end of a long context reliably. Accuracy for information buried in the middle drops, sometimes sharply, and this holds even for models explicitly marketed on their long-context capability. It gets worse as the context grows and as the needle gets harder to distinguish from the surrounding noise.

flowchart LR
    A["Start of context
High recall"] --> B["Middle of context
Recall drops sharply"] --> C["End of context
High recall (recency)"]

    style A fill:#059669,color:#fff,stroke:#047857
    style B fill:#dc2626,color:#fff,stroke:#b91c1c
    style C fill:#059669,color:#fff,stroke:#047857
      

Here's the practical consequence for "just paste the whole codebase in": if the one function that actually matters lands in the middle third of a 400k-token dump, the model is measurably less likely to use it correctly than if it sat in the last ten percent. Order and placement inside a long context aren't neutral. Left to chance, they become a retrieval variable you never decided to introduce.

When Large Context Genuinely Helps

None of this is an argument against long context, only against reaching for it without thinking. Whole-repo code review is a real strength: holding an entire module or service in context lets a model catch cross-file inconsistencies that chunk-based retrieval would miss outright, because the relevant pieces were never sitting next to each other to begin with. Long single-document QA is another — a two-hundred-page contract or a full research paper, where the question depends on synthesizing across the whole document rather than pulling one isolated fact. And large windows make dense few-shot prompting practical, dozens of examples in-context, for the stretch of a project before fine-tuning is justified.

When RAG Still Wins, Even With a Huge Window Available

A 1M-token window doesn't obsolete RAG. It moves the threshold at which building retrieval infrastructure is worth the engineering cost. If your corpus fits comfortably in context and doesn't change often, long-context-only can be simpler and plenty good. Past that — larger, updated frequently, or cost-sensitive at real volume — retrieval still wins, and it isn't close.

Context Caching as a Cost Lever

Major providers now offer some form of prompt or context caching. Send a large, stable prefix once — a system prompt, a document, a codebase snapshot — and the provider caches the processed representation. Calls that reuse the same prefix get billed at a steep discount on the cached portion and come back faster. This is the actual answer to "long context is expensive," not a workaround: if the same multi-hundred-thousand-token context gets reused across many requests, caching turns a per-call cost into something closer to a one-time cost with cheap reads after.

# Illustrative shape of a cached-context call (Anthropic-style prompt caching).
# Exact parameter names and discount rates vary by provider — check current docs.
response = client.messages.create(
    model="claude-sonnet-5",
    system=[
        {
            "type": "text",
            "text": large_codebase_dump,       # e.g. 300k tokens, stable across calls
            "cache_control": {"type": "ephemeral"},
        }
    ],
    messages=[{"role": "user", "content": "Where is the retry logic for the payment webhook?"}],
)
# First call: full price, cache write.
# Subsequent calls within the cache TTL: cached-read pricing, meaningfully cheaper
# and faster on the cached prefix.

Chunking and Summarization Chains for Exceeding Even 1M Tokens

Some corpora outgrow even the largest available window — a large monorepo, a year of support tickets, a multi-document legal case file. The fix that scales is a chain, not a single call. Chunk by logical boundary rather than fixed token count (file, section, function), with a small overlap so a reference that straddles a boundary doesn't get severed. Summarize each chunk independently, then summarize the summaries, recursing if the corpus is large enough to need more than one reduce pass — the classic map-reduce shape, applied to text instead of data. For anything sequential, like a long transcript or a changelog, a running summary that updates as each new window arrives usually beats reprocessing everything from scratch. And when you can afford the extra moving part, retrieving at the summary level first to find which sections matter, then pulling full text only for the ones that turned out relevant, keeps you from either truncating important detail or paying to reprocess a corpus that hasn't meaningfully changed.

// Simplified map-reduce summarization chain
async function summarizeLargeCorpus(chunks: string[], targetTokens: number): Promise<string> {
  // Map: summarize each chunk independently (can run in parallel)
  const summaries = await Promise.all(
    chunks.map((chunk) => llm.summarize(chunk, { maxTokens: 500 }))
  );

  const combined = summaries.join('\n\n');
  if (estimateTokens(combined) <= targetTokens) return combined;

  // Reduce: recursively summarize the summaries until it fits the target budget
  return summarizeLargeCorpus(chunkBySize(combined, 50_000), targetTokens);
}

Key Takeaways