Lost in the middle, context degradation, chunking strategies.

Context Windows — Managing 128k to 2M Token Contexts

Context window is the maximum tokens an LLM can process in one call. Frontier models: GPT-4o (128k), Claude 3.7 (200k), Gemini 1.5 Pro (2M). Larger contexts enable more information but have risks: "lost in the middle" (degraded attention to middle content), higher cost, higher latency. Strategies: RAG for selective retrieval, chunking for long documents, context compression (summarisation), and attention-aware placement (key information at start/end).

Interactive Pipeline Flow

UnderstandImplementOptimise

🔢 Measure — token count

Count tokens before calling the API. Use tiktoken for OpenAI models. Leave headroom for output: max_tokens_for_input = model_limit - expected_output_tokens.

Architecture Diagrams

Research finding on LLM attention across context positions.

Loading diagram...

Key Design Decisions

📌Place critical information at start and end

"Lost in the middle" is real — LLMs have degraded recall for content in the middle of long contexts. Critical instructions in the system prompt (start) and the user question (end) get the most attention.

🔍RAG beats full-context for accuracy on long documents

Counterintuitively, retrieving 5 relevant chunks often produces more accurate answers than sending the entire 100-page document. Less noise, more focused context, lower cost, faster.

💰Long context is expensive — use it for genuinely long tasks

Processing a 100k-token document costs ~$0.25 per request with GPT-4o. For 1000 queries/day = $250/day. RAG with 3k-token retrieval = $7.50/day. Use long context only when the full document is truly necessary.

Discussion