In this article
Problem: Long conversations and large document retrievals exceed context window limits, causing truncation and loss of important context.
Solution: Apply compression strategies to reduce context size while preserving the most relevant information for the current query.
Compression strategies:
- Conversation summarisation: After every N turns (e.g., 10), summarise the full conversation history into a concise summary and replace the old turns
- Selective retention: Score each historical turn for relevance to the current query (embedding similarity); discard low-scoring turns
- Document compression: For retrieved chunks, use an LLM to extract only the relevant sentences from each chunk before inserting into context
- LLMLingua (Microsoft): Token-level compression — removes redundant tokens from prompts using a small surrogate model, achieving 3-20x compression with minimal quality loss
Trade-Offs:
- ✔ Pro: Enables infinite-length conversations within finite context windows
- ✔ Pro: Reduces tokens → reduces cost per turn
- ✖ Con: Summarisation loses nuance; selective retention may discard information that later becomes relevant
- ✖ Con: Adds latency for compression LLM calls
When To Use: Long multi-turn conversations, multi-document RAG, agent loops with many tool outputs. When to avoid: Short, stateless API calls where context never grows.