A document is longer than the model's context window. How can a summarizer handle it?
Why B is the answer
Summarize chunks, then combine the summaries
Map and combine steps keep each call within the window.
In detail
- Summarize parts, then combine the summaries.
- Each call stays within the window.
- Each call stays within the window.
Use case
An illustrative scenario, not a real incident.
A 300-page report exceeds the context window. Each chunk is summarized, and then the summaries are combined.
See it in different ways
flowchart LR A["Curate content"] --> B["Embed and store"] C["User question"] --> D["Retrieve passages"] B --> D D --> E["Prompt with context"] E --> F["LLM answer"]
mindmap
root((LLM service))
Hardware
GPU memory
Throughput
Batching
Experience
Streaming
Conversation history
Packaging
NIMflowchart LR S0["Split into chunks"] --> S1["Summarize each"] --> S2["Combine summaries"]
Key point: Map, then combine.
Why the other options are wrong
- A. Lower the output token limit to make more room for input
- That leaves less room for output.
- C. Raise the temperature until the whole document fits
- Temperature has no effect on size.
- D. Cut the document off at the context limit
- A cut loses the rest of the document.