What is the main cost of a very wide beam in beam search?
Why D is the answer
More computation per generated token
More candidates must be scored at each step.
In detail
- More candidates must be scored at each step.
- Wider does not always mean better.
Use case
An illustrative scenario, not a real incident.
A wide beam keeps many candidates, and each must be scored at each step.
See it in different ways
sequenceDiagram participant B as Batch participant M as Model participant C as KV cache B->>M: Newest token only C-->>M: Past keys and values M->>C: Adds new keys and values M-->>B: Next token
flowchart TD
Q{"What matters more?"}
Q -->|"Speed"| A0["Narrow beam"]
Q -->|"Exploring candidates"| A1["Wider beam"]Key point: Wide beams cost computation.
Why the other options are wrong
- A. A loss of all model weights after each request
- Weights are not lost.
- B. A smaller vocabulary for the output tokens
- Vocabulary is unchanged.
- C. No output tokens beyond the first few
- Output is not limited in that way.