Why does a KV cache speed up token generation?
Why C is the answer
It stores past attention keys and values, so they are not recomputed for each new token
Without it, each new token recomputes attention over the whole sequence.
In detail
- The cache stores past keys and values, so only the newest token is computed.
- The cache uses GPU memory.
Use case
An illustrative scenario, not a real incident.
Generating token 500 would repeat the work for 499 tokens. A KV cache keeps their keys and values.
See it in different ways
flowchart LR A["Queue"] --> B["First token"] B --> C["KV cache reuse"] C --> D["Later tokens"] E["Continuous batching"] --> A
sequenceDiagram participant B as Batch participant M as Model participant C as KV cache B->>M: Newest token only C-->>M: Past keys and values M->>C: Adds new keys and values M-->>B: Next token
flowchart LR S0["Generate a token"] --> S1["Store keys and values"] --> S2["Compute only the new token"]
Key point: A KV cache avoids recomputing.
Why the other options are wrong
- A. It moves the token embeddings to disk and loads them back on each step
- It does not move embeddings to disk.
- B. It reduces the vocabulary to the tokens seen earlier in the same request
- It does not shrink the vocabulary.
- D. It compresses the model weights so more of them fit in GPU memory at once
- It does not compress weights.