What does gradient checkpointing trade?
Why B is the answer
Extra compute for less memory
Activations are recomputed instead of being stored.
In detail
- Activations are recomputed rather than stored.
- Use it when memory is the limit.
Use case
An illustrative scenario, not a real incident.
A training run exceeds memory. Checkpointing recomputes activations, which saves memory and costs time.
See it in different ways
flowchart TD A["Out of memory in training"] --> B["Smaller batch plus gradient accumulation"] A --> C["Mixed precision"] A --> D["Gradient checkpointing"] A --> E["Spread the model across GPUs"]
mindmap
root((Training memory))
Mixed precision
Smaller numbers
Gradient accumulation
Smaller batches
Gradient checkpointing
Recompute activations
Model parallelism
More GPUsmindmap
root(("Gradient checkpointing"))
n1["Saves"]
n2["Memory"]
n3["Costs"]
n4["Extra compute"]Key point: Checkpointing trades compute for memory.
Why the other options are wrong
- A. Fewer tokens for a larger vocabulary size
- That is not this trade.
- C. Less compute for a larger memory footprint
- That is the reverse.
- D. A lower learning rate for faster convergence
- That is not this trade.