Your batch does not fit in GPU memory, but you want its gradient behavior. What helps?
Why A is the answer
Gradient accumulation
Small batches add gradients before each optimizer step.
In detail
- Gradient accumulation adds gradients across small batches before each step.
- Scale the learning rate with care.
Use case
An illustrative scenario, not a real incident.
A batch of 64 does not fit, so a team accumulates four batches of 16 before each update.
See it in different ways
flowchart TD A["Out of memory in training"] --> B["Smaller batch plus gradient accumulation"] A --> C["Mixed precision"] A --> D["Gradient checkpointing"] A --> E["Spread the model across GPUs"]
mindmap
root((Training memory))
Mixed precision
Smaller numbers
Gradient accumulation
Smaller batches
Gradient checkpointing
Recompute activations
Model parallelism
More GPUsflowchart LR S0["Small batch"] --> S1["Add gradients"] --> S2["Repeat"] --> S3["Optimizer step"]
Key point: Accumulation simulates larger batches.
Why the other options are wrong
- B. Raising the learning rate by the same factor
- That does not reduce memory.
- C. Removing the optimizer's weight decay setting
- Weight decay does not affect memory.
- D. Doubling the sequence length of each example
- That uses more memory.