A large model will not fit into a single GPU's memory during training. Which strategy addresses this?
Why C is the answer
Model parallelism across devices
Model parallelism splits the model across devices. Data parallelism copies the whole model to each device, so it does not help here.
In detail
- Model parallelism splits the model across devices.
- Data parallelism keeps a full copy on each device.
Use case
An illustrative scenario, not a real incident.
A very large model cannot fit one GPU, so its layers are placed across several devices.
See it in different ways
flowchart TD
A{"What is the bottleneck?"} -->|"Training is too slow"| B["Data parallelism"]
A -->|"Model exceeds one device"| C["Model parallelism"]stateDiagram-v2 [*] --> FitsInMemory FitsInMemory --> DataParallelism: yes, but too slow FitsInMemory --> ModelParallelism: no DataParallelism --> Train ModelParallelism --> Train Train --> Tune: hyperparameter tuning Tune --> [*]
flowchart TD
Q{"Does the model fit one device?"}
Q -->|"No"| A0["Model parallelism"]
Q -->|"Yes, but it is slow"| A1["Data parallelism"]Key point: Model parallelism splits the model.
Why the other options are wrong
- A. Data parallelism across devices
- Data parallelism still needs the model on each device.
- B. Gradient accumulation on one GPU
- That helps batch size, not model size.
- D. Lowering the learning rate
- A learning rate does not change memory.