A model does not fit on one GPU. Which approach applies?
Why C is the answer
Model parallelism
Data parallelism needs a full replica on every GPU.
In detail
- Model parallelism splits the model across GPUs.
- Data parallelism needs a full replica per GPU.
Use case
An illustrative scenario, not a real incident.
A model exceeds one GPU's memory, so its layers are split across several.
See it in different ways
flowchart TD
A{"Does the model fit on one GPU?"} -->|Yes| B["Data parallelism: replicas with different batches"]
A -->|No| C["Model parallelism: split the layers or tensors"]stateDiagram-v2 [*] --> FitsOnOneGpu FitsOnOneGpu --> DataParallel: yes FitsOnOneGpu --> ModelParallel: no DataParallel --> Profile ModelParallel --> Profile Profile --> [*]: Nsight Systems
flowchart TD
Q{"Does the model fit on one GPU?"}
Q -->|"No"| A0["Model parallelism"]
Q -->|"Yes"| A1["Data parallelism"]Key point: Model parallelism splits the model.
Why the other options are wrong
- A. More workers in the data loader
- Workers load data.
- B. A higher setting for sampling temperature
- Temperature is for generation.
- D. Data parallelism, with one full copy on each GPU
- That needs a full copy on each GPU.