A model fits on one GPU, and you want faster training on eight GPUs. What is the usual approach?
Why D is the answer
Distributed data parallel training
Each GPU trains a replica on different data, and gradients are synchronized.
In detail
- Each GPU trains a replica, and gradients are synchronized.
- Speedup is rarely perfectly linear.
Use case
An illustrative scenario, not a real incident.
Each of eight GPUs trains a replica on different data, and gradients are synchronized.
See it in different ways
flowchart TD
A{"Does the model fit on one GPU?"} -->|Yes| B["Data parallelism: replicas with different batches"]
A -->|No| C["Model parallelism: split the layers or tensors"]stateDiagram-v2 [*] --> FitsOnOneGpu FitsOnOneGpu --> DataParallel: yes FitsOnOneGpu --> ModelParallel: no DataParallel --> Profile ModelParallel --> Profile Profile --> [*]: Nsight Systems
mindmap
root(("Distributed data parallel"))
n1["GPU 1"]
n2["Replica, data part 1"]
n3["GPU 2"]
n4["Replica, data part 2"]Key point: Distributed data parallel speeds up training.
Why the other options are wrong
- A. Splitting each layer by hand, with one layer on each GPU
- Hand splitting is not needed when the model fits.
- B. Running eight unrelated jobs and keeping the best
- Those jobs do not cooperate.
- C. Reducing the dataset to one eighth of its size
- That loses data.