Training is unexpectedly slow, and GPU utilization is low. What is a likely cause?
Why A is the answer
The data loading path is starving the GPUs
Low GPU utilization on a busy job usually means the GPUs are waiting for data.
In detail
- Low utilization often means GPUs are waiting for data.
- Low utilization often means waiting.
Use case
An illustrative scenario, not a real incident.
A job is busy but GPU use is low, because reads from storage cannot keep up.
See it in different ways
flowchart LR A["Storage"] --> B["Network"] B --> C["CPU data loading"] C --> D["GPU"]
stateDiagram-v2 [*] --> LowUtilization LowUtilization --> DataPath: GPUs waiting for data LowUtilization --> Throttling: temperature or power DataPath --> BenchmarkStorage Throttling --> CheckCooling BenchmarkStorage --> [*] CheckCooling --> [*]
flowchart TD
Q{"What do the GPUs wait for?"}
Q -->|"Data"| A0["Fix the loading path"]
Q -->|"Nothing"| A1["Review the job"]Key point: Low GPU use often means waiting for data.
Why the other options are wrong
- B. A storage tier that outruns the GPUs
- Fast storage would help.
- C. A container image that is too large to cache
- Image size does not affect training speed.
- D. Too many GPUs allocated to a single job
- That would not lower utilization this way.