GPU utilization rises and falls in a repeating pattern during training. What do you suspect?
Why D is the answer
Data loading or storage
GPUs wait between batches.
In detail
- A repeating pattern suggests GPUs are waiting between batches.
- Check loader workers and storage.
Use case
An illustrative scenario, not a real incident.
GPU use dips between batches in a pattern, because the data loader cannot keep up.
See it in different ways
flowchart LR A["Storage"] --> B["Network"] B --> C["CPU data loading"] C --> D["GPU"]
stateDiagram-v2 [*] --> LowUtilization LowUtilization --> DataPath: GPUs waiting for data LowUtilization --> Throttling: temperature or power DataPath --> BenchmarkStorage Throttling --> CheckCooling BenchmarkStorage --> [*] CheckCooling --> [*]
stateDiagram-v2 state "GPU busy" as s0 state "GPU waits" as s1 [*] --> s0 s0 --> s1: batch done s1 --> s0: data arrives
Key point: A repeating dip suggests data loading.
Why the other options are wrong
- A. The Fabric Manager log level
- A log level has no effect.
- B. The dashboard refresh rate
- A refresh rate has no effect.
- C. A low BMC fan speed
- A fan fault would show as throttling.