A GPU throttles under load. What should you check?
Why A is the answer
Temperature, power limits and cooling
Throttling usually comes from thermal or power limits. Telemetry shows which.
In detail
- Throttling comes from temperature or power.
- Telemetry shows which limit applies.
Use case
An illustrative scenario, not a real incident.
A GPU slows under load, and telemetry shows a temperature limit.
See it in different ways
flowchart TD
A{"What do you need?"} -->|"Quick view of a host"| B["nvidia-smi"]
A -->|"Fleet health and diagnostics"| C["DCGM"]
A -->|"DGX system health"| D["NVSM"]stateDiagram-v2 [*] --> LowUtilization LowUtilization --> DataPath: GPUs waiting for data LowUtilization --> Throttling: temperature or power DataPath --> BenchmarkStorage Throttling --> CheckCooling BenchmarkStorage --> [*] CheckCooling --> [*]
flowchart TD
Q{"Which limit is reached?"}
Q -->|"Temperature"| A0["Check cooling"]
Q -->|"Power"| A1["Check power limits"]Key point: Throttling points to thermal or power limits.
Why the other options are wrong
- B. The user's Slurm account and its fair-share limits
- Accounts do not throttle GPUs.
- C. The container's name and image tag
- Names do not throttle GPUs.
- D. The DNS TTL on the cluster domain
- DNS does not throttle GPUs.