After bring-up, you want a deep health test of every GPU in a node. What do you run?
Why D is the answer
A DCGM diagnostic such as dcgmi diag
DCGM diagnostics stress and validate GPUs at several levels.
In detail
- dcgmi diag runs deep GPU health tests.
- Higher levels run longer tests.
Use case
An illustrative scenario, not a real incident.
A new node passes a quick check, but dcgmi diag at a higher level finds a faulty GPU under stress.
See it in different ways
flowchart TD A["Single-node stress"] --> B["Single-node NCCL"] B --> C["Single-node HPL"] C --> D["Multi-node NCCL"] D --> E["Multi-node HPL and burn-in"]
stateDiagram-v2 [*] --> SingleNodeStress SingleNodeStress --> SingleNodeNccl SingleNodeNccl --> SingleNodeHpl SingleNodeHpl --> MultiNodeNccl MultiNodeNccl --> MultiNodeHpl MultiNodeHpl --> BurnIn BurnIn --> [*]
flowchart LR S0["Quick check"] --> S1["dcgmi diag"] --> S2["Review failures"] --> S3["Replace or accept"]
Key point: DCGM diagnostics validate GPUs deeply.
Why the other options are wrong
- A. nvidia-smi with no arguments, to see that each GPU is listed
- That only shows that a GPU is listed.
- B. A short PyTorch import test on each node
- An import test does not stress a GPU.
- C. ibstat on every network port of the node
- ibstat checks network ports.