A multi-node training test reports poor bandwidth between two nodes, while each node passes alone. What do you investigate?
Why D is the answer
The network fabric and links between those nodes
Single-node tests pass, so the fault is likely in between. Check cables, transceivers and ports.
In detail
- Each node passes alone, so look at the path between them.
- Check cables, transceivers and ports.
Use case
An illustrative scenario, not a real incident.
Each node passes alone, but bandwidth between two is poor. A bad transceiver turns up on that path.
See it in different ways
flowchart LR A["GPU"] -->|"Single-node NCCL"| B["NVLink and NVSwitch"] B --> C["GPU"] A -->|"Multi-node NCCL"| D["East-west fabric"] D --> E["GPU in another node"]
mindmap
root((Verification))
Compute
HPL
ClusterKit
Fabric
NCCL
Signal quality
Versions
Switches
BlueField
Transceivers
Storage
fioflowchart TD
Q{"Does each node pass alone?"}
Q -->|"Yes"| A0["Check the fabric between them"]
Q -->|"No"| A1["Check that node"]Key point: Pair failures point to the fabric.
Why the other options are wrong
- A. The firmware on the NVMe drives of those two nodes
- NVMe firmware is not on that path.
- B. The GPU driver on a third node in the cluster
- A third node is not on that path.
- C. The Python version in the test container
- Python does not cause low bandwidth.