A large training cluster needs very low latency and RDMA between nodes. Which network is the classic choice?
Why A is the answer
InfiniBand
InfiniBand offers low latency and native RDMA, which suits large-scale training.
In detail
- InfiniBand is the classic choice for low-latency RDMA between nodes.
- Ethernet with RoCE is another route to RDMA.
Use case
An illustrative scenario, not a real incident.
A training cluster spends much of its time exchanging data between nodes. InfiniBand gives low latency and native RDMA.
See it in different ways
flowchart TD
A{"What are the priorities?"} -->|"Largest training, lowest latency"| B["InfiniBand"]
A -->|"Ethernet skills and RDMA"| C["RoCE on Ethernet"]
A -->|"AI on Ethernet at scale"| D["Spectrum-X"]sequenceDiagram participant A as Server A memory participant N as NIC participant B as Server B memory A->>N: Direct read N->>B: Direct write over the fabric B-->>A: Completion
flowchart TD
Q{"What does the team need?"}
Q -->|"Native RDMA at scale"| A0["InfiniBand"]
Q -->|"RDMA with Ethernet skills"| A1["RoCE"]Key point: InfiniBand suits large-scale training.
Why the other options are wrong
- B. Standard 1 GbE Ethernet
- Standard 1 GbE is far too slow.
- C. A VPN over the internet
- That is not a cluster fabric.
- D. Bluetooth
- That is not a cluster fabric.