On an NVSwitch-based system, GPU-to-GPU communication fails after boot. Which service should you check?
Why B is the answer
NVIDIA Fabric Manager
Fabric Manager configures the NVSwitch fabric on these systems, so it must be running.
In detail
- NVSwitch systems depend on Fabric Manager.
- It configures the NVSwitch fabric.
Use case
An illustrative scenario, not a real incident.
GPUs fail to communicate after boot, and Fabric Manager turns out to be stopped.
See it in different ways
flowchart LR A["GPU"] -->|"Single-node NCCL"| B["NVLink and NVSwitch"] B --> C["GPU"] A -->|"Multi-node NCCL"| D["East-west fabric"] D --> E["GPU in another node"]
stateDiagram-v2 [*] --> Symptom Symptom --> Logs: Xid and BMC events Logs --> Diagnostics Diagnostics --> Replace Replace --> Retest Retest --> [*]
flowchart TD
Q{"Is Fabric Manager running?"}
Q -->|"No"| A0["Start it and check its log"]
Q -->|"Yes"| A1["Look further"]Key point: Fabric Manager must be running.
Why the other options are wrong
- A. NVIDIA Container Runtime
- That is for containers.
- C. The NVIDIA persistence daemon
- That daemon keeps the driver loaded.
- D. The Subnet Manager service
- That is for InfiniBand.