One GPU keeps falling off the bus, and reseating does not help. What is the next step?
Why A is the answer
Confirm with diagnostics, then replace it
Confirm, replace, then run the tests again.
In detail
- Confirm, replace, then run the tests again.
- Run tests again after.
Use case
An illustrative scenario, not a real incident.
A GPU keeps falling off the bus, so diagnostics confirm it, and it is replaced.
See it in different ways
flowchart TD
A["Symptom"] --> B{"GPU, fan, NIC or power?"}
B --> C["Check logs and sensors"]
C --> D["Run diagnostics"]
D --> E["Replace the failed part"]
E --> F["Run the tests again"]stateDiagram-v2 [*] --> Symptom Symptom --> Logs: Xid and BMC events Logs --> Diagnostics Diagnostics --> Replace Replace --> Retest Retest --> [*]
flowchart LR S0["Diagnostics confirm"] --> S1["Replace"] --> S2["Run tests again"]
Key point: Confirm, replace, retest.
Why the other options are wrong
- B. Lower the Slurm priority for every user on the node
- Priority does not fix hardware.
- C. Remove the node from the cabling map
- That hides the node.
- D. Reinstall the operating system daily
- A reinstall will not fix a GPU.