A GPU reports a rising number of memory errors. What should operations do?
Why D is the answer
Drain it and investigate
Remove failing hardware from service before jobs are affected.
In detail
- Remove failing hardware from service before jobs are affected.
- Act before jobs are affected.
Use case
An illustrative scenario, not a real incident.
A GPU's memory errors keep rising, so it is drained, and jobs move elsewhere.
See it in different ways
flowchart TD A["DCGM"] --> B["Utilization"] A --> C["Memory"] A --> D["Temperature"] A --> E["Power"] A --> F["ECC and Xid errors"]
mindmap
root((Operations))
Monitoring
nvidia-smi
DCGM
Scheduling
Slurm
Kubernetes
Provisioning
Base Command Manager
Sharing
MIG
vGPUstateDiagram-v2 state "In service" as s0 state "Errors rising" as s1 state "Drained" as s2 state "Investigated" as s3 [*] --> s0 s0 --> s1 s1 --> s2: drain s2 --> s3 s3 --> [*]
Key point: Drain failing hardware.
Why the other options are wrong
- A. Ignore it until the GPU fails
- Waiting risks lost jobs.
- B. Raise its power limit
- That adds stress.
- C. Delete the user's data and restart the job
- That is destructive and unrelated.