A pod requests a GPU and stays Pending, although GPUs are physically present. What do you check first?
Why C is the answer
The GPU device plugin and the node's advertised GPU resources
If the node does not advertise nvidia.com/gpu, the scheduler cannot place the pod.
In detail
- Pending means it was not scheduled, so check what the node reports.
- Check what the node advertises.
Use case
An illustrative scenario, not a real incident.
A pod is Pending, and the node advertises no nvidia.com/gpu, because the device plugin stopped.
See it in different ways
flowchart TD
A["Pod Pending"] --> B["kubectl describe pod"]
B --> C{"Does the node report GPUs?"}
C -->|No| D["Check operator and device plugin"]
C -->|Yes| E["Check quotas, taints and selectors"]stateDiagram-v2 [*] --> Pending Pending --> NoGpuReported: node shows no GPUs Pending --> Taint: no matching toleration Pending --> WrongRequest: no nvidia.com/gpu limit NoGpuReported --> CheckOperator Taint --> AddToleration WrongRequest --> FixPodSpec
flowchart TD
Q{"Does the node advertise nvidia.com/gpu?"}
Q -->|"No"| A0["Fix the device plugin"]
Q -->|"Yes"| A1["Check other constraints"]Key point: Nodes must advertise GPUs.
Why the other options are wrong
- A. The CUDA version inside the container image the pod uses
- The container has not started.
- B. The application's own logs from inside the container
- The container has not started.
- D. The Slurm partition that the node belongs to
- Slurm is not scheduling this pod.