Which resource most limits the size of a model you can load for GPU inference?
Why C is the answer
GPU memory
Model weights and caches must fit in GPU memory.
In detail
- Weights and caches must fit in GPU memory.
- Quantization can help.
Use case
An illustrative scenario, not a real incident.
A model will not load on a GPU, because weights and caches exceed its memory.
See it in different ways
mindmap
root((LLM service))
Hardware
GPU memory
Throughput
Batching
Experience
Streaming
Conversation history
Packaging
NIMflowchart TD
Q{"Do weights and caches fit in GPU memory?"}
Q -->|"Yes"| A0["Load it"]
Q -->|"No"| A1["Choose a smaller option"]Key point: GPU memory limits model size.
Why the other options are wrong
- A. Boot disk capacity
- Disk holds files, not the running model.
- B. Network port count
- Ports do not limit model size.
- D. CPU clock speed
- CPU speed does not limit model size.