What is the usual effect of batching inference requests on a GPU?
Why A is the answer
Higher throughput, with some added latency
Batches use the GPU in parallel, and requests may wait to join one.
In detail
- Batches use the GPU in parallel.
- Requests may wait a little to join one.
- Tune batch size against latency.
Use case
An illustrative scenario, not a real incident.
Batching requests lets a GPU work on many at once, although each may wait briefly to join.
See it in different ways
flowchart LR A["Users"] --> B["API"] B --> C["Batched GPU inference"] C --> D["Streamed tokens"] B --> E["Latency and error monitoring"]
mindmap
root((LLM service))
Hardware
GPU memory
Throughput
Batching
Experience
Streaming
Conversation history
Packaging
NIMmindmap
root(("Batching"))
n1["Gains"]
n2["Higher throughput"]
n3["Costs"]
n4["Some added latency"]Key point: Batching trades latency for throughput.
Why the other options are wrong
- B. Lower throughput, with lower GPU utilization as well
- That is the opposite.
- C. No change, since GPUs handle only one request
- GPUs handle many requests.
- D. Better answers, because prompts share their context
- Prompts in a batch stay separate.