Throughput is low because requests are processed one at a time. Which serving technique gives the biggest gain?
Why D is the answer
Dynamic or continuous batching
Batching shares the GPU across requests. Continuous batching adds and removes requests between steps, which raises utilization.
In detail
- Batching runs requests in parallel on the GPU.
- Batching raises utilization.
Use case
An illustrative scenario, not a real incident.
A server handles one request at a time, and the GPU idles between steps. Continuous batching keeps it busy.
See it in different ways
flowchart LR A["Queue"] --> B["First token"] B --> C["KV cache reuse"] C --> D["Later tokens"] E["Continuous batching"] --> A
sequenceDiagram participant B as Batch participant M as Model participant C as KV cache B->>M: Newest token only C-->>M: Past keys and values M->>C: Adds new keys and values M-->>B: Next token
flowchart TD
Q{"How are requests processed?"}
Q -->|"One at a time"| A0["Low throughput"]
Q -->|"In batches"| A1["Higher throughput"]Key point: Batching shares the GPU.
Why the other options are wrong
- A. Using a larger tokenizer
- A tokenizer does not add throughput.
- B. Disabling the KV cache
- That slows generation.
- C. Lowering max tokens to one
- One token is useless.