Many short requests arrive while long ones are running. Which batching method keeps the GPU full?
Why D is the answer
Continuous batching
Finished sequences leave and new ones join at each step.
In detail
- Finished sequences leave, and new ones join at each step.
- This suits varied lengths.
Use case
An illustrative scenario, not a real incident.
Short chats arrive during a long report. With continuous batching, they join as soon as a slot frees.
See it in different ways
flowchart LR A["Queue"] --> B["First token"] B --> C["KV cache reuse"] C --> D["Later tokens"] E["Continuous batching"] --> A
sequenceDiagram participant B as Batch participant M as Model participant C as KV cache B->>M: Newest token only C-->>M: Past keys and values M->>C: Adds new keys and values M-->>B: Next token
stateDiagram-v2 state "Sequence running" as s0 state "Sequence finished" as s1 state "New request joins" as s2 [*] --> s0 s0 --> s1: done s1 --> s2: next step s2 --> s0
Key point: Continuous batching keeps the GPU full.
Why the other options are wrong
- A. Static batches that all finish together
- Static batches wait for the longest.
- B. One request at a time, in strict arrival order
- One at a time wastes the GPU.
- C. A separate GPU reserved for each user
- That wastes GPUs.