Why C is the answer

Speculative decoding

Accepted drafts save main-model steps, which lowers latency.

In detail

• Accepted drafts save steps on the main model.
• Rejected drafts cost some time.

See it in different ways

flowchart LR
  A["Queue"] --> B["First token"]
  B --> C["KV cache reuse"]
  C --> D["Later tokens"]
  E["Continuous batching"] --> A

stateDiagram-v2
  [*] --> Goal
  Goal --> Quantization: smaller weights
  Goal --> Pruning: fewer weights
  Goal --> Distillation: a smaller student model
  Goal --> Speculative: fewer main model steps

sequenceDiagram
  participant P0 as Draft model
  participant P1 as Main model
  P0->>P1: Proposed tokens
  P1-->>P0: Accept or reject

Why the other options are wrong

A. Gradient accumulation across several batches

That is about training gradients.

B. Dynamic padding of the input tokens

That is about inputs.

D. Knowledge distillation at training time

Distillation happens in training.

Related questions

• Question 2Throughput is low because requests are processed one at a time. Which serving technique gives the biggest gain?
• Question 12Many short requests arrive while long ones are running. Which batching method keeps the GPU full?
• Question 13Which NVIDIA library focuses on optimizing LLM inference on its GPUs?
• Question 1You need a model to fit in less GPU memory and run faster, and you accept a small quality loss. What do you apply?

Learn more

• Our lesson: Performance on GPUs
• TensorRT-LLM on GitHub
• NVIDIA NIM