You need a model to fit in less GPU memory and run faster, and you accept a small quality loss. What do you apply?
Why A is the answer
Quantization
Quantization reduces numeric precision of weights, which cuts memory and usually increases speed.
In detail
- Quantization uses smaller numbers, so models take less memory and run faster.
- Measure quality on your own tasks after quantizing.
Use case
An illustrative scenario, not a real incident.
A 16-bit model will not fit on a team's GPUs. Quantizing it to 8 bits halves the weight memory, with a small quality loss.
See it in different ways
flowchart TD
A{"Which goal?"} -->|"Fewer bits per weight"| B["Quantization"]
A -->|"Remove low-value weights"| C["Pruning"]
A -->|"Teach a smaller model"| D["Knowledge distillation"]stateDiagram-v2 [*] --> Goal Goal --> Quantization: smaller weights Goal --> Pruning: fewer weights Goal --> Distillation: a smaller student model Goal --> Speculative: fewer main model steps
flowchart LR S0["Choose a precision"] --> S1["Quantize weights"] --> S2["Measure quality"] --> S3["Deploy"]
Key point: Quantization trades precision for memory and speed.
Why the other options are wrong
- B. Raising the temperature
- Temperature changes variety.
- C. More attention heads
- More heads add memory.
- D. A larger context window
- A larger window uses more memory.