What does INT8 quantization typically trade?
Why C is the answer
Slight quality loss for less memory and faster inference
Eight-bit weights take a quarter of the space of 32-bit weights.
In detail
- Eight-bit weights take a quarter of the space of 32-bit ones.
- Quality may fall slightly.
- Check quality after conversion.
Use case
An illustrative scenario, not a real incident.
INT8 weights take a quarter of the space of 32-bit weights, which is why the model fits, with slight quality loss.
See it in different ways
flowchart TD
A{"Which goal?"} -->|"Fewer bits per weight"| B["Quantization"]
A -->|"Remove low-value weights"| C["Pruning"]
A -->|"Teach a smaller model"| D["Knowledge distillation"]stateDiagram-v2 [*] --> Goal Goal --> Quantization: smaller weights Goal --> Pruning: fewer weights Goal --> Distillation: a smaller student model Goal --> Speculative: fewer main model steps
mindmap
root(("Weight precision"))
n1["FP32"]
n2["32 bits"]
n3["INT8"]
n4["8 bits, a quarter of the space"]Key point: INT8 trades slight quality for size and speed.
Why the other options are wrong
- A. Slower inference in exchange for a smaller vocabulary
- That is not the trade.
- B. A larger context window in exchange for more parameters
- That is not the trade.
- D. More memory use in exchange for better accuracy
- That is the reverse.