In this article
Quantisation converts model weights from high-precision floats (FP32/FP16) to lower precision (INT8, INT4, NF4). Post-training quantisation (PTQ) applies after training; quantisation-aware training (QAT) incorporates it during training. Tools like GPTQ, AWQ, and bitsandbytes make quantisation accessible. Trade-off: small accuracy loss for 2-4x memory reduction and faster inference.
What it means in practice
Quantisation is not just vocabulary; it is a design handle. Use it as a reference point when comparing architecture choices, debugging implementation trade-offs, or explaining system behaviour to another engineer. It helps teams choose the right model/runtime balance across quality, speed, memory, governance, and cost.
Why engineers care
- It gives teams a shared name for the behaviour, risk, or architecture choice being discussed.
- It helps separate the goal from the implementation detail, so you can compare alternatives instead of copying a tool pattern blindly.
- It creates a useful checklist for reviews: inputs, outputs, failure modes, ownership, cost, latency, and measurement.
Production watch-outs
Benchmarks are only a starting point. Validate with your prompts, data, latency budget, concurrency pattern, and safety requirements.
Related context
Useful neighbouring concepts: Qlora, Inference, Model Serving.