Quantization

The process of reducing the precision of the numbers used to represent a model's parameters (e.g., from 32-bit floats to 8-bit or 4-bit integers) to reduce memory usage and improve inference speed.

How it works

Quantization maps the range of floating-point weight values to a smaller set of integer levels. For 8-bit quantization (INT8), 32-bit floats are mapped to 256 integer values; for 4-bit (INT4), to 16 values. The scale and zero-point parameters define this mapping per-layer or per-group of weights. Post-training quantization (PTQ) applies the mapping after training; quantization-aware training (QAT) simulates quantization during training so weights learn to be more robust to precision loss.

Why it matters

Quantization is the primary technique enabling large models to run on consumer hardware. A 70B-parameter model at FP16 requires ~140GB of VRAM — impossible on a single consumer GPU. At INT4, it requires ~35GB — feasible on a workstation with two or three high-end GPUs. Tools like llama.cpp and GGUF use aggressive 4-bit quantization to run billion-parameter models on laptops and even smartphones, democratising access to powerful AI and enabling local, private inference.