Quantization-Aware Training (QAT)

Training a model while simulating lower-precision arithmetic (for example INT8) so weights learn to stay accurate after quantization, instead of only quantizing after training.

How it works

QAT inserts simulated quantisation operations ('fake quantisation' nodes) into the training graph: forward pass values are rounded to the target quantisation grid, but gradients still flow through as if in full precision (using the Straight-Through Estimator). The model's weights therefore learn to be robust to the rounding that will occur at inference. Calibration data is used to set the quantisation scale factors. The result is a model that, when fully quantised, retains much higher accuracy than a post-training quantised model.

Why it matters

Post-training quantisation (PTQ) is cheap but can degrade quality at aggressive quantisation levels (INT4 or below). QAT recovers most of this quality by making quantisation part of the training process, producing models that are robust to the precision reduction. This is critical for edge deployment (smartphones, IoT devices) where INT4 or lower is necessary to fit in tight memory budgets. As AI inference moves to the edge, QAT is becoming an essential technique for maintaining quality under extreme compression.