INT8

8-bit integer precision. Using INT8 for weights, activations, or KV values cuts memory and can roughly double matrix-multiply throughput on many accelerators versus 16-bit floats.

How it works

INT8 quantisation maps floating-point weights or activations to 8-bit signed integers in the range [-128, 127]. A scale factor per tensor (or per row/column for finer granularity) converts back to approximate floating-point values for computation. Modern GPUs and inference accelerators (NVIDIA A100/H100, Qualcomm) have hardware-accelerated INT8 matrix multiplication that is 2x faster and uses 2x less memory than BF16. Calibration data is used to choose optimal scale factors that minimise quantisation error.

Why it matters

INT8 inference is the standard for production LLM serving at scale. It halves VRAM requirements compared to BF16, doubles throughput on compatible hardware, and maintains quality close to the original model for most tasks. Libraries like bitsandbytes, GPTQ, and TensorRT-LLM implement production-ready INT8 inference. Combined with continuous batching, INT8 quantisation is one of the most cost-effective ways to improve the economics of LLM serving without sacrificing meaningful quality.