bfloat16 (BF16)

A 16-bit floating-point format widely used in deep learning. It keeps the same exponent range as FP32 so training stays stable, while using less memory than full precision.

How it works

BF16 uses 1 sign bit, 8 exponent bits (same as FP32), and 7 mantissa bits (versus 23 in FP32). Retaining the full exponent range means BF16 can represent the same magnitude of numbers as FP32, including very large gradients that would overflow FP16 (which uses only 5 exponent bits). Weights and activations are stored and computed in BF16, while loss scaling (common in FP16 training) is often unnecessary. Modern GPUs and TPUs include native BF16 tensor cores.

Why it matters

BF16 is the preferred mixed-precision format for LLM pre-training because it combines FP32's dynamic range with FP16's memory efficiency — without FP16's notorious gradient overflow problem that requires loss scaling workarounds. Virtually every frontier model is now pre-trained in BF16. Understanding BF16 explains why model files are commonly ~2 bytes per parameter (e.g., a 7B model in BF16 is ~14GB) and why training on hardware without BF16 support (older GPUs) is significantly less stable.