RMSNorm

Root Mean Square Layer Normalization — a lighter normalisation used in many modern Transformers. It scales activations by their RMS without subtracting the mean like classic LayerNorm.

How it works

LayerNorm computes both the mean and variance of each layer's activations and uses them to normalise. RMSNorm simplifies this by only computing the root mean square of the activations and scaling by that: norm(x) = x / RMS(x) × g, where g is a learnable per-dimension scale. Subtracting the mean (re-centring) is omitted. This makes RMSNorm faster to compute than LayerNorm, and empirically, the re-centring appears unnecessary for Transformer training stability.

Why it matters

RMSNorm is used in LLaMA, Mistral, DeepSeek, Gemma, and most modern open-source LLMs. It runs slightly faster than LayerNorm with equivalent training stability and downstream performance, making it the preferred normalisation in modern efficient Transformer designs. Understanding RMSNorm versus LayerNorm is important when comparing model architectures, implementing models from scratch, or debugging normalisation-related issues in custom Transformer code.