LayerNorm (Layer Normalization)

A normalization step inside Transformer blocks that stabilises activations (often applied before each sublayer in modern Pre-LN designs).

How it works

LayerNorm computes the mean and variance of the activations across all features for a single token (not across the batch, unlike BatchNorm), then normalises them to zero mean and unit variance. Learnable scale and bias parameters are applied after normalisation. In modern Pre-LN Transformers, LayerNorm is applied before the attention and feed-forward sublayers (rather than after), which has been found to improve training stability for very deep networks.

Why it matters

Without normalisation, activations in deep Transformers can explode or vanish as they pass through many layers, making training unstable or impossible. LayerNorm is a lightweight but essential component that keeps activations in a stable range, enabling the training of models with dozens to hundreds of layers. Its position (Pre-LN vs Post-LN) and variant (LayerNorm vs RMSNorm) are active areas of architecture design that affect training stability and model quality.