Batch Normalization

A technique used to make training artificial neural networks faster and more stable by normalizing the inputs of the layers.

How it works

Batch normalisation computes the mean and variance of each feature across the current mini-batch of training examples, then normalises the activations to have zero mean and unit variance. Learnable scale and shift parameters allow the layer to undo that normalisation if the network needs to. By reducing internal covariate shift — the way the distribution of each layer's inputs changes during training — it lets the network train with higher learning rates and less sensitivity to initialisation.

Why it matters

Before batch normalisation, training very deep networks required extremely careful tuning of learning rates and initialisations. Its introduction in 2015 made it possible to train significantly deeper models reliably, directly enabling the architectures that powered the first wave of deep learning breakthroughs in image recognition and speech. Although LayerNorm has largely supplanted it in Transformers, BatchNorm remains essential for CNNs used in computer vision.