Dropout

A regularization technique for reducing overfitting in neural networks by preventing complex co-adaptations on training data.

How it works

During each training step, dropout randomly sets a fraction of neuron activations to zero. Because different random subsets are dropped each step, no single neuron can rely on the presence of specific other neurons. This forces the network to learn redundant representations distributed across many neurons. At inference time, dropout is disabled and all neurons are used, but their outputs are scaled down by the dropout rate to match the expected activation magnitude during training.

Why it matters

Overfitting — where a model memorises training examples instead of generalising — was a major obstacle to training large neural networks. Dropout provided a simple, cheap regularisation technique that dramatically improved generalisation without requiring additional data. It is conceptually equivalent to training an ensemble of exponentially many smaller networks simultaneously, which explains its effectiveness. Though less common in modern Transformers (which use other regularisation methods), it remains widely used in CNNs and MLPs.