Overfitting

A modeling error which occurs when a function is too closely fit to a limited set of data points, performing poorly on new data.

How it works

Overfitting occurs when a model is too complex relative to the amount of training data. Instead of learning the underlying patterns, it memorises specific training examples including their noise. This results in very low training loss but high validation loss — the classic signature of overfitting. Prevention techniques include regularisation (L1/L2 penalties, dropout), data augmentation, early stopping (halting training when validation loss starts rising), and collecting more training data.

Why it matters

Overfitting is the most fundamental tension in machine learning: a model must be complex enough to capture real patterns but not so complex that it memorises noise. Understanding and preventing it is a core ML engineering skill. For large language models, overfitting manifests subtly during fine-tuning — a model trained too long on a small fine-tuning dataset may lose its general capabilities and become brittle on inputs slightly outside the training distribution.