Underfitting

When a machine learning model cannot adequately capture the underlying structure of the data.

How it works

Underfitting occurs when a model is too simple to represent the complexity of the data, or when it hasn't been trained long enough. A linear model trying to fit a non-linear relationship will underfit. Symptoms are high training loss and high validation loss simultaneously. Solutions include increasing model capacity (more layers, more parameters), training for more epochs, reducing regularisation, adding more informative features, or switching to a more expressive model architecture.

Why it matters

Underfitting and overfitting are the two fundamental failure modes of machine learning, and diagnosing which is occurring is the first step in any debugging process. A model that underfits is undeployable; it hasn't learned the task. For LLMs, underfitting manifests during pre-training as a model that hasn't converged to low loss, or during fine-tuning as a model that still doesn't follow the target behaviour after training. Scaling laws predict how much compute is needed to avoid underfitting for a given model size and dataset.