Perplexity

A measurement of how well a probability distribution or language model predicts a sample. Lower perplexity indicates the model is better at predicting the sequence.

How it works

Perplexity is the exponent of the average negative log-probability the model assigns to each token in a test set: PPL = exp(-(1/N) Σ log p(tᵢ|t₁…tᵢ₋₁)). Intuitively, it measures the effective branching factor — how many equally likely next tokens the model considers at each step. A perplexity of 10 means the model is, on average, choosing among 10 equally likely options. Lower perplexity means the model is more certain and accurate about the next token.

Why it matters

Perplexity is the standard intrinsic evaluation metric for language model quality during pre-training. Tracking perplexity on a held-out validation set throughout training is how researchers verify that training is progressing. It is also used to evaluate quantisation quality — a model that retains low perplexity after quantisation has lost little capability. However, lower perplexity does not always correlate with better performance on downstream tasks, which is why benchmarks (MMLU, HumanEval) are used alongside perplexity for model comparison.