Top-P (Nucleus Sampling)

A decoding method where the model only considers the smallest set of tokens whose cumulative probability exceeds the probability P, allowing for dynamic vocabulary selection.

How it works

After computing the softmax distribution over the vocabulary, Top-P sorts tokens by descending probability and finds the smallest set (nucleus) whose cumulative probability exceeds the threshold P. Only tokens in this nucleus are eligible for sampling; the rest have their probabilities set to zero. The probabilities of selected tokens are renormalised before sampling. Because the nucleus size varies based on the distribution, top-p adapts: when the model is confident (peaked distribution), few tokens are considered; when uncertain, more are included.

Why it matters

Top-P is the most commonly recommended sampling method for creative and conversational generation. Unlike top-k (fixed number of candidates regardless of confidence), top-p adapts the candidate pool to the model's uncertainty at each step. This avoids the failure mode where top-k always includes unlikely tokens when the model is confident, or too few when the distribution is flat. In practice, top-p of 0.9 or 0.95 combined with temperature of 0.7-1.0 is the default configuration in most production chatbots.