Top-K Sampling

A decoding strategy that restricts the AI model to only sample from the K most likely next tokens, cutting off long-tail probabilities to reduce nonsensical outputs.

How it works

After computing logits and applying softmax (and temperature scaling), top-k sampling retains only the K highest-probability tokens and sets all others to zero probability. The remaining K tokens' probabilities are renormalised and a token is sampled from them. Common values are K=40 to K=100. A very small K (e.g., K=1) is equivalent to greedy decoding. Top-k is often combined with top-p sampling, where the effective token pool is the intersection of the top-k and nucleus sets.

Why it matters

Without truncation of the token distribution, low-probability tokens (gibberish, inappropriate content, off-topic vocabulary) can occasionally be sampled and derail generation. Top-k prevents this by guaranteeing that only reasonably likely tokens are chosen. It is a blunt but reliable safety net against incoherent outputs. Combined with top-p and temperature, it gives practitioners three complementary controls over the creativity-vs-coherence trade-off in text generation.