Repetition Penalty

A decoding adjustment that lowers the chance of recently used tokens so the model is less likely to loop or stall on the same phrases.

How it works

At each generation step, the repetition penalty identifies which tokens appear in the recent context and divides their logits by a penalty factor (or subtracts a penalty from them) before the softmax is applied. This reduces the probability of those tokens being selected again, pushing the model toward more diverse vocabulary. The penalty can be applied to all tokens in the context or only to a sliding window of recent tokens, and its strength is controlled by a hyperparameter.

Why it matters

Without repetition penalty, LLMs frequently fall into repetitive loops, endlessly cycling through the same phrases or sentences, especially at higher temperatures or when stuck on a difficult generation. This is particularly problematic in long-form generation tasks. Repetition penalty is a simple, cheap decoding heuristic that significantly improves generation coherence without any retraining, and it is enabled by default in many inference frameworks and APIs.