Multi-Token Prediction

A training objective where the model predicts several future tokens at once to improve representations; at normal chat inference it still usually emits one token at a time.

How it works

Standard next-token prediction only supervises the model on predicting the immediately next token. Multi-token prediction adds additional prediction heads that simultaneously predict the token two, three, or more steps ahead. These extra objectives force the model to develop richer internal representations that capture longer-range structure. Meta's research showed this improves code and reasoning benchmarks. The extra heads are typically discarded at inference time, or used to enable speculative decoding.

Why it matters

Multi-token prediction is a training efficiency improvement that extracts more learning signal per token seen, effectively making the training data richer without requiring more data. Models trained with it show improved performance on tasks requiring multi-step planning, such as code generation and mathematical reasoning. As a complementary training objective that costs relatively little to add, it represents a practical way to get more quality from the same training compute.