Autoregressive

A generation style where each new token is predicted from all previous tokens only. Future tokens cannot change earlier ones.

How it works

In autoregressive generation, a model factorises the probability of a sequence as a product of conditional probabilities: p(x₁,…,xₙ) = Π p(xᵢ|x₁,…,xᵢ₋₁). At each step, the model takes all previously generated tokens as input, runs a forward pass, samples the next token from the output distribution, appends it to the sequence, and repeats. The KV cache prevents re-computing previous tokens' attention. This sequential dependency means generation cannot be parallelised across token positions.

Why it matters

Autoregressive generation is the dominant paradigm for LLMs because its training objective (next-token prediction) is simple, scales perfectly with data, and requires no human labels. Its weakness is inference speed: generating N tokens requires N sequential forward passes, making very long outputs slow. This drives the development of speculative decoding, parallel decoding methods, and diffusion-based language models that aim to generate multiple tokens simultaneously.