Speculative Decoding

A speed-up where a smaller draft model proposes several tokens and the main model verifies them in parallel, accepting or rejecting the draft.

How it works

A small, fast draft model generates a sequence of k candidate tokens autoregressively. The main target model then processes all k tokens in a single parallel forward pass (which takes approximately the same time as generating one token). It compares its own probability distributions with those the draft model used to generate each token. Tokens where the draft and target agree are accepted; the first disagreement causes rejection, and a corrected token is sampled from the target model. The process then repeats.

Why it matters

Speculative decoding provides wall-clock speedups of 2-3x for text generation without any loss in output quality, since the target model still governs the final output distribution. This is valuable because the decode phase of LLM inference is memory-bandwidth-bottlenecked — the GPU waits for data from VRAM between each token. By verifying multiple tokens per target model pass, speculation amortises this latency. It has been deployed in production by Google, Meta, and others.