Prefill and Decode

Two phases of Transformer inference: prefill processes the full prompt (often in parallel), then decode generates new tokens one by one using the KV cache.

How it works

In the prefill phase, all tokens in the input prompt are processed simultaneously in parallel, computing and storing key and value vectors for every token in the KV cache. This phase is compute-bound and fast. In the decode phase, the model generates one new token at a time, attending back to all previous tokens via the KV cache (which avoids recomputing previous keys and values). Decode is memory-bandwidth-bound — the bottleneck is reading the KV cache from GPU memory.

Why it matters

Understanding the prefill-decode split is essential for optimising LLM serving. Long prompts consume significant GPU time during prefill (affecting TTFT), while long output sequences consume memory bandwidth during decode (affecting TPOT). Techniques like speculative decoding, continuous batching, and chunked prefill all target specific bottlenecks in this pipeline. For cost-sensitive applications, the ratio of prompt length to output length determines which phase dominates inference cost.