KV Cache (Key-Value Cache)

A memory optimization technique used during Transformer inference to store previously computed key and value vectors, avoiding redundant calculations for past tokens.

How it works

During Transformer inference, each attention layer computes Query, Key, and Value vectors for every token. For autoregressive generation, all previous tokens' K and V vectors must be accessible at each new step. Without caching, these would be recomputed every step at O(n²) cost. The KV cache stores all previously computed K and V tensors in GPU memory. Each new token only needs its own Q/K/V computed; it attends to cached K/V from all past tokens. This reduces per-step compute from O(n²) to O(n).

Why it matters

The KV cache is what makes real-time LLM generation feasible. Without it, generating a 2000-token response would require reprocessing all previous tokens on every new step — prohibitively slow. However, the KV cache is also the primary memory bottleneck in LLM serving: for long contexts, it can occupy gigabytes of VRAM per request. Techniques like MLA, MQA, GQA, and quantised KV caches all exist specifically to shrink the KV cache and enable longer contexts and higher batch sizes.