MLA (Multi-Head Latent Attention)

An attention design that compresses key/value caches into a smaller latent vector to cut memory use on long contexts, used in models such as DeepSeek-V3 and Kimi K2.

How it works

Standard multi-head attention stores full K and V vectors for each head and each token in the KV cache. MLA compresses the K and V projections into a single small latent vector c using a down-projection matrix, then recovers the full K and V for attention computation by projecting c back up. Only the compact latent vector c needs to be stored in the KV cache rather than full K and V matrices. RoPE is applied to a decoupled key projection that remains uncompressed, preserving positional information.

Why it matters

MLA was introduced in DeepSeek-V2 and subsequently adopted in DeepSeek-V3 and Kimi K2. It reduces KV cache memory by 5-13x compared with standard MHA, directly enabling longer context windows and higher batch sizes at the same VRAM budget. This memory saving translates to significant inference cost reduction and is one of the key reasons DeepSeek-V3 can serve long contexts efficiently. MLA represents one of the most impactful attention architecture improvements since the introduction of GQA.