DeepSeek-V2

Earlier DeepSeek MoE model (about 236B total / 21B active, 128K context) that introduced Multi-Head Latent Attention (MLA) as a headline feature (arXiv:2405.04434).

How it works

DeepSeek-V2 combines a Mixture of Experts architecture (236B total parameters, only 21B active per token) with the newly introduced Multi-Head Latent Attention (MLA). MLA compresses the K and V projections into a small latent vector using low-rank decomposition, then projects back up for each attention head. This dramatically reduces the KV cache memory required for long contexts while maintaining model quality close to full multi-head attention.

Why it matters

DeepSeek-V2 was the paper that introduced MLA to the research community and demonstrated it could substantially reduce inference memory costs without sacrificing quality. MLA has since been adopted in other large models. V2 also showed that massive total parameter counts in MoE architectures are achievable without proportional increases in inference compute, setting a template for efficient large-scale model design.