MoE (Mixture of Experts)

A machine learning technique where only a subset of a neural network's parameters (the 'experts') are activated for any given input, allowing for massive model scale without a proportional increase in compute cost.

How it works

In an MoE Transformer, the standard dense feed-forward network in each block is replaced by a set of N expert FFNs and a router. For each token, the router computes scores over all experts and selects the top-k (typically 2-8) to process that token. Only the selected experts' weights are used, so the FLOPs per token are k/N of a dense model with the same total parameters. Load balancing mechanisms ensure experts are used roughly equally across the batch.

Why it matters

MoE allows models to have far more total parameters — and thus more knowledge and capacity — while keeping inference compute constant. DeepSeek-V3 has 671B total parameters but uses only 37B per token, achieving frontier quality with the inference cost of a ~37B dense model. This architectural insight is why MoE is used in Gemini, GPT-4, and many other frontier models. It is arguably the single most impactful scaling innovation in modern LLM architecture.