DeepSeek-V3

DeepSeek’s MoE language model with about 671B total parameters and 37B activated per token, known for MLA, fine-grained experts, and auxiliary-loss-free load balancing (arXiv:2412.19437).

How it works

DeepSeek-V3 scales the MoE architecture to 671B total parameters while activating only ~37B per token via fine-grained expert routing. It introduces auxiliary-loss-free load balancing — a technique that uses a bias term on routing scores instead of a loss penalty to keep expert utilisation even, preventing the quality degradation associated with auxiliary losses. FP8 mixed-precision training is used throughout to reduce memory and speed up training.

Why it matters

DeepSeek-V3 achieved GPT-4-class performance at a reported training cost of approximately $5.5M — an order of magnitude cheaper than comparable Western frontier models. This shocked the AI industry and caused a significant stock market reaction. It validated the MoE approach for frontier models and demonstrated that compute efficiency innovations could rapidly erode advantages held by the largest AI labs.