Model Parallelism

Splitting a single large model across devices (for example tensor, pipeline, sequence, or expert parallelism) so training or inference can exceed one GPU’s memory.

How it works

Model parallelism comes in several forms. Tensor parallelism splits individual weight matrices across GPUs (e.g., each GPU holds half the attention heads). Pipeline parallelism assigns different Transformer layers to different GPUs, with activations passed between them. Sequence parallelism (Ring Attention) distributes the sequence dimension. Expert parallelism in MoE models places different experts on different GPUs. These strategies are often combined — 3D parallelism (tensor + pipeline + data) is used to train the largest models.

Why it matters

Model parallelism is the core infrastructure challenge of frontier AI. A 671B-parameter model at BF16 requires ~1.3TB of VRAM for weights alone — far beyond any single GPU. Distributing the model across hundreds of GPUs while maintaining efficient utilisation requires sophisticated parallelism strategies and custom communication libraries. The engineering teams that master model parallelism have a decisive advantage in training the next generation of frontier models, which is why it remains one of the most specialised and valued skills in AI infrastructure.