FSDP (Fully Sharded Data Parallel)

A distributed training method that shards model parameters, gradients, and optimiser state across GPUs so very large models can train without fitting the full copy on every device.

How it works

In standard data parallel training, each GPU holds a full copy of the model. FSDP shards model parameters, gradients, and optimiser states evenly across all GPUs. During forward and backward passes, each GPU all-gathers the parameters it needs for each layer, computes its portion of the forward/backward pass, then discards the gathered parameters (keeping only its shard). Gradients are reduced and sharded before the optimiser step. This reduces per-GPU memory by a factor equal to the number of GPUs.

Why it matters

FSDP enables training models whose full parameter set would not fit on a single GPU or even a single node. A 70B model at BF16 requires ~140GB for weights alone, plus gradients and optimiser state (often 3x more). FSDP distributes this across 16 or more GPUs, each holding a small fraction. PyTorch's FSDP implementation has made large-scale distributed training accessible to organisations that cannot afford custom distributed training infrastructure, democratising frontier-scale model training.