Squinch

A 6-bit block-wise gradient compression algorithm invented by Noam Shazeer at Character.ai to cut multi-node communication bandwidth during large-scale pretraining while limiting reconstruction error.

How it works

During distributed training, gradient tensors must be communicated across GPU nodes via All-Reduce operations. These communications are often the bottleneck at large scale. Squinch compresses gradients to 6 bits per value using block-wise quantisation before transmission, then decompresses after aggregation. The block-wise approach — computing scale factors per small block of values rather than per tensor — limits quantisation error while maintaining the compression ratio. This reduces inter-node communication bandwidth significantly.

Why it matters

Communication bandwidth between nodes is a critical bottleneck in large-scale distributed training: inter-node bandwidth (often 200-400Gbps InfiniBand) is orders of magnitude slower than intra-node NVLink bandwidth. Squinch addresses this by compressing gradient traffic, allowing more nodes to be used efficiently without being limited by network bandwidth. Its invention by Noam Shazeer — one of the original Transformer paper authors — at Character.ai reflects how much innovation is happening outside the most visible frontier labs.