Ring Attention

A distributed attention method that shards a long sequence across devices in a ring so models can handle much longer contexts than a single GPU’s memory allows.

How it works

Ring Attention partitions a long sequence into chunks and distributes each chunk to a different device. The devices are arranged logically in a ring. Each device holds the Q, K, V of its chunk and computes local attention. Simultaneously, K and V chunks are passed around the ring to adjacent devices, so each device can compute attention against all other chunks by the time it has received each chunk once. The communication is overlapped with computation, making Ring Attention communication-efficient.

Why it matters

Ring Attention removes the single-device VRAM limit as the constraint on maximum context length. A sequence of 1M tokens can be processed by distributing it across 8 or more GPUs, each handling 125K tokens, with the ring protocol enabling global attention. This is how Kimi and other labs achieve million-token contexts in production. As context length has become a key competitive differentiator, Ring Attention and related sequence parallelism techniques have become critical infrastructure components for frontier model serving.