Sparse Attention

An attention pattern that only looks at a subset of tokens instead of every pair, reducing the quadratic cost of full attention on long sequences.

How it works

Standard attention computes dot products between every pair of tokens, resulting in O(N²) compute and memory for a sequence of length N. Sparse attention patterns restrict each token to attending only to a carefully chosen subset: local windows of nearby tokens, strided global tokens, or a combination. Sliding window attention, Longformer, BigBird, and Mamba-style linear recurrences are all variants. Some approaches interleave sparse attention layers with occasional global attention layers to maintain long-range coherence.

Why it matters

Quadratic attention is the primary bottleneck preventing standard Transformers from scaling to very long sequences cheaply. Sparse attention breaks this barrier, enabling efficient processing of book-length documents, full codebases, or hours of audio without the memory exploding. While FlashAttention makes dense attention faster, sparse attention is needed for truly extreme lengths (1M+ tokens) where even O(N²) with a small constant is infeasible. Sliding window attention is used in Mistral and Gemini for efficient long-context inference.