Sliding Window Attention

An attention pattern where most layers only attend to a recent window of tokens (for example 1024), often interleaved with occasional global-attention layers to keep long-context ability cheaper.

How it works

In sliding window attention, each token can only attend to the W tokens immediately preceding it in the sequence (the window). This reduces attention's compute and memory from O(N²) to O(N × W). In practice, a few layers in the model use global attention (attending to all tokens) to maintain the ability to integrate information from very distant positions, while the majority use windowed attention for efficiency. Mistral's Sliding Window Attention and Longformer are prominent examples.

Why it matters

Sliding window attention is a practical compromise between the expressiveness of full attention and the efficiency needed for long contexts. At window size 4096, a sequence of 100K tokens requires 25x less attention compute than full attention, with minimal quality loss for tasks where nearby context is most relevant (like code generation or document editing). Understanding sliding window attention is important for reasoning about the context utilisation properties of models like Mistral that employ it.