Attention Sink

A pattern where early tokens (often the system prompt) keep receiving disproportionate attention across a long generation, helping the model stay anchored to core instructions.

How it works

During autoregressive generation, each new token's attention layer looks back at all previous tokens. Researchers have observed that certain early tokens — particularly the very first token or the beginning of the system prompt — tend to accumulate disproportionately high attention scores across most layers. This appears to act as a 'sink' that absorbs attention weight that doesn't need to go anywhere meaningful, helping the model avoid errors in distributing attention over very long sequences.

Why it matters

Understanding attention sinks helps engineers design better context management strategies. When extending context windows or streaming very long conversations, knowing which tokens act as anchors informs decisions about what to keep in the KV cache and what to evict. It also sheds light on why truncating the very beginning of a prompt can sometimes degrade output quality more than truncating the middle.