RoPE (Rotary Position Embedding)

A mechanism used in modern Transformers to inject positional information into tokens, allowing models to better handle relative distances and extrapolate to longer sequence lengths.

How it works

RoPE encodes the position of each token by rotating its Query and Key vectors by a position-dependent angle in complex number space before computing attention dot products. Because rotations are applied to both Q and K, the dot product between two tokens naturally captures their relative position (the angle between them) rather than their absolute positions. This relative encoding makes RoPE more amenable to length extrapolation than sinusoidal or learned absolute position embeddings.

Why it matters

RoPE is used in LLaMA, DeepSeek, Mistral, and most modern open-source Transformers, having largely replaced sinusoidal position encodings. Its relative encoding property is key to enabling context window extension via YaRN, LongRoPE, and NTK interpolation. Understanding RoPE is necessary for understanding how modern models handle long contexts and why position embedding design choices affect a model's maximum usable sequence length.