YaRN

A method for extending Rotary Position Embeddings so a model trained on shorter sequences can handle much longer contexts at inference time.

How it works

RoPE encodes position by rotating token embeddings in frequency space. YaRN (Yet another RoPE extensioN) extends the maximum position by interpolating the frequencies used in RoPE — stretching the position encoding to cover a longer range — while applying different interpolation factors to different frequency components (NTK-aware interpolation). A small amount of continued pre-training on longer sequences after applying YaRN allows the model to adapt and perform well at the extended context length.

Why it matters

Training a model with a long context window from scratch is enormously expensive. YaRN and related methods (LongRoPE, ABF) allow existing models to be extended cheaply, enabling community-built long-context versions of LLaMA and Mistral models without full retraining. As context windows have become a key competitive dimension — with Claude and Gemini supporting millions of tokens — efficient context extension techniques have become a critical tool in the open-source LLM ecosystem.