Context Window

The amount of text (measured in tokens) that a large language model can consider at one time when generating a response.

How it works

Every time an LLM generates a response, the entire conversation history — system prompt, user messages, and assistant replies — is bundled into a sequence of tokens and passed through the model. The context window is the maximum length of this sequence. Tokens beyond the limit are simply cut off and the model has no access to them. Extending the context window requires architectural changes (like RoPE with YaRN), more memory for the KV cache, and careful training on long sequences.

Why it matters

Context window size is a key practical constraint for many real-world applications. Processing long documents, maintaining multi-session memory, performing multi-step code generation, or analysing large codebases all require fitting large amounts of text into a single context. The arms race to extend context windows — from 4K to 128K to 1M tokens — reflects how central this limit is to unlocking new categories of AI application.