Token

The basic unit of text that an LLM processes. A token can be a word, part of a word, or just a character.

How it works

A tokeniser (e.g., BPE, SentencePiece) splits raw text into tokens according to a fixed vocabulary learned during training. Common English words are often single tokens; rare words are split into subword pieces (e.g., 'unbelievable' might be 3 tokens). Numbers and code are tokenised differently — each digit may be its own token. The model converts each token to an integer ID, looks up its embedding, and processes the resulting sequence. All LLM context window limits are measured in tokens, not words.

Why it matters

Tokens are the atomic unit of LLM economics. API pricing is per-token; context window limits are in tokens; generation speed is in tokens per second. Understanding tokenisation helps practitioners reason about costs ('1M tokens ≈ 750K words'), debug unexpected model behaviour (models sometimes 'see' digit-level breakdowns of numbers), and optimise prompts for efficiency. Efficient tokenisation of code and domain-specific text is an active area of model improvement.