Tokenization

The process of breaking down raw text into smaller pieces (tokens) such as words, subwords, or characters, so they can be processed by a neural network.

How it works

Tokenisation converts raw text strings into sequences of integer IDs using a fixed vocabulary. BPE tokenisers learn a vocabulary by iteratively merging the most frequent adjacent byte or character pairs until the vocabulary reaches a target size (e.g., 50,000 or 128,000 tokens). SentencePiece and tiktoken are common implementations. Special tokens like [BOS] (beginning of sequence), [EOS], [PAD], and role delimiters are added to the vocabulary during training to structure inputs.

Why it matters

Tokenisation directly affects model capability and efficiency. A tokeniser that splits a language inefficiently (many tokens per word) wastes context window space and increases cost. Models tokenise numbers digit-by-digit, making arithmetic reasoning harder. Multilingual tokenisers must balance coverage across hundreds of languages. The choice of tokeniser and vocabulary size is a key design decision with downstream effects on model performance, context utilisation, and the cost of every API call.