BPE (Byte-Pair Encoding)

A subword tokenization algorithm that iteratively merges the most frequently occurring character or byte pairs, widely used by models like GPT to handle out-of-vocabulary words.

How it works

BPE starts with a character-level vocabulary of all unique characters in the training corpus. It then iteratively finds the most frequent adjacent pair of symbols and merges them into a single new symbol. This process repeats until the vocabulary reaches the target size. The result is a vocabulary where common words are represented as single tokens, while rare words are split into frequent subword pieces. At inference, text is tokenised by greedily applying the learned merge rules.

Why it matters

BPE elegantly solves the vocabulary problem in NLP: a fixed-size word vocabulary cannot represent all words (especially names, technical terms, and new words), but character-level tokenisation produces very long sequences. BPE finds the sweet spot, representing common words compactly while handling novel words via subword decomposition. GPT-2, GPT-3, GPT-4, LLaMA, and Mistral all use BPE or closely related algorithms. The choice of vocabulary size and BPE training corpus directly impacts how efficiently a model processes different languages and domains.