You type "unhappiness" and the model never sees that word. It sees three pieces: "un", "happi", "ness". Why? Because tokenization turns language into a vocabulary a neural network can manage, and the choice of how to split words has measurable consequences for cost, quality, and multilingual capability. By the end of this module you will understand exactly how BPE works, why vocabulary size is a design tradeoff, and how the model turns those tokens into vectors it can reason over.
Why Not Just Use Words?
The naive approach to representing text is to assign every word its own entry in a vocabulary. The English language has over 170,000 words in active use, and that is before you add technical terms, names, code, URLs, or any other language. A word-level vocabulary large enough to cover real-world text would be enormous. Worse, it would still fail on unknown words. "Tokenization" might be in the vocabulary; "tokenisation" (British spelling) and "tokenize" might not be.
Character-level tokenization fixes the unknown-word problem: every character is always in the vocabulary. But it creates a new problem. The sentence "The cat sat" becomes 11 tokens instead of 3. Sequences become much longer, which is expensive: attention cost scales quadratically with sequence length in the standard implementation. A model trained on character tokens needs to learn much longer-range dependencies to connect meaning.
Subword tokenization is the compromise. Split words into pieces such that common words stay whole, while rare words are broken into recognizable parts. "unhappiness" might become ["un", "happi", "ness"] or ["un", "happiness"] depending on the tokenizer. The model learns representations for these subwords, and can compose representations for words it has never seen by combining familiar pieces.
Byte Pair Encoding (BPE)
BPE was originally a data compression algorithm, adapted for NLP tokenization in Sennrich et al. (arXiv:1508.07909). The algorithm is iterative. Start with a vocabulary of individual characters. Find the most frequent pair of adjacent symbols in the training corpus. Merge that pair into a single new symbol. Add the merged symbol to the vocabulary. Repeat until the vocabulary reaches the desired size.
For example: starting with the text "low low low low lower lower newest newest newest widest widest", the most frequent adjacent pair might be "e" and "s". Merge them into "es". Now "es" is a single token and any word containing "es" will use one token for that pair instead of two. Repeat: find the next most frequent pair in the updated corpus. Continue until you reach, say, 50,000 merges.
The resulting vocabulary contains both individual characters and common subwords. Very common words appear as single tokens. Rare words decompose into familiar pieces. GPT models use a variant of BPE that operates on bytes rather than characters, which handles any Unicode text without special cases. This is why the GPT tokenizer can handle Japanese, Arabic, and emoji, even though those require multiple bytes per character.
WordPiece and SentencePiece
WordPiece, used in BERT and its derivatives, is similar to BPE but uses a different merge criterion. Rather than merging the most frequent pair, it merges the pair that maximizes the likelihood of the training data under a language model. In practice, this often produces similar tokenizations to BPE, but WordPiece's greediness in the merge step gives it different behavior on rare words.
A distinctive feature of WordPiece is its prefix notation. Word-initial subwords have no special marker. Subwords that appear inside a word are prefixed with "##". So "tokenization" might become ["token", "##ization"]. This allows a model to know, from the token itself, whether it starts a word or continues one.
SentencePiece (Kudo and Richardson, arXiv:1808.06226) takes a different approach. Rather than starting from words, it treats the entire input text as a raw stream of Unicode characters, including whitespace. Whitespace is represented by a special character (usually the "▁" symbol). This makes SentencePiece language-agnostic: it works equally well on languages that do not use spaces between words, like Japanese, Chinese, and Thai. It is the tokenizer used in many multilingual models including models in the T5 and LLaMA families.
Why Vocabulary Size Is a Design Tradeoff
Vocabulary size is not a free variable. It touches compute, quality, and cost simultaneously.
A larger vocabulary means fewer tokens per piece of text. "tokenization" is one token, not three. Sequences are shorter, which is good: shorter sequences are cheaper to process (attention cost is quadratic in sequence length), and the model needs to model fewer long-range dependencies. The embedding matrix is larger (vocabulary size times d_model), but this cost is usually smaller than the savings from shorter sequences.
A smaller vocabulary produces longer sequences for the same text. It also means more token collisions: similar strings might map to the same token, losing distinctiveness. On the other hand, every token in the vocabulary has been seen more frequently during training, so its representation is better learned. With a very large vocabulary, rare tokens may appear only a handful of times in training data, resulting in poor embeddings.
GPT-4's tokenizer uses a vocabulary of roughly 100,277 tokens. This is a deliberate choice: large enough that common English words and code constructs are single tokens, small enough that each token appears often enough in training to learn a good embedding. Multilingual models often need larger vocabularies to cover many scripts without excessive fragmentation of non-English text.
Practical implication
Non-English text and code consume more tokens than equivalent English prose in many models. A 1,000-word essay in Japanese will use more tokens than the same essay in English, because many Japanese characters require multiple tokens under a vocabulary primarily built for English. This is why vocabulary size and composition are active research areas for multilingual models.
Token Embeddings: From Integers to Vectors
After tokenization, each token is an integer index. "the" might be token 262. "transformer" might be token 47221. These integers are fed into an embedding lookup table: a matrix of shape (vocabulary_size, d_model). Row 262 is the learned vector for "the". Row 47221 is the learned vector for "transformer".
The embedding matrix is initialized randomly and updated by gradient descent during training. After training, tokens with similar meanings or grammatical roles cluster together in the d_model-dimensional space. This is not programmed in; it emerges from the model minimizing prediction error. The word "happy" will end up close to "joyful" because they appear in similar contexts in the training data, and similar contexts produce similar gradient updates.
These embeddings capture relationships that surprised researchers when they were first studied. The famous example from Mikolov et al. (arXiv:1301.3781) on word2vec: the vector for "king" minus "man" plus "woman" is close to the vector for "queen". Linear algebra on token embeddings reflects analogical relationships in language. Modern LLM embeddings are richer and higher-dimensional, but the same principle applies.
Positional Embeddings: Adding Location
Token embeddings alone do not tell the model where a token appears in the sequence. The sentence "The dog bit the man" and "The man bit the dog" contain the same tokens in different order. Without positional information, the transformer would produce identical representations for both.
The original transformer added positional encodings using sine and cosine functions of different frequencies: position p and dimension i get PE(p, 2i) = sin(p / 10000^(2i/d_model)) and PE(p, 2i+1) = cos(p / 10000^(2i/d_model)). These fixed encodings allow the model to compute relative distances between positions from their encoded values.
Most modern models use learned positional embeddings: a separate embedding table with one vector per position, learned during training. GPT-2 uses learned positional embeddings up to a maximum sequence length (1,024 for GPT-2 small). More recent models use rotary positional embeddings (RoPE) or ALiBi, which encode relative rather than absolute position and generalize better to sequences longer than those seen in training.
Python · Simple BPE Implementation
from collections import Counter
def get_pairs(vocab):
"""Count all adjacent symbol pairs in the vocabulary."""
pairs = Counter()
for word, freq in vocab.items():
symbols = word.split()
for i in range(len(symbols) - 1):
pairs[(symbols[i], symbols[i+1])] += freq
return pairs
def merge_pair(pair, vocab):
"""Merge the most frequent pair in all words."""
merged = ' '.join(pair)
new_vocab = {}
for word, freq in vocab.items():
new_word = word.replace(merged, ''.join(pair))
new_vocab[new_word] = freq
return new_vocab
# Starting vocabulary: characters + end-of-word marker
vocab = {
'l o w </w>': 5,
'l o w e r </w>': 2,
'n e w e s t </w>': 6,
'w i d e s t </w>': 3,
}
print("Initial vocabulary:", list(vocab.keys()))
# Run 5 BPE merges
for i in range(5):
pairs = get_pairs(vocab)
if not pairs:
break
best = pairs.most_common(1)[0][0]
vocab = merge_pair(best, vocab)
print(f"Merge {i+1}: {best} -> {''.join(best)}")
print(f" Updated: {list(vocab.keys())}")
Try this
Go to platform.openai.com/tokenizer and paste in three versions of the same content: a paragraph in English, the same paragraph translated to another language, and a block of Python code. Compare the token counts. You will find that code and non-English text typically use significantly more tokens per character than English prose. This is a direct consequence of vocabulary composition: most tokens in GPT tokenizers correspond to common English subwords.