LLM Tokenization Explained: BPE, WordPiece, and SentencePiece

Why Not Just Use Characters or Words?

The two naive choices for vocabulary units, individual characters and individual words, both have serious drawbacks that subword tokenization is designed to avoid.

A character-level tokenizer has a tiny vocabulary (256 bytes covers all ASCII and most common content) and can handle any word without ever encountering an unknown token. But it produces very long sequences. "Transformer" becomes 11 tokens instead of 1. Since attention scales quadratically with sequence length and training cost scales linearly with it, character-level models are expensive to train and to run.

A word-level tokenizer produces short sequences and maps naturally to human intuition about language units. But natural language has a very long tail of rare words. A vocabulary that covers 99% of words in English might need 100,000 entries, and it will still encounter out-of-vocabulary words (proper nouns, technical terms, misspellings, new words). Those words are either dropped or replaced with a single UNK token, losing all information about their form.

Subword tokenization finds a middle ground: a vocabulary of 32,000 to 128,000 entries, where common words are single tokens and rare words are split into recognizable subword units. "Transformer" might be one token in the vocabulary, while "Transformer-based" might split into "Transformer", "-", "based". The model can still process any text without unknown tokens.

Byte-Pair Encoding

BPE was originally a data compression algorithm, adapted for NLP by Sennrich et al. (2015) to handle rare words in neural machine translation. The algorithm is greedy and iterative:

1. Start with a vocabulary of individual characters (or bytes).
2. Count the frequency of all adjacent symbol pairs in the corpus.
3. Merge the most frequent pair into a new symbol and add it to the vocabulary.
4. Repeat until the vocabulary reaches the target size.

After 30,000 to 50,000 merge operations, the vocabulary contains single characters, common syllables, common words, and some common multi-word sequences. Rare words are represented as sequences of more common subword units.

Consider how a BPE tokenizer might split the phrase "tokenization is efficient":

Each colored block is one token. The word "tokenization" splits into "token" and "ization" because "ization" is a common suffix that appears in many words (globalization, optimization, regularization) and the merger was learned during BPE training.

WordPiece

WordPiece, used in BERT and related models, is similar to BPE but uses a different criterion for choosing which pair to merge. Instead of the most frequent pair, WordPiece merges the pair that maximizes the likelihood of the training data under the current vocabulary. In practice this produces similar vocabularies to BPE but with slightly different tokenization decisions on rare words.

WordPiece uses a distinctive convention to mark tokens that continue a word: a "##" prefix. So "tokenization" might become "token" + "##ization". This makes it immediately visible in the tokenized output which tokens are word-initial and which are continuations. The ## prefix is absent on the first subword of each word.

SentencePiece

BPE and WordPiece both require pre-tokenized text: they assume that spaces mark word boundaries and work on pre-split word sequences. This is a problem for languages like Chinese, Japanese, or Thai, which do not use spaces to separate words, and also for handling whitespace around punctuation consistently across different text formats.

SentencePiece, introduced by Kudo and Richardson (2018), treats the raw byte stream as the input. It includes spaces as part of the vocabulary and treats them as just another character. The underscore character _ is used to mark the beginning of a word. This makes the tokenization language-agnostic and fully reversible: you can always reconstruct the original text from the token sequence.

Most modern multilingual models (Llama, Mistral, Gemma) use SentencePiece or a variant, because it handles diverse scripts and languages without language-specific preprocessing.

The vocabulary size is not a free choice. It is a tradeoff between sequence length (shorter is cheaper), embedding matrix size (smaller is more memory-efficient), and coverage of rare words and non-English text.

Vocabulary Size Tradeoffs

Every token ID must be embedded into a vector of size d_model. The embedding matrix has shape (V, d_model). With V=32,000 and d_model=4,096 (a common configuration for 7B-parameter models), the embedding matrix alone contains 131 million parameters. With V=128,256 (Llama 3's vocabulary) and the same d_model, it is 525 million parameters, roughly 7.5% of the total parameter count of a 7B model.

A larger vocabulary also means the language model head (the final projection from d_model to V logits) is larger. In most models, the LM head shares weights with the embedding matrix (called weight tying), so doubling the vocabulary doubles the size of this layer at both input and output.

Against this cost, a larger vocabulary reduces average sequence length, which reduces the cost of attention (quadratic in sequence length) and the number of forward passes needed to generate a fixed-length response (linear in the number of tokens generated). For multilingual models, a larger vocabulary is usually worth the parameter cost because it allows common words across many languages to have single-token representations rather than fragmenting into many subword pieces.

Token Embeddings and Positional Encoding

Once a text is converted to a sequence of token IDs, each ID is looked up in the embedding matrix to retrieve a vector of size d_model. These lookup vectors are the learned representation of each token's identity. At initialization they are random; by the end of training they encode semantic and syntactic information about the token's role across many different contexts.

The embedding lookup gives the model information about what each token is, but not where it appears in the sequence. Positional encoding adds that information. The original transformer used fixed sinusoidal functions of the position. Most modern models use learned positional embeddings or a relative encoding scheme like RoPE (Rotary Position Embedding), which encodes position as a rotation in the embedding space and generalizes better to sequences longer than those seen during training.

Practical Implications for Engineers

Understanding tokenization matters when estimating the cost of an API call, which is priced per token. A 1,000-word document is roughly 1,300 to 1,500 tokens in most English-language BPE tokenizers, but a 1,000-character Chinese text might become 600 tokens or 1,800 tokens depending on whether the model uses a vocabulary with extensive Chinese coverage.

Code tokenizes differently from prose. Variable names, indentation, and syntax characters often fragment into many tokens. A function name like "calculate_compound_interest" might become 6 or 7 tokens depending on the tokenizer, because underscores and compound words are handled differently across models. This affects both cost and the model's ability to reason about identifiers.

Prompt engineering interacts with tokenization in subtle ways. Asking the model to write a haiku "in exactly 17 syllables" is asking it to count syllables, which it cannot do directly because it sees tokens, not syllables. Similarly, asking it to count letters in a word fails because individual characters are not always individual tokens.

Byte-Level Fallback and Unicode Handling

Standard BPE tokenizers that operate on characters face a problem with rare Unicode characters and symbols outside their training vocabulary. A character that never appeared during vocabulary construction has no token representation. To solve this, GPT-2 and its successors use a byte-level BPE vocabulary: the base vocabulary consists of 256 byte values rather than characters, so any byte sequence can be encoded, regardless of whether the characters were seen during training.

Byte-level encoding guarantees that no text is unrepresentable, but it can produce very long token sequences for text that contains many unusual Unicode characters. Emoji, for example, often require 3 to 4 bytes each in UTF-8 encoding, and each byte may become a separate token if no merge was learned for that byte sequence. A single emoji might consume 4 tokens that contribute nothing to the model's understanding of the surrounding text.

SentencePiece handles this differently. Its training operates on the raw byte stream with a configurable fallback for rare characters, and it can be configured to treat any character that falls below a frequency threshold as a single byte-level token. This maintains reasonable sequence lengths for common content while preserving the ability to encode any input.

Multilingual Tokenization Challenges

Tokenizers trained primarily on English text perform poorly on other languages. The core problem is vocabulary allocation: if the training corpus is 90% English, the BPE merge operations will predominantly learn English word pieces, leaving little vocabulary capacity for other languages. Chinese, Arabic, and other non-Latin scripts are especially affected because their characters share no visual form with English letters and are unlikely to appear in merged sequences learned from English text.

The consequence is a phenomenon sometimes called "token inflation" for underrepresented languages. A sentence that takes 20 tokens to express in English may require 60 tokens or more in a language that received little vocabulary allocation. Since inference cost scales with token count, this means that users of underrepresented languages pay more per word of useful content than English users do. It also means that the effective context window is shorter for those users, because more tokens are consumed per sentence.

Llama 3 addressed this by expanding its vocabulary from 32,000 tokens (Llama 2) to 128,256 tokens, with a training corpus that included more non-English content. The larger vocabulary allows common words across many languages to have single-token representations, improving both efficiency and model quality for multilingual users. The tradeoff is a larger embedding matrix, as discussed in the vocabulary size section above.

Tokenization and Model Arithmetic Failures

One of the most frequently cited limitations of LLMs is arithmetic: models often make errors on calculations that humans find trivial. Tokenization is a significant contributor to this problem. Numbers are tokenized inconsistently depending on how frequently they appear in the training data. The number "2024" may be a single token, while "2027" is split into "202" and "7" in a vocabulary trained before that year was common in text. The number "1,000,000" may split into "1", ",", "000", ",", "000", losing the structural information that the commas are digit separators.

This inconsistency means the model cannot perform digit-by-digit operations reliably, because the digits are not consistently represented as individual tokens. When a model adds two large numbers, it is not computing in the conventional sense: it is predicting what text typically follows two numbers and an addition sign. For many common arithmetic expressions, this prediction happens to be correct because the training data contains the answer. For novel expressions, the prediction often fails.

Understanding this helps set realistic expectations for numeric tasks. Models handle arithmetic much more reliably when given access to a code interpreter that can actually execute the computation, rather than being asked to predict the digits of the result directly.

The Tokenizer as a Design Choice

The tokenizer is not an afterthought in model design. It is one of the first decisions made and one of the hardest to change after training. Because the model's vocabulary, embedding matrix, and LM head are all fixed at the tokenizer's vocabulary size, switching to a different tokenizer requires retraining the model from scratch. This makes tokenizer selection a high-stakes architectural decision.

The choice of tokenizer affects which prompting strategies work well. Models with large, multilingual vocabularies handle code-switching more naturally. Models with character-informed tokenizers can reason about word morphology more reliably. The absence of certain characters from the vocabulary can produce surprising edge cases, such as models that cannot correctly reproduce a specific emoji or symbol even when it appears in the prompt, because the byte-level representation of that symbol is not merged into a coherent token.

For practitioners building applications on top of existing models, the practical implication is to test tokenization behavior explicitly for the content types your application will handle. Do not assume that a token count from one model's tokenizer will match another model's. Do not assume that a document that fits within the context window in English will also fit when translated to another language. And do not expect reliable character-level manipulation from a model whose tokenizer has merged most common letter combinations into single tokens.

Tokenization and Cost Estimation

API providers charge per token, not per word or per character. This makes tokenization directly relevant to cost management. Accurately estimating the token count of a request before sending it allows you to predict costs, enforce budget limits, and choose between models with different price points at different context lengths.

Most major model providers make their tokenizers available as open-source libraries, allowing exact token count estimation before an API call is made. Tiktoken, the tokenizer used by GPT-4 class models, and the SentencePiece implementations used by Llama-family models can both be run locally with negligible overhead. Building token count estimation into your application's request pipeline avoids surprises from unexpectedly long inputs and allows you to truncate or summarize content before it exceeds the model's context limit.

Cost optimization through tokenization is not just about input compression. The output token count matters equally, because output tokens are typically priced the same as or higher than input tokens. Applications that require the model to produce concise outputs should include explicit length constraints in the prompt, because models left unconstrained tend to generate longer responses than necessary, especially after alignment fine-tuning that has exposed them to annotators who preferred detailed answers. Concise system prompts and targeted instructions reduce both latency and cost without sacrificing answer quality on well-defined tasks.

The tokenizer is one of the most underappreciated variables in LLM system design. Most engineering attention focuses on model size, context length, and inference hardware. But the tokenizer shapes every other variable: it determines sequence length, which drives attention cost and memory requirements; it determines vocabulary coverage, which drives model quality on specific content types; and it determines the granularity at which the model can reason about the structure of the input. Building a robust understanding of tokenization behavior for your specific data is a prerequisite for deploying LLMs predictably and cost-effectively at scale.