Lesson 4.3 · 25 min

BPE Tokenization: How LLMs Split Text into Tokens

How can a model with a fixed list of about 100,000 tokens read any word ever written, including typos, names and emoji it has never seen?

In short: Before a language model can read text, a tokenizer cuts the text into tokens and maps each one to an integer ID. Byte Pair Encoding (BPE) builds that token list by starting from single characters (or bytes) and repeatedly merging the most frequent neighbouring pair, so common words become one token while rare words split into reusable pieces. To tokenize new text it replays the learned merges in order, which is why BPE never meets a truly unknown word.

What is Tokenization?

A neural network only understands numbers. Tokenization is the first step that turns text into numbers: we cut the text into small units called tokens and replace each token with its position in a fixed list called the vocabulary. That position is the token ID.

For example, a tokenizer might turn “unhappiness is rare” into the tokens un happiness is rare and then into IDs such as [403, 23157, 318, 4071] (illustrative numbers). The model reads and writes these IDs; a decoder turns IDs back into text at the end. Note that many tokenizers attach the leading space to the word, so is (with a space) and is are different tokens.

Think of it like LEGO Imagine building every possible house from a box of standard LEGO pieces. If the box has only tiny 1×1 bricks, any house is possible but takes forever to build. If the box has a ready-made piece for every possible house, it would be impossibly large. The best box has common big pieces (whole walls, windows) plus small bricks for unusual details. BPE designs exactly that kind of box for text.

Tokens matter in daily practice: API prices, context-window limits and generation speed are all counted in tokens, not words. In English, one token is roughly three quarters of a word on average, but this varies by tokenizer and language.

The Problem: How to Break Text into Tokens?

There are three obvious ways to cut text, and each has a serious flaw at one extreme.

Pause and think: Why is character-level tokenization expensive for a Transformer even though its vocabulary is tiny?

It makes sequences several times longer. Self-attention compares every token with every other token, so cost grows roughly with the square of sequence length, and the context window fills up with far less actual text.

What is BPE (Byte Pair Encoding)?

Byte Pair Encoding began as a data-compression trick, described by Philip Gage in 1994: find the most common pair of adjacent bytes and replace it with a new symbol, then repeat. In 2016, Sennrich, Haddow and Birch adapted it to build vocabularies for neural machine translation. GPT-2 (2019) popularised byte-level BPE, which starts from the 256 possible byte values instead of characters.

The core idea in one sentence: start with the smallest units and keep gluing together the pair that appears next to each other most often, until the vocabulary reaches the size we want. The result is an ordered list of merge rules plus the vocabulary they create.

• Training corpus: a large sample of text used only to learn the merges.
• Merge rule: an instruction such as “replace e followed by s with es”.
• Vocabulary size: a number we choose in advance; it equals the base symbols plus the number of merges.
• End-of-word marker: in classic BPE a symbol like _ marks where a word ends, so est_ (word ending) differs from est inside a word. Byte-level BPE instead keeps the space as part of the next token.

How BPE Works: Step by Step

Let us train BPE by hand on a tiny corpus with word counts: low ×5, lower ×2, newest ×6, widest ×3. We split each word into characters and add _ at the end: l o w _, l o w e r _, n e w e s t _, w i d e s t _.

One BPE training run

1. Start with characters: The base vocabulary is every character that appears: l, o, w, e, r, n, s, t, i, d and _ (11 symbols). The corpus has 95 symbols in total (word length × count).
2. Count adjacent pairs: Count each neighbouring pair, weighted by word frequency. e s appears in newest (6) and widest (3): 9 times. s t and t _ also appear 9 times. l o appears 7 times.
3. Merge the top pair: Take the most frequent pair, e s (ties are broken by a fixed rule; here, the first one found), add es to the vocabulary, and rewrite the corpus: n e w es t _.
4. Repeat: Recount and merge again: es t → est, then est _ → est_, then l o → lo, lo w → low, n e → ne. Each merge adds one vocabulary entry.
5. Stop at the target size: Stop when the vocabulary hits the chosen size. Real tokenizers run tens of thousands of merges on gigabytes of text. Save the merges in order: that ordered list is the tokenizer.

Code: train BPE and tokenize new words

This is a complete, minimal BPE trainer and tokenizer. It learns six merges from our corpus and then tokenizes three words that were not in the corpus.

bpe_mini.py

from collections import Counter
# Training corpus: word -> how often it appears. "_" marks the end of a word.
corpus = {"low": 5, "lower": 2, "newest": 6, "widest": 3}
words = {tuple(w) + ("_",): f for w, f in corpus.items()}
def pair_counts(words):
pairs = Counter()
for symbols, freq in words.items():
for a, b in zip(symbols, symbols[1:]):
pairs[(a, b)] += freq
return pairs
def merge(words, pair):
out = {}
for symbols, freq in words.items():
s, i = [], 0
while i < len(symbols):
if i < len(symbols) - 1 and (symbols[i], symbols[i + 1]) == pair:
s.append(symbols[i] + symbols[i + 1]); i += 2
else:
s.append(symbols[i]); i += 1
out[tuple(s)] = freq
return out
# Training: repeatedly merge the most frequent adjacent pair
merges = []
for step in range(6):
pairs = pair_counts(words)
best = max(pairs, key=pairs.get)
merges.append(best)
words = merge(words, best)
print(f"merge {step + 1}: {best[0]!r:6} + {best[1]!r:5} (seen {pairs[best]:2}x) -> {best[0] + best[1]!r}")
# Tokenizing NEW text: apply the learned merges in the same order
def tokenize(word):
symbols = {tuple(word) + ("_",): 1}
for pair in merges:
symbols = merge(symbols, pair)
return list(next(iter(symbols)))
for w in ["lowest", "newer", "wider"]:
print(f"{w:7} -> {tokenize(w)}")

Output:

merge 1: 'e'    + 's'   (seen  9x) -> 'es'
merge 2: 'es'   + 't'   (seen  9x) -> 'est'
merge 3: 'est'  + '_'   (seen  9x) -> 'est_'
merge 4: 'l'    + 'o'   (seen  7x) -> 'lo'
merge 5: 'lo'   + 'w'   (seen  7x) -> 'low'
merge 6: 'n'    + 'e'   (seen  6x) -> 'ne'
lowest  -> ['low', 'est_']
newer   -> ['ne', 'w', 'e', 'r', '_']
wider   -> ['w', 'i', 'd', 'e', 'r', '_']

“lowest” was never in the corpus, yet it becomes just two tokens, low + est_, because both pieces were learned from other words. “newer” and “wider” share fewer learned pieces with only six merges, so they fall back to smaller units. With tens of thousands of merges, a real tokenizer would keep them in one or two tokens. Nothing ever becomes “unknown”: the worst case is single characters.

How BPE Tokenizes New Text

Once trained, the tokenizer is fixed. To encode any new text, it does the following.

Pause and think: With our six merges, how would BPE tokenize “slow”? Work it out before reading on.

Start from s l o w _. No merge involves s l, but l o → lo applies, then lo w → low. Result: s + low + _ (3 tokens). The est-related merges do not apply because there is no e s pair.

Order matters Merges must be applied in the order they were learned. Applying them in a different order can produce different tokens than the model saw in training, and the model would then receive unfamiliar ID sequences.

Why BPE is Used in Modern LLMs

• No unknown tokens: byte-level BPE can encode any string, including code, typos, rare names and emoji.
• Efficient sequences: common words and word pieces are single tokens, so a context window holds much more text than with characters.
• Controllable vocabulary size: we pick the number of merges to balance embedding-table size against sequence length.
• Shared pieces: suffixes and prefixes like ing, est and un are reused, helping the model generalise across related words.
• Simple and fast: training is counting and merging; encoding is deterministic and can be implemented very efficiently.

Close relatives exist. WordPiece (used by BERT) picks merges that most increase the likelihood of the training data rather than raw frequency. Unigram tokenization (available in the SentencePiece library) starts with a big vocabulary and prunes it. All three are subword methods with the same goal.

Common mistakes and limits

Tokens are not words A common bug is estimating cost or context limits by counting words or characters. Always count with the model's own tokenizer. Different models use different tokenizers, so the same text can be a different number of tokens for each.

• Language fairness: tokenizers trained mostly on English often split other languages, especially non-Latin scripts, into more tokens per word, which costs more and uses more context.
• Numbers and arithmetic: numbers can be split inconsistently (for example 1234 as 12 + 34 or 123 + 4), which makes digit-level reasoning harder. Some tokenizers split digits individually or in fixed groups for this reason.
• Spelling tasks: the model sees strawberry as a few tokens, not letters, so counting letters is surprisingly hard for it.
• Whitespace sensitivity: “hello” and “ hello” are different tokens; trailing spaces in prompts can change outputs.
• Tokenizer and model must match: never feed IDs from one tokenizer into a model trained with another.
• Glitch tokens: rare strings that got their own token but almost never appeared in model training can trigger odd behaviour.

Going one level deeper

Our hand example started from characters. Byte-level BPE starts one step lower, from bytes. Text on a computer is stored in an encoding called UTF-8, where each character becomes one to four bytes, and each byte is a number from 0 to 255. Plain English letters take one byte. Accented letters take two. Many Asian scripts take three per character, and most emoji take four.

This gives a base vocabulary of exactly 256 entries that covers every possible text. Each merge then takes the next free ID: the first merge becomes 256, the second 257, and so on. GPT-2's 50,257 entries are 256 bytes, 50,000 merges and 1 special end-of-text token.

The round trip, and where it can break

1. Encode: Turn the text into bytes, then replay the merges in learned order. Each merge replaces two neighbouring IDs with one new ID.
2. Decode: Expand every ID back into the bytes it stands for, join the bytes, and read them as UTF-8. Encoding then decoding always returns the original text.
3. Failure: a cut character: A token can hold only part of a multi-byte character. If we decode a slice that ends in the middle of one, the bytes are not valid UTF-8.
4. How to spot it: The symptom is a decode error, or a replacement mark such as � flashing in streamed output. The fix is to hold back incomplete bytes until the next token completes the character.

The table also explains a cost we met earlier. A script that starts at three bytes per character needs many learned merges before it reaches one token per word. If the training corpus had little text in that script, those merges were never learned, and the text stays expensive.

Practice: try it yourself

We will build a tiny byte-level tokenizer: look at the raw bytes of four strings, apply two merges that create token IDs 256 and 257, and decode the IDs back into text.

practice_byte_bpe.py

# Byte-level BPE starts from UTF-8 bytes, so every string has a base encoding.
# Samples: "low", "cafe" with an accent, a two-character Japanese word, an emoji.
samples = ["low", "caf\u00e9", "\u65e5\u672c", "\U0001F600"]
for text in samples:
ids = list(text.encode("utf-8"))   # base token IDs are the byte values 0-255
print(f"{ascii(text):14} chars={len(text)} bytes={len(ids)} ids={ids}")
# Each learned merge gets the next free ID after the 256 byte values.
merges = {(108, 111): 256, (256, 119): 257}   # 'l'+'o' -> 256, then 256+'w' -> 257
def encode(text):
ids = list(text.encode("utf-8"))
for pair, new_id in merges.items():       # replay merges in learned order
out, i = [], 0
while i < len(ids):
if tuple(ids[i:i + 2]) == pair:
out.append(new_id); i += 2
else:
out.append(ids[i]); i += 1
ids = out
return ids
def decode(ids):
table = {i: bytes([i]) for i in range(256)}
for (a, b), new_id in merges.items():     # a merged ID expands to its two parts
table[new_id] = table[a] + table[b]
return b"".join(table[i] for i in ids).decode("utf-8")
ids = encode("slower")
print("encode('slower') =", ids)
print("decode back      =", decode(ids))

Output:

'low'          chars=3 bytes=3 ids=[108, 111, 119]
'caf\xe9'      chars=4 bytes=5 ids=[99, 97, 102, 195, 169]
'\u65e5\u672c' chars=2 bytes=6 ids=[230, 151, 165, 230, 156, 172]
'\U0001f600'   chars=1 bytes=4 ids=[240, 159, 152, 128]
encode('slower') = [115, 257, 101, 114]
decode back      = slower

Now change it:

• Change encode("slower") to encode("lowlow"). Predict first: how many IDs come out, and which ones?
• Add a third merge (101, 114): 258 (that is e + r). Predict the new IDs for “slower” before running.
• Replace the last two prints with print(decode([230, 151])). Predict what happens when we decode only two of the three bytes of a Japanese character.

Pause and think: Suppose we swap the two entries of merges, so (256, 119) is tried before (108, 111). What does encode("slower") return, and why?

It returns 5 IDs: [115, 256, 119, 101, 114]. When the (256, 119) rule runs first there is no 256 in the list yet, so nothing happens. Then l + o becomes 256, but the rule that would glue it to w has already passed. Later merges depend on earlier ones, so order is part of the tokenizer.

Pause and think: The emoji is 1 character but 4 base tokens, while “low” is 3 characters and 3 base tokens. In a real tokenizer a very common emoji often ends up as a single token. What made that happen?

Merges. That emoji's byte sequence appeared often enough in the training corpus that its bytes were merged step by step into one symbol. Nothing about emoji is special-cased: frequent byte sequences get short encodings, rare ones stay long.

Key takeaways

• Tokenization turns text into integer IDs from a fixed vocabulary; models, prices and limits all count tokens.
• Word-level has unknown words; character-level is too long; subword methods like BPE balance both.
• BPE training: start from characters or bytes and repeatedly merge the most frequent adjacent pair.
• BPE encoding: replay the learned merges in order; byte-level BPE can encode any text with no unknown token.
• Tokenizers vary by model and language, so always count tokens with the model's own tokenizer.

Key terms

• Token: A unit of text (word, word piece, character or byte) that the model reads as one ID.
• Vocabulary: The fixed list of all tokens a tokenizer can produce, each with an integer ID.
• Byte Pair Encoding (BPE): A subword method that builds a vocabulary by repeatedly merging the most frequent adjacent pair of symbols.
• Merge rule: One learned instruction to glue two adjacent symbols into a new symbol, applied in learned order.
• Byte-level BPE: BPE that starts from the 256 byte values, so any text can be encoded.
• Out-of-vocabulary (OOV): A word a tokenizer cannot represent; byte-level BPE avoids this.

← 4.2 Autoregressive Models: Predicting One Token at a Time · 4.4 Embeddings: Encoding Meaning as Vectors →