Vocabulary Size, Merge Order, and Fertility

In the previous post we looked at how different tokenization algorithms carve text into subword units. We glossed over a question that matters more than the algorithm choice itself: how big should the vocabulary actually be?

This is a hyperparameter that quietly shapes everything downstream: sequence length, embedding table size, how well your model handles languages other than English, even how tensors get laid out across GPUs.

I want to walk through the three-way tradeoff that vocabulary size creates, introduce a metric called fertility that makes the consequences measurable, and then look at what recent scaling law research says about how to actually pick a number.

Prerequisites: BPE from the BPE post and a sense of the WordPiece/Unigram/SentencePiece algorithms from the previous post. A rough feel for the embedding table helps but we preview the calculation here.

The three-way tradeoff

When you increase vocabulary size, three things shift at once. Two of them help you, one costs you.

Sequences get shorter. If your tokenizer knows the word "unfortunately" as a single token, that's one position in the sequence. A smaller vocabulary might split it into "un", "for", "tun", "ately" and now you've consumed four positions to say the same thing. Shorter sequences mean less work for self-attention, which scales super-linearly with sequence length.

The embedding table grows too, and by more than you'd guess. It has one row per token in the vocabulary, each a dense vector of dimension $d_{\text{model}}$, so the parameter count for the embedding layer is:

For a model with $d_{\text{model}} = 4096$, a 32k vocabulary means roughly 131 million embedding parameters. A 200k vocabulary means about 819 million, a 6× increase for a single lookup table.

And the training signal per token gets sparser. With 200k entries, each individual token appears less frequently in the training corpus than it would in a 32k vocabulary. Rare tokens get fewer gradient updates, so their embeddings stay closer to random initialization. The tail of the vocabulary is, in a real sense, undertrained.

Where does the optimum sit? For most of the field's history, people just picked a round number and moved on.

Merge order: what gets learned first

The BPE post covered how training builds a ranked merge list and how encoding replays it greedily in that order, so two tokenizers with the same rules but different priority can split the same word differently. One consequence of that ranking matters for vocabulary size specifically: the merge list is implicitly ordered by frequency, from universal, language-agnostic pairs like th and er down to increasingly rare, domain-specific ones. Setting the vocabulary size means drawing a line in that ranked list: everything above it becomes a dedicated token, everything below gets decomposed at encoding time.

This is why vocabulary size has such a disproportionate effect on non-English languages. In a tokenizer trained primarily on English, English subwords dominate the early merges. Korean, Arabic, and Hindi subwords only start appearing in later merges. A 32k vocabulary might capture most English words as single tokens while fragmenting Korean words into 3–5 pieces each.

Fertility: the diagnostic

There's a clean way to measure how well a tokenizer handles a given language, and it's called token fertility. The idea is simple:

A fertility of 1.0 means each word maps to exactly one token. A fertility of 3.0 means each word gets broken into three tokens on average. Lower is better. You can also measure fertility in tokens per character, which is more meaningful for languages like Chinese that do not use whitespace between words, but tokens-per-word is the most intuitive starting point.

For English text tokenized with a well-trained BPE vocabulary, fertility sits around 1.1–1.3. Most common English words are single tokens. Short function words ("the", "and", "is") are always single tokens. Occasionally a longer or rarer word gets split into two pieces.

For other languages, the picture is dramatically different. Rust et al. (2020) measured tokenizer fertility across dozens of languages and found that English-centric tokenizers routinely produce fertility values of 2.5–4.0 for languages like Telugu, Myanmar, and Khmer. Those languages need 2–4× more tokens to express the same content, which translates directly into 2–4× the inference cost and 2–4× fewer words fitting in the context window.

Toggle the vocabulary sizes below to see how fertility shifts:

The pattern is consistent. Larger vocabularies narrow the gap, but they do not eliminate it. Even at 200k tokens, Hindi requires roughly 1.7× the tokens that English does for equivalent content. This is what some researchers call the "multilingual tax," and the gap has been persistent.

The Frontiers in AI team measuring Ukrainian tokenization efficiency found that GPT-4's tokenizer (cl100k_base, ~100k vocabulary) produced a fertility of about 1.9 for Ukrainian text, compared to 1.1 for English. Ukrainian speakers are effectively paying almost double per API call, and they can fit roughly half as much Ukrainian text into the context window.

A worked example

Let me make this concrete. Take a short English paragraph:

The quick brown fox jumps over the lazy dog near the riverbank.

With a 32k vocabulary (similar to Llama 2's SentencePiece tokenizer), this becomes roughly 13 tokens. "Riverbank" might get split into "river" + "bank", but most words survive intact. Fertility is about 1.08.

With a 100k vocabulary (GPT-4's cl100k_base), it's 12 tokens. "Riverbank" is now a single token. Fertility drops to 1.0.

With a 200k vocabulary, it's still 12 tokens. For this particular sentence, we've already hit the floor since every word is common enough to have its own token at 100k. The gains from 200k show up more on technical or multilingual text.

Now take the same meaning in Hindi:

तेज़ भूरी लोमड़ी सुस्त कुत्ते के पास नदी किनारे कूदती है।

With a 32k English-centric tokenizer, this explodes to 35–40 tokens. Each Devanagari word gets fragmented into individual Unicode byte sequences. Fertility is around 3.5–4.0.

With 200k, it drops to maybe 18–20 tokens. Still more than English, but substantially better. Here's Python code that makes this comparison reproducible:

import tiktoken
 
# GPT-4's tokenizer (100k vocab)
enc_100k = tiktoken.get_encoding("cl100k_base")
# GPT-4o's tokenizer (200k vocab)
enc_200k = tiktoken.get_encoding("o200k_base")
 
def measure_fertility(text: str, encoding) -> dict:
    """Compute token fertility for a piece of text."""
    tokens = encoding.encode(text)
    words = text.split()
    word_count = max(len(words), 1)
    return {
        "tokens": len(tokens),
        "words": word_count,
        "fertility": len(tokens) / word_count,
    }
 
english = "The quick brown fox jumps over the lazy dog near the riverbank."
spanish = "El rápido zorro marrón salta sobre el perro perezoso cerca del río."
arabic  = "الثعلب البني السريع يقفز فوق الكلب الكسول بالقرب من ضفة النهر."
 
for label, text in [("English", english), ("Spanish", spanish), ("Arabic", arabic)]:
    r100 = measure_fertility(text, enc_100k)
    r200 = measure_fertility(text, enc_200k)
    print(f"\n{label}: {r100['words']} words")
    print(f"  cl100k: {r100['tokens']} tokens, fertility = {r100['fertility']:.2f}")
    print(f"  o200k : {r200['tokens']} tokens, fertility = {r200['fertility']:.2f}")
 
# Output:
# English: 12 words
#   cl100k: 14 tokens, fertility = 1.17
#   o200k : 14 tokens, fertility = 1.17
# Spanish: 12 words
#   cl100k: 21 tokens, fertility = 1.75
#   o200k : 18 tokens, fertility = 1.50
# Arabic: 11 words
#   cl100k: 44 tokens, fertility = 4.00
#   o200k : 22 tokens, fertility = 2.00

English barely changes between vocabulary sizes because it's already well-served at 100k. Arabic sees a major improvement going from 100k to 200k, because the larger vocabulary has room for more Arabic subword units. Larger vocabularies help underrepresented languages more than they help English.

The scaling law for vocabulary

For a long time, vocabulary size was treated as a minor design decision. Llama 1 and 2 shipped with 32k tokens. GPT-2 used about 50k. People debated whether 32k or 64k was better, but nobody had a principled way to choose.

Then Tao et al. (2024) published "Scaling Laws with Vocabulary" at NeurIPS, and the answer turned out to be surprisingly clean. They found that optimal vocabulary size scales as a power law with compute budget:

where $C$ is the total compute budget (in FLOPs). As you train larger models on more data, the optimal vocabulary grows substantially.

The intuition for why this works: bigger models have more capacity in their embedding layers and process more training tokens, so the "sparse training signal" problem from a larger vocabulary becomes less severe. A 70B model sees enough data during training to adequately learn embeddings even for its 100,000th most common token. A 7B model might not.

To make the exponent concrete: Llama 2-7B's training budget was roughly $C \approx 6 \times N \times D \approx 6 \cdot (7\!\times\!10^9) \cdot (2\!\times\!10^{12}) \approx 8.4 \times 10^{22}$ FLOPs. Llama 2-70B is about 10× bigger and trained on the same 2T tokens, so $C \approx 8.4 \times 10^{23}$. Plugging into $V_{\text{opt}} \propto C^{0.36}$, a 10× jump in compute asks for a $10^{0.36} \approx 2.3$× jump in vocabulary. Anchoring the 7B point near its 32k vocab, the 70B would want something closer to $\sim$75k. Llama 2-70B shipped with the same 32k instead, leaving it undersized for its compute class.

This paper helped explain a trend that was already emerging in practice. GPT-4 moved to 100k tokens. GPT-4o pushed to 200k. Gemma 2 adopted 256k. The frontier labs had likely run their own internal scaling experiments and converged on the same conclusion: larger models should have larger vocabularies.

Practical engineering considerations

There are a handful of implementation details around vocabulary size that matter a lot in production but rarely show up in tutorials.

Divisibility and padding. When you shard a model across multiple GPUs using tensor parallelism, the embedding matrix gets split column-wise across devices, so if the vocabulary size isn't evenly divisible by the GPU count, you need padding, which wastes memory. GPU matmul kernels also want dimensions that are multiples of 64 or 128. This is why you see vocabulary sizes like 32,000, 128,256, or 256,000 rather than a round number with a few special tokens tacked on: Llama 3's jump from 32k to 128,256 tokens was chosen to divide cleanly across 8-way tensor parallelism while also landing on a multiple of 128 for kernel efficiency. Frameworks that don't hit a clean number pad silently, and those padded rows still consume memory (as we discussed in Post 1.7).

The cost of adding tokens. Extend a pretrained model's 32k vocabulary with 1,000 domain-specific tokens and the new embedding rows start out randomly initialized, unrelated to the pretrained representations. Fine-tuning, at least on the new embeddings, is required before they're useful.

The output layer mirrors the input. The embedding table is not the only place vocabulary size shows up. The unembedding (output projection) layer also has shape $d_{\text{model}} \times V$ because the model produces a probability distribution over the entire vocabulary at every token position. So the parameter cost of vocabulary size is actually doubled:

For a 200k-vocabulary model at $d_{\text{model}} = 4096$, that's about 1.6 billion parameters tied up just in the vocabulary layers. For a 7B model, that's over 20% of total parameters. For a 70B model, it's about 2%. So larger models can more easily absorb the cost of larger vocabularies.

The long tail

One last thing about merge order that connects to the next post on the embedding table.

The tokens created by early merges (common byte pairs, frequent English subwords) appear millions of times in the training corpus. Their embedding vectors receive millions of gradient updates and end up well-positioned in the embedding space, with rich geometric relationships to nearby tokens.

The tokens created by very late merges (rare technical terms, uncommon scripts) might appear only thousands of times. Their embeddings are noisier and less reliable. In the worst case, a very rare token's embedding is essentially still close to its random initialization.

Make the vocabulary too large for the model size and training budget, and you get a long tail of tokens with effectively garbage embeddings, tokens that would have been better off staying split into well-trained subwords.

This is what Tao et al.'s scaling law is really capturing. There's a sweet spot where you have enough vocabulary to compress sequences efficiently but not so much that you're spending parameters on poorly-trained embeddings. That sweet spot scales with model size and training data.

Misconceptions

"Bigger vocab is always better." Larger vocabularies trade shorter sequences for more parameters and sparser per-token training signal. Past the scaling-law sweet spot, you're burning parameters on a long tail of rare tokens whose embeddings never get trained to a useful state, tokens that would be better off split into well-trained subwords. Bigger only helps if your model and training budget are large enough to fill the extra rows.

"Vocabulary size is a capacity number." It's closer to a cut position in a frequency-ranked list. A 32k tokenizer doesn't have "room for 32k words." It has the first 32k merges that beat the BPE frequency threshold. A tokenizer trained primarily on English treats every extra 1k slot as "one more English subword," and only picks up non-Latin scripts once the English tail is exhausted.

"Fertility only matters for non-English languages." Fertility is the cleanest cross-tokenizer comparison metric we have, and it moves for English too. Smaller vocabularies fragment medical terms, code identifiers, URLs, and long domain-specific words. The non-English numbers are the most dramatic, but if you're deploying an LLM in a narrow domain (law, medicine, genomics), measuring fertility on your actual corpus is the first thing I'd do before committing to a base model.

"Adding special tokens is free." New rows in the embedding table are randomly initialized and unrelated to the pretrained representations. You need enough domain data to train them to usefulness, and they silently break the vocabulary divisibility math that made the original model's tensor-parallel layout work. Adding 3 special tokens can cost you a full 125-row padding slab in memory.

What's next

So vocabulary size is a three-way tradeoff between sequence compression, parameter cost, and training signal density, and the scaling law gives us a principled way to navigate it.

The next post covers The Embedding Table and Its Geometry, the matrix the vocabulary feeds into, the geometry it settles into after training, and the anisotropy problem that makes rare tokens' embeddings so unreliable.

Additional reading (and watching)

• Rust, P. et al. (2020). How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models. Showed that a dedicated monolingual tokenizer plays an equally important role as pretraining data size in downstream performance, studying nine typologically diverse languages.
• Petrov, A. et al. (2023). Language Model Tokenizers Introduce Unfairness Between Languages. NeurIPS 2023. Showed that tokenizer inequity directly translates to cost and capability gaps across languages.
• Tao, C. et al. (2024). Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies. NeurIPS 2024. Derived $V_{\text{opt}} \propto C^{0.36}$ and estimated Llama 2-70B's vocabulary was ~7× below optimal.
• Dubey, A. et al. (2024). The Llama 3 Herd of Models. Meta technical report. Documents the vocabulary expansion from 32k to 128,256 tokens, the rationale for the specific number (tensor-parallel divisibility, kernel alignment), and the resulting multilingual performance gains.
• Ali, M. et al. (2023). Tokenizer Choice For LLM Training: Negligible or Crucial?. NAACL 2024. Includes discussion of vocabulary sizing and its interaction with hardware constraints and multilingual performance.
• OpenAI (2024). GPT-4o System Card. Describes the o200k_base tokenizer and its improved efficiency for non-English languages compared to cl100k_base.