Vocabulary Size, Merge Order, and Fertility

The algorithms in the previous post construct and apply a subword vocabulary. How large should that vocabulary be?

Vocabulary size affects sequence length, embedding table size, how well your model handles languages other than English, even how tensors get laid out across GPUs.

Fertility measures token use for a chosen unit of text. Combined with parameter counts and training experiments, it helps assess whether a larger vocabulary is worth its cost.

Prerequisites: BPE from the BPE post and a sense of the WordPiece/Unigram/SentencePiece algorithms from the previous post. A rough feel for the embedding table helps but we preview the calculation here.

The three-way tradeoff

Sequences get shorter. If your tokenizer knows the word "unfortunately" as a single token, that's one position in the sequence. A smaller vocabulary might split it into "un", "for", "tun", "ately" and now you've consumed four positions to say the same thing. Shorter sequences mean less work for self-attention, which scales super-linearly with sequence length.

The embedding table grows too. It has one row per token in the vocabulary, each a dense vector of dimension $d_{\text{model}}$, so the parameter count for the embedding layer is:

For a model with $d_{\text{model}} = 4096$, a 32k vocabulary means roughly 131 million embedding parameters. A 200k vocabulary means about 819 million, a 6× increase for a single lookup table.

A larger vocabulary can also introduce rare entries. An input embedding row gets a direct lookup gradient only when that token occurs; sharing the matrix with the output layer changes that picture.

Merge order: what gets learned first

The BPE post covered the ranked merge list. Within one training run, extending the list adds the pairs selected in later iterations. Vocabulary size includes the base alphabet, retained merges, and special tokens. Two independently trained tokenizers of the same size need not share any merge ordering.

A corpus dominated by English can spend much of its vocabulary on English subwords. Adding slots may help other scripts, but there is no stage at which English is exhausted and other languages begin. Corpus composition and pre-tokenization affect the entire merge sequence.

Fertility: the diagnostic

Token fertility measures how many tokens a tokenizer uses for a chosen unit of text:

A fertility of 1.0 means one token per word on average, including whatever punctuation your counting procedure includes. It does not mean every word is one token. The denominator matters: whitespace splitting is convenient for this example but is not a comparable linguistic definition of a word across languages.

Measure fertility on representative text from your intended use. A lower value can fit more of that text into a token-limited context, but it does not by itself measure model quality, latency, or the cost of expressing the same meaning in another language.

A worked example

The following sentences let us reproduce the chart. English produces 14 tokens in both encodings; Arabic produces 44 and 22 respectively. This demonstrates a difference between these two vocabularies on these strings, not a controlled experiment changing vocabulary size alone.

import tiktoken
 
# GPT-4's tokenizer (100k vocab)
enc_100k = tiktoken.get_encoding("cl100k_base")
# GPT-4o's tokenizer (200k vocab)
enc_200k = tiktoken.get_encoding("o200k_base")
 
def measure_fertility(text: str, encoding) -> dict:
    """Compute token fertility for a piece of text."""
    tokens = encoding.encode(text)
    words = text.split()
    word_count = max(len(words), 1)
    return {
        "tokens": len(tokens),
        "words": word_count,
        "fertility": len(tokens) / word_count,
    }
 
english = "The quick brown fox jumps over the lazy dog near the riverbank."
spanish = "El rápido zorro marrón salta sobre el perro perezoso cerca del río."
arabic  = "الثعلب البني السريع يقفز فوق الكلب الكسول بالقرب من ضفة النهر."
 
for label, text in [("English", english), ("Spanish", spanish), ("Arabic", arabic)]:
    r100 = measure_fertility(text, enc_100k)
    r200 = measure_fertility(text, enc_200k)
    print(f"\n{label}: {r100['words']} words")
    print(f"  cl100k: {r100['tokens']} tokens, fertility = {r100['fertility']:.2f}")
    print(f"  o200k : {r200['tokens']} tokens, fertility = {r200['fertility']:.2f}")
 
# Output:
# English: 12 words
#   cl100k: 14 tokens, fertility = 1.17
#   o200k : 14 tokens, fertility = 1.17
# Spanish: 12 words
#   cl100k: 21 tokens, fertility = 1.75
#   o200k : 18 tokens, fertility = 1.50
# Arabic: 11 words
#   cl100k: 44 tokens, fertility = 4.00
#   o200k : 22 tokens, fertility = 2.00

The Arabic sample uses fewer tokens under o200k_base. To assess your own workload, repeat the measurement on a corpus and report the distribution, not just a single sentence. Keep the tokenizer artifact and text normalization fixed so the comparison is reproducible.

The scaling law for vocabulary

Tao et al. fit vocabulary scaling relationships in controlled training experiments. One result relates the optimal vocabulary parameter count $N_v$ to the non-vocabulary parameter count $N_{nv}$:

Vocabulary parameters depend on model width as well as the number of entries. If $N_v = Vd$, converting the fit to a vocabulary size requires $d$; untied output weights add another copy.

The paper also studies data and compute constraints. Its fitted coefficients and optimum depend on the experimental setup, so applying them to another corpus or architecture calls for validation.

Practical engineering considerations

Divisibility and padding. Vocabulary-parallel implementations split the vocabulary rows across devices. Depending on the framework and kernels, the allocated table may need more rows than the tokenizer has entries. Check both the tokenizer vocabulary and the model configuration: padding is an implementation cost, not evidence that every allocated row represents a text token.

The cost of adding tokens. Extending a pretrained vocabulary adds embedding rows and may add output rows. Their initialization depends on the library and configuration. Those rows need training before you can assume that the model uses the new tokens reliably.

The output layer mirrors the input. The output projection produces one score per vocabulary entry. With separate input and output weights, its parameters double the vocabulary cost:

With tied weights the two operations share one matrix, so the stored parameter count is $Vd$ instead. Computing output scores still has a vocabulary-dependent cost.

For an untied 200k-vocabulary model at $d_{\text{model}} = 4096$, that's about 1.6 billion parameters tied up just in the vocabulary layers. For a 7B model, that's over 20% of total parameters. For a 70B model, it's about 2%. So larger models can more easily absorb the cost of larger vocabularies.

The long tail

The frequency of a token affects the evidence available to train its embedding.

A merge’s rank is not its frequency rank in the final tokenized corpus. Later merges can absorb earlier pieces; tokenize the actual corpus to count the remaining occurrences.

A useful diagnostic is to tokenize the actual training corpus, inspect the low-frequency tail, and evaluate held-out loss and task performance. That gives evidence for whether the shorter sequences justify the extra rows.

Misconceptions

"Bigger vocab is always better." Extra entries can shorten sequences but add parameters and may receive little direct evidence. Evaluate the tradeoff under a fixed training budget.

"A 32k vocabulary is 32k words." It contains base units, subword pieces, and special tokens. Some words need several entries; some entries include punctuation or whitespace.

"Fertility is a model-quality score." It measures segmentation under a chosen denominator. A tokenizer that compresses your corpus well still needs a model trained to use it.

"Adding special tokens is free." Extra rows need initialization and training, and model allocation may round their count up for sharding or kernels.

What's next

The next post covers The Embedding Table and Its Geometry, the matrix the vocabulary feeds into, and how to inspect the geometry of its learned vectors.

Additional reading (and watching)

• Rust, P. et al. (2020). How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models. Showed that a dedicated monolingual tokenizer plays an equally important role as pretraining data size in downstream performance, studying nine typologically diverse languages.
• Petrov, A. et al. (2023). Language Model Tokenizers Introduce Unfairness Between Languages. NeurIPS 2023. Showed that tokenizer inequity directly translates to cost and capability gaps across languages.
• Tao, C. et al. (2024). Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies. NeurIPS 2024. Studies vocabulary parameters jointly with model size, data, and compute.
• Dubey, A. et al. (2024). The Llama 3 Herd of Models. Meta technical report. Documents the Llama 3 tokenizer and training setup.
• Ali, M. et al. (2023). Tokenizer Choice For LLM Training: Negligible or Crucial?. NAACL 2024. Includes discussion of vocabulary sizing and its interaction with hardware constraints and multilingual performance.