WordPiece, Unigram, and SentencePiece

Toy segmentations illustrate three conventions. BPE merges frequent pairs, WordPiece encodes by longest match, Unigram prunes a start-big vocabulary.

In the last post we built BPE from scratch. BPE appears in many language models, but it is one way to build a subword vocabulary. WordPiece and Unigram make different choices about vocabulary construction and how to segment new text.

BERT uses WordPiece tokens such as ##ing, where the prefix marks a continuation. T5 uses SentencePiece Unigram and tokens containing ▁, its whitespace marker. The marker alone does not identify the algorithm: SentencePiece also implements BPE.

Prerequisites: comfort with the BPE algorithm from the previous post, and basic probability and logarithms. We'll also touch briefly on Viterbi decoding.

WordPiece: BPE's likelihood-maximizing cousin

WordPiece was introduced by Schuster and Nakajima in 2012 for Japanese and Korean voice search at Google. It looks a lot like BPE at first glance. Both are bottom-up algorithms: you start with a small vocabulary of characters and iteratively merge pairs into larger tokens. The difference is how you choose which pair to merge.

BPE chooses the most frequent pair. WordPiece training is motivated by improving corpus likelihood, but Google did not release the original BERT vocabulary trainer. A common educational reconstruction scores candidate pairs using:

This is proportional to the exponentiated pointwise mutual information when counts share the same normalization. A frequent pair can score lower than a rarer, tightly associated pair.

So the two algorithms score the same candidate pairs and generally pick different winners. Toggle the criterion below to see it happen.

The ## convention

The second thing that makes WordPiece distinctive is its continuation marker. For a toy WordPiece vocabulary, "embeddings" might be segmented as:

["em", "##bed", "##ding", "##s"]

The ## prefix means "this token is a continuation of the previous word, not a new word." The first token in every word has no prefix. SentencePiece instead marks whitespace boundaries with ▁.

Tokenization at inference time

During BPE inference (applying a trained vocabulary to new text), you replay the merge rules in order. WordPiece uses a greedy longest-match-first algorithm instead. For each word, you try to match the longest token in the vocabulary starting from the left. If embed is in the vocabulary, you take it. Then you try the longest match starting from the remainder, dings, and find ##ding. Then ##s.

This greedy approach is fast, but it isn't globally optimal. There might be a better segmentation you miss because you greedily consumed too many characters early. Unigram instead scores complete segmentations under a probabilistic model.

Why Google picked WordPiece for BERT

When Devlin et al. published BERT in 2018, they inherited WordPiece from Google's internal NLP toolkit. The bert-base-uncased checkpoint has 30,522 vocabulary entries. Its basic tokenizer lowercases text and strips accents before WordPiece runs. These are preprocessing choices, not requirements of WordPiece. bert-base-uncased turns "The" into ["the"] while bert-base-cased keeps it as ["The"] because their preprocessing differs.

Unigram: top-down and probabilistic

Unigram is a fundamentally different approach, introduced by Taku Kudo in 2018. Instead of building a vocabulary up from small pieces by merging (bottom-up), Unigram starts with a large candidate vocabulary and prunes it down (top-down).

Here's the algorithm:

1. Initialize with a huge candidate vocabulary. Candidate generation can collect frequent substrings up to a maximum length, together with the selected character alphabet. This can be 100,000+ candidates.
2. Assign probabilities. Each token $x_i$ gets a unigram probability $p(x_i)$ estimated from the training data via EM (expectation-maximization). A segmentation into tokens $x_1, x_2, \ldots, x_n$ receives the product of their probabilities as its score:

For a fixed input string, normalize these scores across its valid segmentations to obtain a conditional distribution. The most likely path is unchanged by that shared denominator.

3. Evaluate the loss. For each token in the vocabulary, compute how much the overall corpus likelihood would decrease if you removed it. Tokens that barely affect the likelihood are candidates for pruning.
4. Prune. Remove a fraction (typically 10–20%) of the lowest-impact tokens. Keep the selected training alphabet so pruning does not destroy coverage of those characters. Unseen characters still need an unknown token or byte fallback.
5. Repeat steps 2–4 until you reach your target vocabulary size.

BPE, by contrast, makes purely local decisions at each merge step and never revisits them.

Viterbi decoding: optimal segmentation

Once you have a trained Unigram vocabulary with token probabilities, how do you tokenize a new sentence? You need the segmentation that maximizes total log-probability. In log space the product turns into a sum:

This is a classic dynamic programming problem, and the Viterbi algorithm solves it in $O(nL)$ time, where $n$ is the length of the input string and $L$ is the maximum token length.

For "unigram", use the illustrative scores below to trace the maximum-sum recurrence. Higher scores are better; these chosen values are not a normalized log-probability table:

We build a table left-to-right. At each boundary $j$ in the string, we store the best total score achievable for the substring from 0 to $j$, and which token (arc) ended there. Each new boundary looks back at every arc that could end there and picks the one with the largest total score.

Step through the lattice below. Each arc is a candidate token. Blue arcs are the ones being considered at the current boundary, and the chosen winner at each step gets a solid stroke. At the end, the backtraced optimal path lights up green across the whole word.

The dynamic program considers all paths without enumerating each one. The $O(nL)$ bound assumes efficient candidate lookup; vocabulary storage and implementation also affect runtime.

Subword regularization

Because Unigram maintains a full probability distribution over all possible segmentations, you can sample different segmentations during training instead of always using the single best one.

For the input "unigram," the best segmentation might be ["uni", "gram"], but there are other valid segmentations like ["u", "n", "ig", "ra", "m"] or ["un", "ig", "ra", "m"]. During training, you sample from these according to their probabilities. The model sees the same word broken up different ways across different epochs, which can encourage robustness to alternative tokenizations.

Kudo reported improved machine-translation results with subword sampling.

SentencePiece: the framework, not the algorithm

SentencePiece is not a tokenization algorithm. It's a library and framework that implements both BPE and Unigram.

SentencePiece can train directly from raw sentences, normalizing the text and representing whitespace with a visible symbol. It does not require an external word segmenter. Configurable piece boundaries still apply: by default, split_by_whitespace=True prevents a piece from spanning whitespace-delimited words.

To make this work, SentencePiece replaces space characters with ▁ (Unicode U+2581, LOWER ONE EIGHTH BLOCK) at word boundaries. So the sentence "I like cats" becomes the character sequence ▁I▁like▁cats internally. The ▁ is a first-class character that can participate in merges and become part of tokens. This is why SentencePiece-trained vocabularies have tokens like ▁The, ▁is, ▁un. By default SentencePiece also adds a dummy prefix, so an initial ▁ need not correspond to a literal leading space in the source.

SentencePiece without word boundaries

Chinese, Japanese, and Thai do not consistently separate words with spaces. SentencePiece can train on those strings without first supplying word boundaries. Korean does use spaces, though those units need not correspond to the subwords useful to a model.

T5 uses SentencePiece Unigram. To inspect another checkpoint, check its tokenizer model and normalization rules rather than inferring the algorithm from the library name.

Using SentencePiece in practice

The Python API can train a SentencePiece model and inspect its segmentation:

import sentencepiece as spm
 
# Train a new model on raw text
spm.SentencePieceTrainer.train(
    input="training_data.txt",
    model_prefix="my_tokenizer",
    vocab_size=32000,
    model_type="unigram",  # or "bpe"
    character_coverage=0.9995,  # important for CJK
)
 
# Load and use the trained model
sp = spm.SentencePieceProcessor()
sp.load("my_tokenizer.model")
 
# Encode text to token strings or IDs
tokens = sp.encode("The tokenizer works.", out_type=str)
print(tokens)
# Token strings depend on training_data.txt.
 
ids = sp.encode("The tokenizer works.", out_type=int)
print(ids)
# IDs are specific to the trained vocabulary.
 
# Decode back to text
text = sp.decode(ids)
print(text)
# Returns normalized text when the input is fully covered; unknown pieces
# cannot reconstruct characters discarded by the tokenizer.
 
# Subword regularization: sample different segmentations
for _ in range(3):
    sampled = sp.encode("unigram", out_type=str,
                         enable_sampling=True,
                         alpha=0.1, nbest_size=-1)
    print(sampled)

character_coverage controls what fraction of character occurrences in the training data must be covered, selecting characters by frequency. For English, 1.0 is fine. For CJK-heavy corpora, you might set it to 0.9995 to avoid exploding the vocabulary size with every rare kanji. Characters below the coverage threshold get mapped to a special unknown token or decomposed to bytes if byte_fallback=True was enabled during training.

Seeing the differences

The examples below use illustrative segmentations to compare token boundaries and the ## and ▁ conventions. They are not outputs from three matched tokenizers.

Unigram maximizes a segmentation score; it does not minimize token count. WordPiece can produce fewer tokens than Unigram on some inputs. Multilingual coverage depends on the training data, retained alphabet, normalization, and vocabulary allocation as well as the algorithm.

Choosing an algorithm

For an existing checkpoint, use its tokenizer. For a new model, compare candidate tokenizers on your corpus: inspect unknowns, token counts, round-trip behavior, training cost, and downstream model quality. Byte-level BPE offers byte coverage; WordPiece provides a greedy encoding rule; Unigram provides scored paths and sampling.

Misconceptions

"SentencePiece is a tokenization algorithm." It isn't. SentencePiece is a framework that implements both BPE and Unigram. When someone says "SentencePiece tokenization," they usually mean Unigram-inside-SentencePiece, but you should check the model config to know which algorithm is actually running.

"WordPiece and BPE are the same thing." They use different encoding rules: ranked pair merges versus greedy longest match. Their training procedures also differ, though the exact original BERT vocabulary trainer was not released.

"The ▁ and ## markers are just cosmetic." They describe tokenization conventions. If a vocabulary contains both ▁The and The with different IDs, those IDs select different embedding rows. The model sees IDs, not the printed marker itself; changing segmentation or ID assignments can break the learned mapping.

What's next

Next, Vocabulary Size, Merge Order, and Fertility examines vocabulary cost and how to measure token counts across languages.

Additional reading (and watching)

• Schuster, M. & Nakajima, K. (2012). Japanese and Korean Voice Search. ICASSP 2012. Introduced the algorithm that later became known as WordPiece.

• Kudo, T. (2018). Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates. ACL 2018. The Unigram language-model tokenizer and the subword-sampling training trick.

• Kudo, T. & Richardson, J. (2018). SentencePiece: A Simple and Language-Independent Subword Tokenizer. EMNLP 2018. The library paper; introduces the ▁ whitespace-as-character convention.

• Devlin, J. et al. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL 2019. The reason most people have ever heard of WordPiece.

• Raffel, C. et al. (2020). Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. JMLR 2020. T5 popularized SentencePiece Unigram for encoder-decoder models.

• Xue, L. et al. (2022). ByT5: Towards a Token-Free Future with Pre-training for Byte-Level Models. The reference for tokenizer-free byte-level modeling.

• Yu, L. et al. (2023). MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers. Scaling byte-level models with a hierarchical architecture.

• Hugging Face. Summary of the Tokenizers. The canonical reference for how BPE, WordPiece, and Unigram actually get plugged together in production pipelines.

• Hugging Face. WordPiece tokenization. Explicitly distinguishes the reconstructed training procedure from the unavailable original implementation.

• Google. SentencePiece documentation. Normalization, dummy-prefix, character-coverage, and byte-fallback options.