WordPiece, Unigram, and SentencePiece

Three algorithms, same sentence, different segmentations. BPE merges frequent pairs, WordPiece maximizes likelihood, Unigram prunes a start-big vocabulary.

In the last post we built BPE from scratch and watched a vocabulary grow out of raw characters through iterative pair merging. BPE is by far the most common tokenization algorithm in modern LLMs. GPT-2, GPT-4, Llama, Claude, Mistral all use some variant of byte-level BPE.

So why does anyone use anything else?

When you load a BERT checkpoint, you encounter tokens like ##ing and ##tion. That ## prefix is not BPE. It's WordPiece, a different algorithm with different merge criteria. When you load T5 or mBART, you see a vocabulary where every word-initial token starts with ▁ (a special Unicode character standing in for whitespace). That's Unigram, running inside a framework called SentencePiece. The three are not interchangeable, and they make genuinely different decisions about how to split text.

This post walks through all three algorithms, compares their outputs on the same text, and builds enough intuition to know which is which and why each one exists.

Prerequisites: comfort with the BPE algorithm from the previous post, and basic probability and logarithms. We'll also touch briefly on Viterbi decoding.

WordPiece: BPE's likelihood-maximizing cousin

WordPiece was introduced by Schuster and Nakajima in 2012 for Japanese and Korean voice search at Google. It looks a lot like BPE at first glance. Both are bottom-up algorithms: you start with a small vocabulary of characters and iteratively merge pairs into larger tokens. The difference is how you choose which pair to merge.

BPE counts raw frequency. At every step it merges the pair that appears the most times in the corpus. WordPiece merges the pair that maximizes the likelihood of the training data under a unigram language model. For a candidate merge of tokens $A$ and $B$ into $AB$, WordPiece computes:

That ratio is essentially a pointwise mutual information score. A pair gets merged not just because it appears often in absolute terms, but because it appears more often than you would expect given how common $A$ and $B$ are individually. The pair "t+h" might appear 5,000 times, but if "t" appears 50,000 times and "h" appears 40,000 times, the ratio is small. Meanwhile, a rarer pair like "q+u" can have a much higher ratio, because "q" almost never appears without "u."

So the two algorithms score the same candidate pairs and generally pick different winners. Toggle the criterion below to see it happen.

In practice, this means WordPiece tends to produce different merge orders than BPE, especially for common character pairs. But the big picture is similar: you end up with a subword vocabulary where common words become single tokens and rare words get split into pieces.

The ## convention

The second thing that makes WordPiece distinctive is its continuation marker. When BERT tokenizes the word "embeddings," it produces something like:

["em", "##bed", "##ding", "##s"]

The ## prefix means "this token is a continuation of the previous word, not a new word." The first token in every word has no prefix. This is the opposite convention from SentencePiece, which marks word beginnings instead of continuations. We'll get to that.

Step through the segmentation below to see exactly how this works. WordPiece tries the longest possible match starting from the left, shrinks until it finds a vocab hit, locks that token in, then moves on to the remainder with ## prefixed.

Tokenization at inference time

There's a subtle difference here that's easy to miss. During BPE inference (applying a trained vocabulary to new text), you replay the merge rules in order. WordPiece uses a greedy longest-match-first algorithm instead. For each word, you try to match the longest token in the vocabulary starting from the left. If embed is in the vocabulary, you take it. Then you try the longest match starting from the remainder, dings, and find ##ding. Then ##s.

This greedy approach is fast, but it isn't globally optimal. There might be a better segmentation you miss because you greedily consumed too many characters early. The Unigram algorithm addresses exactly this problem.

Why Google picked WordPiece for BERT

When Devlin et al. published BERT in 2018, they inherited WordPiece from Google's internal NLP toolkit. BERT's 30,522 WordPiece vocabulary was trained on English Wikipedia and BooksCorpus. The cased vs. uncased model distinction is partly a WordPiece decision: the uncased vocabulary lowercases everything before tokenization, which shrinks the required vocabulary size at the cost of losing case information. If you've ever wondered why bert-base-uncased turns "The" into ["the"] while bert-base-cased keeps it as ["The"], that's the pre-tokenization step doing its thing before WordPiece even runs.

Unigram: top-down and probabilistic

Unigram is a fundamentally different approach, introduced by Taku Kudo in 2018. Instead of building a vocabulary up from small pieces by merging (bottom-up), Unigram starts with a large candidate vocabulary and prunes it down (top-down).

Here's the algorithm:

1. Initialize with a huge candidate vocabulary. Typically you start with all substrings up to some maximum length that appear in the training corpus, plus all individual characters. This can be 100,000+ candidates.
2. Assign probabilities. Each token $x_i$ gets a unigram probability $p(x_i)$ estimated from the training data via EM (expectation-maximization). The probability of a particular segmentation of a sentence into tokens $x_1, x_2, \ldots, x_n$ is just the product of the individual token probabilities:

3. Evaluate the loss. For each token in the vocabulary, compute how much the overall corpus likelihood would decrease if you removed it. Tokens that barely affect the likelihood are candidates for pruning.
4. Prune. Remove a fraction (typically 10–20%) of the lowest-impact tokens. Never remove single characters, since they're your safety net for encoding any input.
5. Repeat steps 2–4 until you reach your target vocabulary size.

The key insight is that pruning lets the algorithm consider the global picture. A token only survives if it genuinely helps model the training corpus. BPE, by contrast, makes purely local decisions at each merge step and never revisits them.

Viterbi decoding: optimal segmentation

Once you have a trained Unigram vocabulary with token probabilities, how do you tokenize a new sentence? You need the segmentation that maximizes total log-probability. In log space the product turns into a sum:

This is a classic dynamic programming problem, and the Viterbi algorithm solves it in $O(nL)$ time, where $n$ is the length of the input string and $L$ is the maximum token length.

Let me work it on a small example. Suppose we want to tokenize "unigram" and our vocabulary has these tokens with log-probabilities (higher is better):

We build a table left-to-right. At each boundary $j$ in the string, we store the best total score achievable for the substring from 0 to $j$, and which token (arc) ended there. Each new boundary looks back at every arc that could end there and picks the one with the largest total score.

Step through the lattice below. Each arc is a candidate token. Blue arcs are the ones being considered at the current boundary, and the chosen winner at each step gets a solid stroke. At the end, the backtraced optimal path lights up green across the whole word.

So the common claim that "Unigram is slower because it's probabilistic" is wrong. Viterbi is $O(nL)$ and greedy longest-match is $O(nL)$. Same complexity class. The difference is that Viterbi considers every way to cut the word and picks the best total, while greedy longest-match commits early and can't back out.

Subword regularization

This is my favorite feature of Unigram. Because Unigram maintains a full probability distribution over all possible segmentations, you can sample different segmentations during training instead of always using the single best one.

For the input "unigram," the best segmentation might be ["uni", "gram"], but there are other valid segmentations like ["u", "n", "ig", "ra", "m"] or ["un", "ig", "ram"]. During training, you sample from these according to their probabilities. The model sees the same word broken up different ways across different epochs, and becomes more robust to rare or out-of-distribution tokenizations.

Kudo showed that subword regularization improved BLEU scores on machine translation benchmarks by 0.5–1.0 points, which is a meaningful margin in that world. It's essentially a free regularization technique that falls out of the probabilistic formulation.

SentencePiece: the framework, not the algorithm

SentencePiece is not a tokenization algorithm. It's a library and framework that implements both BPE and Unigram.

What makes SentencePiece distinctive is not which algorithm it uses, but how it treats the input text. Standard tokenizers (including the original BPE and WordPiece implementations) assume the input has already been split into words by whitespace. They operate on pre-tokenized text. SentencePiece operates on raw, unsegmented text. It treats the entire input as a flat sequence of characters, and whitespace is just another character.

To make this work, SentencePiece replaces space characters with ▁ (Unicode U+2581, LOWER ONE EIGHTH BLOCK) at word boundaries. So the sentence "I like cats" becomes the character sequence ▁I▁like▁cats internally. The ▁ is a first-class character that can participate in merges and become part of tokens. This is why SentencePiece-trained vocabularies have tokens like ▁The, ▁is, ▁un. The ▁ prefix tells you "a space came before this token in the original text."

The picture that finally made this click for me is the side-by-side below. Watch the two rails on the same input. The classical flow splits on whitespace first and then runs the subword model word-by-word. SentencePiece promotes the space to a symbol, hands the whole string to the subword model in one pass, and lets the algorithm discover where the word boundaries actually are.

Why does this matter?

For English, the difference is mostly cosmetic. For languages like Chinese, Japanese, Korean, and Thai, where words aren't separated by spaces, a whitespace-based pre-tokenizer has nothing to grab onto. It either fails to split anything (treating an entire sentence as one "word") or relies on a language-specific segmentation tool, which introduces a dependency and a possible error source.

SentencePiece sidesteps the entire problem. Because it treats the input as a flat character sequence, it works identically on English, Japanese, Arabic, and code. The algorithm discovers the right subword boundaries from the training data regardless of whether the language uses spaces.

This is why T5, mBART, XLM-RoBERTa, and most multilingual models use SentencePiece. When people say "T5 uses SentencePiece Unigram," they mean: T5 uses the Unigram algorithm as implemented inside the SentencePiece framework, trained on raw unsegmented text with the ▁ convention.

Using SentencePiece in practice

If you want to train your own SentencePiece model or inspect an existing one, the Python API is straightforward:

import sentencepiece as spm
 
# Train a new model on raw text
spm.SentencePieceTrainer.train(
    input="training_data.txt",
    model_prefix="my_tokenizer",
    vocab_size=32000,
    model_type="unigram",  # or "bpe"
    character_coverage=0.9995,  # important for CJK
)
 
# Load and use the trained model
sp = spm.SentencePieceProcessor()
sp.load("my_tokenizer.model")
 
# Encode text to token strings or IDs
tokens = sp.encode("The tokenizer works.", out_type=str)
print(tokens)
# ['▁The', '▁token', 'izer', '▁works', '.']
 
ids = sp.encode("The tokenizer works.", out_type=int)
print(ids)
# [494, 14846, 7629, 1330, 5]
 
# Decode back to text
text = sp.decode(ids)
print(text)
# 'The tokenizer works.'
 
# Subword regularization: sample different segmentations
for _ in range(3):
    sampled = sp.encode("unigram", out_type=str,
                         enable_sampling=True,
                         alpha=0.1, nbest_size=-1)
    print(sampled)
# ['▁uni', 'gram']
# ['▁un', 'ig', 'ram']
# ['▁uni', 'g', 'ram']

One parameter to flag: character_coverage. It controls what fraction of characters in the training data must be representable by the vocabulary. For English, 1.0 is fine. For CJK-heavy corpora, you might set it to 0.9995 to avoid exploding the vocabulary size with every rare kanji. Characters below the coverage threshold get mapped to a special unknown token or decomposed to bytes.

Seeing the differences

Theory is useful, but the differences really click when you see the same text processed by all three algorithms. Click through the examples below. Pay attention to where the token boundaries land and how each algorithm handles the ## and ▁ conventions.

A few things to notice.

Token counts vary. Unigram often produces fewer tokens for the same input because its Viterbi decoding finds globally optimal splits. WordPiece tends to produce more tokens because its greedy left-to-right matching can miss opportunities to combine.

The multilingual example is revealing. Look at "Mixed languages." SentencePiece Unigram merges the CJK characters into bigram tokens like ▁機械 because it learned from raw text that these characters co-occur. WordPiece and BPE, which pre-tokenize on whitespace, treat the CJK block as individual characters more often.

Prefix conventions are opposite. WordPiece marks continuations with ##. SentencePiece marks word starts with ▁. "Is this the beginning of a word?" is answered by looking for the absence of ## in WordPiece, but the presence of ▁ in SentencePiece. The information is equivalent, just encoded differently.

When each algorithm wins

So which should you use? The algorithm usually matters less than the vocabulary size and the training data. But here are the real patterns as of 2026.

BPE (specifically byte-level BPE) dominates decoder-only LLMs: GPT-2 through GPT-4, Claude, Llama, Mistral, Gemma. It's simple, fast, deterministic, and well-understood. The byte-level variant means no unknown tokens ever, which is important for models that need to handle arbitrary input including code, URLs, and binary-looking strings.

WordPiece is mainly found in the BERT ecosystem: BERT, DistilBERT, ELECTRA, and their descendants. It works well, but has largely been superseded by BPE and Unigram in newer architectures. If you're training a new model from scratch, there's little reason to choose WordPiece over the alternatives. Google themselves moved to SentencePiece for T5 and subsequent models.

Unigram via SentencePiece wins for multilingual and encoder-decoder models: T5, mT5, mBART, XLM-RoBERTa, NLLB. The combination of no pre-tokenization (language-agnostic) and subword regularization (training robustness) makes it the natural choice when your model needs to handle 100+ languages. The Viterbi optimal segmentation is a nice theoretical property, though in practice the quality difference versus BPE is small for any single language.

Misconceptions

"SentencePiece is a tokenization algorithm." It isn't. SentencePiece is a framework that implements both BPE and Unigram. When someone says "SentencePiece tokenization," they usually mean Unigram-inside-SentencePiece, but you should check the model config to know which algorithm is actually running.

"WordPiece and BPE are the same thing." They're similar in structure (both are bottom-up merge algorithms) but they use different merge criteria. BPE merges the most frequent pair. WordPiece merges the pair that maximizes corpus likelihood, the mutual-information-style score we walked through above. The resulting vocabularies overlap heavily but aren't identical.

"The ▁ and ## markers are just cosmetic." They're actual characters in the token string and they change what the embedding table sees. The token ▁The and the token The are two different rows in the embedding matrix. A model fine-tuned on one convention and then served with the other will silently degrade.

What's next

Now you know the three algorithms and when each one shows up. The frontier has mostly converged on byte-level BPE, with Unigram holding ground in multilingual settings.

In the next post we'll dig into Vocabulary Size, Merge Order, and Fertility: how big the vocabulary should be, why merge order matters, and how to measure the "multilingual tax" a tokenizer imposes.

Additional reading (and watching)

• Schuster, M. & Nakajima, K. (2012). Japanese and Korean Voice Search. ICASSP 2012. Introduced the algorithm that later became known as WordPiece.
• Kudo, T. (2018). Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates. ACL 2018. The Unigram language-model tokenizer and the subword-sampling training trick.
• Kudo, T. & Richardson, J. (2018). SentencePiece: A Simple and Language-Independent Subword Tokenizer. EMNLP 2018. The library paper; introduces the ▁ whitespace-as-character convention.
• Devlin, J. et al. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL 2019. The reason most people have ever heard of WordPiece.
• Raffel, C. et al. (2020). Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. JMLR 2020. T5 popularized SentencePiece Unigram for encoder-decoder models.
• Xue, L. et al. (2022). ByT5: Towards a Token-Free Future with Pre-training for Byte-Level Models. The reference for tokenizer-free byte-level modeling.
• Yu, L. et al. (2023). MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers. Scaling byte-level models with a hierarchical architecture.
• Hugging Face. Summary of the Tokenizers. The canonical reference for how BPE, WordPiece, and Unigram actually get plugged together in production pipelines.