GloVe, FastText, and the Embedding Zoo

Why are there so many word embedding methods?

I remember the first time I tried to actually use pretrained word vectors for a project. I went to download them and immediately hit a fork in the road: GloVe? word2vec? FastText? They all produce the same kind of artifact (a big table mapping words to dense vectors), and they all seem to capture similar semantic relationships. So what's the difference? And does it matter?

I think the cleanest way to understand this is to look at where these methods came from intellectually. Before word2vec showed up in 2013, NLP researchers had been building word representations from co-occurrence statistics for decades. Latent Semantic Analysis, pointwise mutual information matrices, all that stuff. These were "count-based" methods. Then Mikolov's word2vec came along and said: forget counting, just train a neural network to predict context words. That's a "prediction-based" method.

For a while, the field treated these as fundamentally different philosophies. Count-based versus prediction-based, old school versus new school. They turn out to be closer than anyone expected.

GloVe: Making Co-occurrence Counts Learnable

GloVe (Global Vectors for Word Representation) came out of Stanford in 2014. Pennington, Socher, and Manning noticed that raw co-occurrence counts are noisy and hard to use directly. But ratios of co-occurrence probabilities are stable and meaningful in a way raw counts aren't.

Here's the classic example from the paper. Let $P(k \mid w)$ be the probability that word $k$ appears in the context of word $w$, estimated from a big corpus. Consider the words "ice" and "steam," and look at how various probe words $k$ relate to them:

"Solid" co-occurs much more with "ice" than "steam," so the ratio is large. "Gas" goes the other way. "Water" and "fashion" are either related to both or neither, so their ratios sit near 1. The ratio $P(k \mid \text{ice}) / P(k \mid \text{steam})$ cleanly separates words that discriminate between the two concepts from words that don't.

GloVe is motivated by these ratios, but the implemented objective is something simpler: a weighted least-squares regression that fits dot products (plus bias terms) to log co-occurrence counts. The actual training equation is:

where $X_{ij}$ is the raw co-occurrence count, $w_i$ and $\tilde{w}_j$ are the center and context vectors, and $b_i$, $\tilde{b}_j$ are scalar biases per word. The biases aren't cosmetic. They absorb word- and context-frequency effects, which lets the dot product focus on relational structure rather than just raw popularity. A weighting function on the loss caps the influence of extremely frequent co-occurrences (so "the" doesn't dominate). The model learns two vector tables ($w$ and $\tilde{w}$), and the final embedding for a word is typically their sum.

The connection back to ratios is what makes the simplification work: $\log(P(k|i) / P(k|j)) = \log P(k|i) - \log P(k|j)$, so a model that fits log-counts well also encodes the discriminative structure the ratio table exposes, as differences between learned vectors.

So GloVe is not a context-prediction model in the word2vec sense. There is no softmax or negative-sampling classifier. It first constructs sparse global co-occurrence statistics — only the non-zero $X_{ij}$ entries, which is far smaller than the full $V \times V$ matrix — then trains a shallow log-bilinear model on those entries with a weighted regression objective. The training loop is just gradient descent.

The Unification: Closer Than the Framing Suggested

In 2014, Levy and Goldberg published a paper proving that word2vec's skip-gram model with negative sampling (SGNS) is implicitly factorizing a matrix. When SGNS converges, the dot product of the word and context vectors equals the pointwise mutual information (PMI) of the word pair, shifted by a constant:

where $k$ is the number of negative samples. PMI is a classical co-occurrence statistic that NLP researchers had been computing from count matrices for years. So skip-gram, the supposedly "prediction-based" method, is implicitly factorizing a shifted PMI matrix. GloVe explicitly fits a weighted log-count matrix with bias terms. Those are different objectives over closely related co-occurrence statistics, not the same matrix.

The methods aren't as philosophically different as the original count-vs-predict framing suggested. They're both recovering low-rank structure from word-context co-occurrence signal. They get there by different objectives, with different weighting functions, different treatment of frequent words, and different optimization paths — and those differences matter for practical trade-offs (which we'll get to). Baroni et al. found that prediction-based methods tended to win benchmarks; Levy et al. (2015) later showed that hyperparameter choices and design details explain much of that gap, not any deep architectural advantage of one family over the other.

I find this result satisfying in a way that's hard to articulate. Two communities thought they were doing different things. Turns out they were learning low-rank structure from closely related co-occurrence matrices, using different objectives and optimization paths. Goldberg's textbook on neural methods for NLP walks through the equivalence more carefully if the algebra is clicking.

FastText: Words Aren't Atomic

Word2vec and GloVe both treat words as atomic units. The word "unforgettable" gets one vector, learned from its contexts. If "unforgettable" doesn't appear in your training corpus, you get nothing. It's out-of-vocabulary (OOV), and your only option is a generic unknown token.

Bojanowski et al. at Facebook addressed this in 2017 with FastText. The core idea is simple: represent each word as a bag of character n-grams, and learn vectors for the n-grams instead of (or in addition to) whole words.

Take the word "unforgetting." FastText would decompose it into character n-grams (using $n$ from 3 to 6 by default), plus special boundary markers:

The vector for "unforgetting" is the sum of vectors for all these n-gram fragments. And here's the payoff: even if "unforgetting" never appeared in training, many of its n-grams (like "for", "get", "ing") appeared in thousands of other words. So FastText can construct a reasonable vector for it by composing the pieces.

This matters a lot for morphologically rich languages. In Turkish or Finnish, a single word can carry what English spreads across an entire phrase. GloVe and word2vec need to see every inflected form independently. FastText shares information across forms that share substrings, which is a much better inductive bias for these languages.

The skip-gram training procedure is basically the same as word2vec. The only difference is that instead of looking up a single vector per word, you sum up the n-gram vectors. Gradients flow back to the n-grams, and the model learns that "un-" tends to negate, "-ing" tends to mark progressive aspect, and so on.

The Embedding Zoo: A Comparison

So which one should you actually use? Here's where they differ in practice:

In terms of raw quality on English benchmarks, the three methods are close enough that hyperparameter tuning and corpus choice matter more than the algorithm. The real differentiator is the OOV story. If you're working with social media text, medical terminology, or agglutinative languages, FastText's subword approach is a clear win.

Zoom in on the geometry for a second. One of the things that made word embeddings so compelling in 2013 was the analogy trick: take the vector for "king," subtract "man," add "woman," and the nearest neighbor is "queen." What's interesting is that this works across all three methods, even though they're fitting different objectives. The relational structure (gender axis, royalty axis) shows up in every recovered space.

Worked Example: Loading GloVe and Computing Similarities

Let's make this concrete. GloVe vectors are distributed as plain text files (one word per line, followed by its vector components). Loading them is almost embarrassingly simple.

import numpy as np
 
def load_glove(path):
    """Load GloVe vectors from a text file."""
    embeddings = {}
    with open(path, "r", encoding="utf-8") as f:
        for line in f:
            parts = line.strip().split()
            word = parts[0]
            vec = np.array(parts[1:], dtype=np.float32)
            embeddings[word] = vec
    return embeddings
 
def cosine_sim(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))
 
# Load pretrained GloVe (glove.6B.100d.txt)
glove = load_glove("glove.6B.100d.txt")
 
# Classic analogy: king - man + woman ≈ queen
# Exclude the input words from the search — otherwise one of them
# usually comes back as the nearest neighbor of the query vector.
query = glove["king"] - glove["man"] + glove["woman"]
exclude = {"king", "man", "woman"}
sims = {
    w: cosine_sim(query, v)
    for w, v in glove.items()
    if w not in exclude
}
top5 = sorted(sims, key=sims.get, reverse=True)[:5]
print("king - man + woman =", top5)
# ['queen', 'monarch', 'princess', 'throne', 'prince']
 
# Semantic similarity
pairs = [("cat", "dog"), ("cat", "car"), ("good", "great")]
for w1, w2 in pairs:
    sim = cosine_sim(glove[w1], glove[w2])
    print(f"  sim({w1}, {w2}) = {sim:.3f}")
# sim(cat, dog)   = 0.922
# sim(cat, car)   = 0.342
# sim(good, great) = 0.816

And for FastText's subword decomposition, here's what happens under the hood when it encounters a word:

The key thing to notice: GloVe gives you nothing for a word that's not in its vocabulary. FastText can usually construct a vector from subword parts, which is often much better than a generic unknown-token fallback — but the vector is only as good as the substring evidence learned during training. That difference matters a lot once you're processing real-world text full of misspellings, slang, and domain jargon.

Misconceptions

"GloVe is count-based, word2vec is neural, so they're fundamentally different." They're not. The Levy and Goldberg result is the whole point of the middle section above. Skip-gram with negative sampling is implicitly factoring a shifted PMI matrix. GloVe explicitly factors a weighted log-count matrix. Different objectives, very similar low-rank factorizations of a similar underlying signal. The "neural vs count" framing was a useful narrative for a few years, then the math caught up with it.

"FastText is always better because it handles OOV." The n-gram bag gives FastText its robustness, but it also means a FastText vector mixes in pieces that might have nothing to do with the word's actual meaning. For morphologically simple English, FastText's subword trick sometimes hurts a little on semantic similarity benchmarks compared to word2vec trained on the same corpus. So if your pipeline is mostly clean English, the OOV advantage stops mattering and you might actually prefer word2vec. On messy text, the OOV story dominates and FastText wins comfortably.

"These methods capture meaning." They capture distributional statistics. "Meaning," in the GloVe/word2vec/FastText sense, is the pattern of words a word tends to appear near. This is enough to reproduce striking semantic relationships, and it's also why these models cheerfully encode every stereotype in the training corpus. Bolukbasi et al. showed back in 2016 that "man : programmer :: woman : homemaker" falls out cleanly from pretrained word vectors because that's the distributional pattern in the text. Static embeddings reflect the data, not some platonic semantics.

What's next

Static vectors give every word a single fixed point in space, no matter how it's being used. The next step is sequences: models that read one word at a time and update a running state as they go. That's Recurrent Neural Networks and Sequence Modeling, where the hidden state becomes the thing doing the work.

Additional reading (and watching)

• Pennington, J., Socher, R., & Manning, C.D. (2014). GloVe: Global Vectors for Word Representation. EMNLP 2014.
• Levy, O. & Goldberg, Y. (2014). Neural Word Embedding as Implicit Matrix Factorization. NeurIPS 2014. Proved that skip-gram with negative sampling is implicitly factorizing a shifted PMI matrix, unifying "prediction-based" and "count-based" embedding methods.
• Bojanowski, P., Grave, E., Joulin, A., & Mikolov, T. (2017). Enriching Word Vectors with Subword Information. TACL, 5, 135-146.
• Baroni, M., Dinu, G., & Kruszewski, G. (2014). Don't Count, Predict! A Systematic Comparison of Context-counting vs. Context-predicting Semantic Vectors. ACL 2014.
• Goldberg, Y. (2017). Neural Network Methods for Natural Language Processing. Morgan & Claypool.
• Mikolov, T., Grave, E., Bojanowski, P., Puhrsch, C., & Joulin, A. (2018). Advances in Pre-Training Distributed Word Representations. LREC 2018. Authors' own comparison of FastText vs word2vec across benchmarks.
• Bolukbasi, T., Chang, K.-W., Zou, J., Saligrama, V., & Kalai, A. (2016). Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings. NeurIPS 2016. The canonical demonstration that static embeddings inherit corpus biases.