GloVe, FastText, and the Embedding Zoo

Why are there so many word embedding methods?

GloVe, word2vec, and FastText all produce word vectors that can capture similar relationships. Their training procedures differ, and FastText also changes how a word gets its vector. Those differences matter when a word is rare or absent from the training vocabulary.

Before word2vec, methods such as Latent Semantic Analysis and PMI matrix factorization built word representations from co-occurrence counts. Word2vec trained vectors by predicting context words. These became known as count-based and prediction-based approaches.

The count-based and prediction-based approaches have a mathematical connection: both can be understood through matrix factorization.

GloVe: Making Co-occurrence Counts Learnable

GloVe (Global Vectors for Word Representation) came out of Stanford in 2014. Pennington, Socher, and Manning noticed that raw co-occurrence counts are noisy and hard to use directly. Ratios of co-occurrence probabilities can highlight a useful distinction between two words.

Here's the classic example from the paper. Let $P(k \mid w)$ be the probability that word $k$ appears in the context of word $w$, estimated from a big corpus. Consider the words "ice" and "steam," and look at how various probe words $k$ relate to them:

"Solid" co-occurs much more with "ice" than "steam," so the ratio is large. "Gas" goes the other way. "Water" and "fashion" are either related to both or neither, so their ratios sit near 1. The ratio $P(k \mid \text{ice}) / P(k \mid \text{steam})$ cleanly separates words that discriminate between the two concepts from words that don't.

GloVe is motivated by these ratios, but the implemented objective is something simpler: a weighted least-squares regression that fits dot products (plus bias terms) to log co-occurrence counts. The actual training equation is:

where $X_{ij}$ is the raw co-occurrence count, $w_i$ and $\tilde{w}_j$ are the center and context vectors, and $b_i$, $\tilde{b}_j$ are scalar biases per word. The biases absorb word- and context-frequency effects, which lets the dot product focus on relational structure rather than just raw popularity. A weighting function on the loss caps the influence of extremely frequent co-occurrences (so "the" doesn't dominate). The model learns two vector tables ($w$ and $\tilde{w}$), and the final embedding for a word is typically their sum.

The connection back to ratios is what makes the simplification work: $\log(P(k|i) / P(k|j)) = \log P(k|i) - \log P(k|j)$, so a model that fits log-counts well also encodes the discriminative structure the ratio table exposes, as differences between learned vectors.

So GloVe is not a context-prediction model in the word2vec sense. There is no softmax or negative-sampling classifier. It first constructs sparse global co-occurrence statistics — only the non-zero $X_{ij}$ entries, which is far smaller than the full $V \times V$ matrix — then trains a shallow log-bilinear model on those entries with a weighted regression objective. The training loop is just gradient descent.

The Connection to Matrix Factorization

In 2014, Levy and Goldberg published a paper proving that word2vec's skip-gram model with negative sampling (SGNS) is implicitly factorizing a matrix. If each word-context dot product can be optimized independently and negatives follow the empirical unigram distribution, its optimum is the pointwise mutual information (PMI), shifted by a constant:

where $k$ is the number of negative samples. Low-dimensional embeddings cannot generally reach every independent optimum, and the usual frequency-to-the-0.75 negative sampler changes the target further. PMI is a classical co-occurrence statistic that NLP researchers had been computing from count matrices for years. Under these assumptions, skip-gram is implicitly factorizing a shifted PMI matrix. GloVe explicitly fits a weighted log-count matrix with bias terms. Those are different objectives over closely related co-occurrence statistics, not the same matrix.

The objectives differ in how they weight word-context observations and handle frequent words. Baroni et al. found that predictive methods tended to win the benchmarks they tested; Levy et al. (2015) showed that preprocessing and hyperparameter choices explained much of the measured gap.

Goldberg's textbook on neural methods for NLP develops the matrix-factorization connection in more detail.

FastText: Words Aren't Atomic

Word2vec and GloVe both treat words as atomic units. The word "unforgettable" gets one vector, learned from its contexts. If "unforgettable" doesn't appear in your training corpus, you get nothing. It's out-of-vocabulary (OOV), and your only option is a generic unknown token.

Bojanowski et al. at Facebook addressed this in 2017 with FastText. The core idea is simple: represent each word as a bag of character n-grams, and learn vectors for the n-grams instead of (or in addition to) whole words.

Take the word "unforgetting." FastText would decompose it into character n-grams (using $n$ from 3 to 6 by default), plus special boundary markers:

The paper represents a word using the sum of its n-gram vectors, plus a whole-word vector when that word is in the vocabulary. The fastText library averages the contributing vectors when returning a word vector. Even if "unforgetting" never appeared in training, many of its n-grams (like "for", "get", "ing") appeared in thousands of other words. FastText constructs the new word’s vector by composing those pieces.

Subword sharing is useful to investigate for languages with many inflected forms, such as Turkish and Finnish. Whole-word tables learn a separate row for each form; FastText lets forms with shared substrings update shared parameters.

The skip-gram training objective resembles word2vec, but its input representation combines the word and subword vectors. Gradients update the shared n-gram vectors. This can capture recurring spelling and morphological patterns, without explicitly labeling prefixes or suffixes.

The Embedding Zoo: A Comparison

The methods differ in their training data structures and treatment of unseen words:

Analogy relationships have been observed with all three methods. The constructed coordinates below show the geometry of a parallel offset, and how rotating the space preserves it.

Worked Example: Loading GloVe and Computing Similarities

GloVe vectors are distributed as plain text files, with one word and its vector per line. The loader below expects the downloaded and extracted glove.6B.100d.txt file from Stanford's GloVe project.

import numpy as np
 
def load_glove(path):
    """Load GloVe vectors from a text file."""
    embeddings = {}
    with open(path, "r", encoding="utf-8") as f:
        for line in f:
            parts = line.strip().split()
            word = parts[0]
            vec = np.array(parts[1:], dtype=np.float32)
            embeddings[word] = vec
    return embeddings
 
def cosine_sim(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))
 
# Load pretrained GloVe (glove.6B.100d.txt)
glove = load_glove("glove.6B.100d.txt")
 
# Classic analogy: king - man + woman ≈ queen
# Exclude the input words from the search — otherwise one of them
# usually comes back as the nearest neighbor of the query vector.
query = glove["king"] - glove["man"] + glove["woman"]
exclude = {"king", "man", "woman"}
sims = {
    w: cosine_sim(query, v)
    for w, v in glove.items()
    if w not in exclude
}
top5 = sorted(sims, key=sims.get, reverse=True)[:5]
print("king - man + woman =", top5)
# Inspect the actual ranking for the file you loaded.
 
# Semantic similarity
pairs = [("cat", "dog"), ("cat", "car"), ("good", "great")]
for w1, w2 in pairs:
    sim = cosine_sim(glove[w1], glove[w2])
    print(f"  sim({w1}, {w2}) = {sim:.3f}")
 

The following code enumerates the character n-grams used to construct a FastText word representation:

Misconceptions

"GloVe is count-based, word2vec is neural, so they're fundamentally different." Levy and Goldberg showed a connection between the objectives. Skip-gram with negative sampling is implicitly factoring a shifted PMI matrix. GloVe explicitly factors a weighted log-count matrix. Different objectives, very similar low-rank factorizations of a similar underlying signal.

"FastText is always better because it handles OOV." Subword sharing lets FastText construct vectors for unseen words, but shared spelling need not imply shared meaning. Compare methods on the intended task, including rare words and unseen forms, to measure whether that sharing helps.

"These methods capture meaning." They capture distributional statistics. "Meaning," in the GloVe/word2vec/FastText sense, is the pattern of words a word tends to appear near. This is enough to reproduce striking semantic relationships, and it also allows them to reproduce stereotypes present in the training corpus. Bolukbasi et al. showed back in 2016 that pretrained word vectors can reproduce associations such as "man : programmer :: woman : homemaker". Those associations reflect patterns in the training data.

What's next

Static vectors give every word a single fixed point in space, no matter how it's being used. The next step is sequences: models that read one word at a time and update a running state as they go. That's Recurrent Neural Networks and Sequence Modeling, where the hidden state carries information from earlier tokens.

Additional reading (and watching)

• fastText. Word vector implementation. getWordVector averages the contributing input vectors.

• fastText. Word representations tutorial. Character n-gram settings, model files, and vectors for unseen words.

• Levy, O., Goldberg, Y., & Dagan, I. (2015). Improving Distributional Similarity with Lessons Learned from Word Embeddings. TACL. How preprocessing and hyperparameters affect comparisons between embedding methods.

• Pennington, J., Socher, R., & Manning, C.D. (2014). GloVe: Global Vectors for Word Representation. EMNLP 2014.

• Levy, O. & Goldberg, Y. (2014). Neural Word Embedding as Implicit Matrix Factorization. NeurIPS 2014. Proved that skip-gram with negative sampling is implicitly factorizing a shifted PMI matrix, unifying "prediction-based" and "count-based" embedding methods.

• Bojanowski, P., Grave, E., Joulin, A., & Mikolov, T. (2017). Enriching Word Vectors with Subword Information. TACL, 5, 135-146.

• Baroni, M., Dinu, G., & Kruszewski, G. (2014). Don't Count, Predict! A Systematic Comparison of Context-counting vs. Context-predicting Semantic Vectors. ACL 2014.

• Goldberg, Y. (2017). Neural Network Methods for Natural Language Processing. Morgan & Claypool.

• Mikolov, T., Grave, E., Bojanowski, P., Puhrsch, C., & Joulin, A. (2018). Advances in Pre-Training Distributed Word Representations. LREC 2018. Authors' own comparison of FastText vs word2vec across benchmarks.

• Bolukbasi, T., Chang, K.-W., Zou, J., Saligrama, V., & Kalai, A. (2016). Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings. NeurIPS 2016. The canonical demonstration that static embeddings inherit corpus biases.