Word2vec and the Embedding Revolution

How do you tell a computer that "cat" and "kitten" are related, but "cat" and "economics" aren't?

A simple answer was: you don't. Words were one-hot vectors. If your vocabulary has 50,000 words, "cat" is a vector with a 1 in position 7,243 and zeros everywhere else. "Kitten" is a 1 in position 12,891. The dot product between them is zero, same as the dot product between "cat" and "economics." Every word is equally distant from every other word. No notion of similarity is baked into the representation.

Dense word representations existed before word2vec. Latent Semantic Analysis built them by factoring word-document co-occurrence matrices in the late 1980s. Bengio's neural probabilistic language model (2003) learned word vectors as a byproduct of next-word prediction. Collobert and Weston (2008), Turian, Mnih, Huang, and the RNN language modeling line all produced learned distributed representations through the late 2000s. What word2vec did wasn't invent dense vectors. It made high-quality dense vectors cheap to train at billion-token scale on a single machine, which is what tipped them from research curiosity into ubiquitous infrastructure.

This is a problem if you're trying to build language models, because language is built on similarity. You know what a sentence means partly because you know which words could substitute for which. So what you really want is a representation where similar words live nearby in some continuous space, and dissimilar words live far apart. You want a dense vector, maybe 300 dimensions, where the position of the word carries information about what it means.

In 2013, Tomas Mikolov and his team at Google published a paper that changed the field. The method was called word2vec, and the core idea was simple.

The distributional hypothesis

There's a line from the linguist J.R. Firth, written in 1957, that became the entire theoretical backbone of word embeddings: "You shall know a word by the company it keeps."

The distributional hypothesis says that words appearing in similar contexts tend to have similar meanings. "Cat" and "dog" both show up near words like "pet," "fur," "veterinarian." "King" and "queen" both show up near "throne," "crown," "royal." If you could somehow compress a word's typical context into a fixed-size vector, words with similar contexts would end up with similar vectors.

Word2vec is a clever way of learning those context-based vectors.

Before we get into how it learns them, here's what the finished product feels like. In a well-trained embedding space, cosine similarity between vectors ranks words by meaning. The top match for "cat" is "kitten", then "dog", then "tiger". The top match for "car" is "truck", then "bike". "Chair" doesn't cleanly belong to any of those clusters, so it stays near the bottom for everything:

Skip-gram: predict context from a word

The skip-gram model works like this. You slide a window across your training corpus. In the center of the window is your center word, and the words around it (within some radius, say 2 words on each side) are the context words. The model's job is: given the center word, predict the context words.

Say your sentence is "the cat sat on the mat" and your window size is 2. When the center word is "sat," the training pairs are:

You generate these pairs for every position in the corpus. Millions of them.

The architecture is barely a neural network. It's a single hidden layer with no activation function. The input is a one-hot vector for the center word. That gets multiplied by a weight matrix $W \in \mathbb{R}^{V \times d}$, where $V$ is the vocabulary size and $d$ is the embedding dimension (typically 100-300). This multiplication is a lookup in disguise: pulling out the row of $W$ corresponding to the center word. That row is the word's embedding.

The hidden representation then gets multiplied by a second weight matrix $W' \in \mathbb{R}^{d \times V}$ to produce a score for every word in the vocabulary. A softmax over those scores gives a probability distribution over the vocabulary, and we train the model to assign high probability to the actual context words. (If you want to see this architecture drawn out stage by stage, Jay Alammar's illustrated explanation is still the clearest picture around.)

And here's the key insight: we don't actually care about the predictions. We care about the weight matrix $W$. After training, the rows of $W$ are the word embeddings. The prediction task is just a vehicle for learning useful representations.

CBOW: the mirror image

There's a second variant called Continuous Bag of Words (CBOW) that flips the direction. Instead of predicting context from a center word, CBOW predicts the center word from its context. You average the embeddings of all the context words and use that average to predict the word in the middle.

CBOW trains faster because each training example uses multiple context words simultaneously (one forward pass, one gradient update). Skip-gram is slower but tends to do better on rare words, because each rare word still generates several training pairs. The most famous pretrained word2vec vectors used skip-gram with negative sampling, and that's the variant most downstream work built on. But the architecture barely matters compared to the next trick.

Skip-gram learns two vector tables, one for words used as the center and one for words used as context. The objective uses both. In common usage, people export the center (input) vectors as "the" word2vec embeddings, but the context (output) vectors are equally part of the model and show up later in analyses like Levy and Goldberg's matrix-factorization view.

Negative sampling: making it practical

There's a computational problem with the skip-gram softmax. That denominator sums over the entire vocabulary. If you have 50,000 words, every single training step requires computing 50,000 dot products and exponentials just to normalize the probability. For millions of training pairs, this is brutal.

Mikolov's second paper introduced negative sampling as a solution. The intuition is straightforward: instead of computing a probability distribution over all 50,000 words, just train a binary classifier. For a real (center, context) pair, the model maximizes the log-sigmoid of the dot product between the center's input vector and the positive context's output vector. For each pair it also samples $k$ "negative" context words, and it maximizes the log-sigmoid of the negated score for each of those.

The negatives aren't drawn uniformly. In the original word2vec setup, they're sampled from a smoothed unigram distribution proportional to $f(w)^{0.75}$, which oversamples common words relative to rare words but less aggressively than raw frequency. Mikolov found this beat both uniform sampling and unsmoothed unigram sampling.

For each real training pair like (sat, cat), you sample maybe 5 negatives (say "elephant," "quarterly," "Nebraska," "jumps," "the"). The gradient step touches the center word's input vector, the positive context word's output vector, and the $k$ sampled negative output vectors. Everything else in the embedding tables stays put. So instead of updating roughly $V \cdot d$ output parameters per step, you update $(k+1) \cdot d$ of them.

The negative sampling objective for a single (center, context) pair looks like:

where $\sigma$ is the sigmoid function and $k$ is the number of negative samples (typically 5-20). The first term pushes the real pair's dot product up. The sum pushes the fake pairs' dot products down.

This took word2vec from an interesting idea to something you could actually train on a big corpus in hours on a single machine. Levy and Goldberg later showed that skip-gram with negative sampling is, under reasonable assumptions, implicitly factorizing a shifted pointwise mutual information matrix between words and contexts. So a "neural" predictive model turns out to be closely related to the classical count-and-factorize approach (LSA, Hellinger PCA on co-occurrence counts), just with a different loss and an SGD-based solver. But in 2013, what mattered was that it worked, and it was fast.

The other tricks that made it work

Negative sampling shrinks the cost of the output side. Two other tricks shrink the input data problem and round out the recipe.

Frequent-word subsampling. Words like "the," "of," and "a" are everywhere. They dominate every window but carry little distributional signal — almost any noun could be next to "the." Word2vec randomly discards each token during training with probability roughly $1 - \sqrt{t / f(w)}$, where $f(w)$ is the word's frequency and $t$ is a small constant (around $10^{-5}$). Common words get dropped often, rare words almost never. The corpus shrinks, training speeds up, and rare-word vectors get noticeably sharper because their context windows aren't drowning in stop words.

Hierarchical softmax. Negative sampling became the famous shortcut, but it wasn't the only one. The earlier word2vec implementation could also use hierarchical softmax, which arranges the vocabulary as a binary Huffman tree and predicts a path through the tree instead of scoring every word. That brings the per-step cost from $O(V)$ down to $O(\log V)$. The follow-up paper compares hierarchical softmax, NCE, and negative sampling head-to-head; negative sampling won on most tasks, especially with subsampling turned on, which is why it became the default story.

The point: the win wasn't a single objective. It was a recipe — sliding windows, cheap local updates, smoothed negative sampling, frequent-word subsampling, hierarchical softmax as an alternative output layer, and an aggressive multithreaded C implementation.

The analogy property

And then something interesting fell out. When people started probing the trained embeddings, they found linear structure.

Take the vector for "king," subtract the vector for "man," add the vector for "woman," and the nearest neighbor to the result is "queen."

This suggested that the embedding space had learned a consistent "gender" direction, and a consistent "royalty" direction, and those directions composed linearly. The same trick worked for country-capital relationships (Paris - France + Italy ≈ Rome), verb tenses (walking - walked + swam ≈ swimming), and a bunch of other semantic relationships.

Here's what that looks like projected down to 2D. Click the analogy buttons to see the vector arithmetic:

Notice how the royalty cluster sits in one region, the gender words in another, and the countries/capitals off to the right. The spacing within each cluster encodes structure too. The offset from "man" to "king" is roughly parallel to the offset from "woman" to "queen." That parallelism is the analogy property showing up geometrically.

Implementation sketch

The actual model code is really small. Here's a minimal skip-gram with negative sampling in PyTorch:

import torch
import torch.nn as nn
import torch.nn.functional as F
 
class SkipGram(nn.Module):
    def __init__(self, vocab_size: int, embed_dim: int):
        super().__init__()
        # These two matrices are the whole model
        self.center_embeddings = nn.Embedding(vocab_size, embed_dim)
        self.context_embeddings = nn.Embedding(vocab_size, embed_dim)
 
    def forward(self, center_ids, context_ids, negative_ids):
        # center_ids:   (batch,)
        # context_ids:  (batch,)
        # negative_ids: (batch, num_negatives)
 
        center = self.center_embeddings(center_ids)      # (batch, d)
        context = self.context_embeddings(context_ids)    # (batch, d)
        negatives = self.context_embeddings(negative_ids) # (batch, k, d)
 
        # Positive: dot product of real (center, context) pairs
        pos_score = (center * context).sum(dim=-1)        # (batch,)
        pos_loss = F.logsigmoid(pos_score)
 
        # Negative: dot product with random "wrong" context words
        neg_score = torch.bmm(
            negatives, center.unsqueeze(2)
        ).squeeze(2)                                      # (batch, k)
        neg_loss = F.logsigmoid(-neg_score).sum(dim=-1)
 
        # Maximize positive, minimize negative
        return -(pos_loss + neg_loss).mean()
 
# Usage:
# model = SkipGram(vocab_size=50000, embed_dim=300)
# loss = model(center_batch, context_batch, negative_batch)
# loss.backward()

Two embedding matrices, some dot products, a sigmoid. The center_embeddings matrix after training is your word embedding table. Each row is a 300-dimensional vector that encodes something about what that word means, learned entirely from context patterns.

The training data generation is the tedious part: you need to slide your window across the corpus, extract (center, context) pairs, and sample negatives proportional to word frequency (raised to the 0.75 power, a trick Mikolov found worked well empirically). But the model itself is almost disappointingly simple.

Misconceptions

"Word2vec understands meaning." It doesn't. Word2vec learns distributional similarity, which correlates with meaning but isn't the same thing. The model has no grounding. It doesn't know that a cat is a small furry animal. It knows that "cat" appears in similar contexts to "dog" and "kitten." Useful, but calling it understanding is a stretch. The vectors encode co-occurrence statistics compressed through a neural bottleneck.

"The analogy trick works reliably." The king/queen example is famous precisely because it works so cleanly. In practice, total analogy accuracy on the Google Analogy Test Set in the original papers ranged from the low 50s to the mid 60s depending on corpus size, dimensionality, architecture, and training objective: the 300-dimensional skip-gram trained on 783M words landed near 53%, a 1000-dimensional skip-gram trained on 6B words got to about 66%, and skip-gram with negative sampling fell roughly in the high 50s to low 60s depending on $k$ and subsampling. Syntactic analogies (walk:walked :: swim:swam) tend to work better than semantic ones, and the method is sensitive to how you measure nearest neighbor and whether the input words are excluded from the search. The analogy property is real and interesting, but it's a tendency, not a law.

"You need huge datasets." The pretrained vectors Google released alongside the second word2vec paper were trained on a Google News corpus of about 100 billion words, which is part of where this assumption comes from. But you can get surprisingly useful embeddings from much smaller corpora, especially for domain-specific tasks. A few million words of medical text will give you medical embeddings where "aspirin" and "ibuprofen" cluster together even if "aspirin" and "ibuprofen" rarely appear in the same sentence.

"One vector per word is enough." Word2vec is a static embedding model: every word type gets exactly one vector. "Bank" in "river bank" and "bank" in "investment bank" share the same point in space, even though they mean very different things. Contextual models like ELMo and BERT, and the token embeddings inside modern transformers, fix this by computing a token's vector from its surrounding sentence rather than looking it up directly. That shift is one of the main things that separates pre-2018 NLP from what came after.

"Word2vec handles new words gracefully." It doesn't. If a word wasn't in the training vocabulary, there's no row for it in the embedding table and no way to construct one. fastText (the next post) addresses this by composing a word's vector from character n-grams, which lets it produce reasonable vectors for unseen words and morphological variants — useful for languages with richer morphology than English.

"The analogy property is purely a feature." The same geometry that lets you do king − man + woman ≈ queen also encodes social stereotypes present in the training corpus. Bolukbasi et al. (2016) showed that word2vec vectors trained on Google News encode gender stereotypes along the same kinds of directions that produce the clean analogies; Caliskan et al. (2017) showed that text corpora contain recoverable human-like biases that embeddings reproduce. This isn't a bug in the arithmetic. It's a consequence of learning statistics from human text, and any downstream system that uses these vectors inherits it.

What's next

The next post covers GloVe, FastText, and the embedding zoo, how GloVe recasts this whole thing as factorizing a word co-occurrence matrix, how FastText adds a subword trick that fixes word2vec's out-of-vocabulary problem, and what survives into modern embedding models.

Additional reading (and watching)

• Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). Efficient Estimation of Word Representations in Vector Space. arXiv:1301.3781.
• Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013). Distributed Representations of Words and Phrases and their Compositionality. arXiv:1310.4546.
• Levy, O. & Goldberg, Y. (2014). Neural Word Embedding as Implicit Matrix Factorization. NeurIPS 2014.
• Alammar, J. The Illustrated Word2vec.
• Bengio, Y., Ducharme, R., Vincent, P., & Jauvin, C. (2003). A Neural Probabilistic Language Model. JMLR.
• Collobert, R. & Weston, J. (2008). A Unified Architecture for Natural Language Processing. ICML 2008.
• Vaswani, A., et al. (2017). Attention Is All You Need. NeurIPS 2017. The transformer paper; modern LLMs still begin with a learned token-embedding lookup, but those embeddings are trained end-to-end inside a much larger contextual model rather than by the word2vec objective.
• Karpukhin, V., et al. (2020). Dense Passage Retrieval for Open-Domain Question Answering. EMNLP 2020. One of the key papers showing that dense dual-encoder retrieval could beat strong sparse baselines like BM25 on open-domain QA, helping establish the dense-retrieval side of modern RAG systems.
• Barkan, O. & Koenigstein, N. (2016). Item2Vec: Neural Item Embedding for Collaborative Filtering. IEEE MLSP 2016. Showed that the same skip-gram-with-negative-sampling template applies to collaborative filtering by treating item co-occurrence the way word2vec treats word co-occurrence.
• Bolukbasi, T., et al. (2016). Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings. NeurIPS 2016. Showed that the same geometry behind the analogy results encodes gender stereotypes inherited from the training corpus.
• Caliskan, A., Bryson, J., & Narayanan, A. (2017). Semantics derived automatically from language corpora contain human-like biases. Science. Demonstrated that human-like biases (gender, race) are recoverable from word embeddings via implicit-association-style tests.
• Devlin, J., et al. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL 2019. The contextual-embedding turn: token vectors computed from the surrounding sentence in all layers, instead of one fixed vector per type.