Word2vec and the Embedding Revolution

How do you tell a computer that "cat" and "kitten" are related, but "cat" and "economics" aren't?

A simple answer was: you don't. Words were one-hot vectors. If your vocabulary has 50,000 words, "cat" is a vector with a 1 in position 7,243 and zeros everywhere else. "Kitten" is a 1 in position 12,891. The dot product between them is zero, same as the dot product between "cat" and "economics." Every word is equally distant from every other word. No notion of similarity is baked into the representation.

Dense word representations existed before word2vec. Latent Semantic Analysis built them by factoring word-document co-occurrence matrices in the late 1980s. Bengio's neural probabilistic language model (2003) learned word vectors as a byproduct of next-word prediction. Collobert and Weston (2008), Turian, Mnih, Huang, and the RNN language modeling line all produced learned distributed representations through the late 2000s. Word2vec made dense word vectors cheap to train at billion-token scale on a single machine, helping make them widely available for NLP tasks.

Dense vectors let a model represent similarities that one-hot identities do not. For example, training can place "cat" and "kitten" near one another when they appear in overlapping contexts. Similarity then depends on the learned coordinates rather than the arbitrary word IDs.

Mikolov and colleagues' word2vec papers in 2013 described efficient ways to learn those vectors by predicting nearby words.

The distributional hypothesis

There's a line from the linguist J.R. Firth, written in 1957, that neatly expresses the distributional intuition behind word embeddings: "You shall know a word by the company it keeps."

The distributional hypothesis says that words appearing in similar contexts tend to have similar meanings. "Cat" and "dog" both show up near words like "pet," "fur," "veterinarian." "King" and "queen" both show up near "throne," "crown," "royal." If you could somehow compress a word's typical context into a fixed-size vector, words with similar contexts would end up with similar vectors.

Skip-gram: predict context from a word

The skip-gram model works like this. You slide a window across your training corpus. In the center of the window is your center word, and the words around it (within some radius, say 2 words on each side) are the context words. The model's job is: given the center word, predict the context words.

Say your sentence is "the cat sat on the mat" and your window size is 2. When the center word is "sat," the training pairs are:

Skip-gram uses a single hidden layer with no activation function. The input is a one-hot vector for the center word, multiplied by $W \in \mathbb{R}^{V \times d}$, where $V$ is the vocabulary size and $d$ the embedding dimension. That multiplication selects the corresponding row of $W$: the word's embedding.

The hidden representation then gets multiplied by a second weight matrix $W' \in \mathbb{R}^{d \times V}$ to produce a score for every word in the vocabulary. A softmax over those scores gives a probability distribution over the vocabulary, and we train the model to assign high probability to the actual context words. (If you want to see this architecture drawn out stage by stage, Jay Alammar's illustrated explanation is still the clearest picture around.)

Training adjusts $W$ through the context-prediction objective. Its rows can then be reused as word embeddings in other tasks.

CBOW: the mirror image

There's a second variant called Continuous Bag of Words (CBOW) that flips the direction. Instead of predicting context from a center word, CBOW predicts the center word from its context. You average the embeddings of all the context words and use that average to predict the word in the middle.

CBOW trains faster because each training example uses multiple context words simultaneously (one forward pass, one gradient update). Skip-gram is slower but tends to do better on rare words, because each rare word still generates several training pairs. Negative sampling makes either architecture cheaper to train.

Skip-gram learns two vector tables, one for words used as the center and one for words used as context. The objective uses both. In common usage, people export the center (input) vectors as "the" word2vec embeddings, but the context (output) vectors are equally part of the model and show up later in analyses like Levy and Goldberg's matrix-factorization view.

Negative sampling: making it practical

There's a computational problem with the skip-gram softmax. That denominator sums over the entire vocabulary. If you have 50,000 words, every single training step requires computing 50,000 dot products and exponentials just to normalize the probability. That cost accumulates over millions of training pairs.

Mikolov's second paper introduced negative sampling as a solution. The intuition is straightforward: instead of computing a probability distribution over all 50,000 words, just train a binary classifier. For a real (center, context) pair, the model maximizes the log-sigmoid of the dot product between the center's input vector and the positive context's output vector. For each pair it also samples $k$ "negative" context words, and it maximizes the log-sigmoid of the negated score for each of those.

The negatives aren't drawn uniformly. In the original word2vec setup, they're sampled from a smoothed unigram distribution proportional to $f(w)^{0.75}$, which oversamples common words relative to rare words but less aggressively than raw frequency. Mikolov found this beat both uniform sampling and unsmoothed unigram sampling.

For each real training pair like (sat, cat), you sample maybe 5 negatives (say "elephant," "quarterly," "Nebraska," "jumps," "the"). The gradient step touches the center word's input vector, the positive context word's output vector, and the $k$ sampled negative output vectors. Everything else in the embedding tables stays put. So instead of updating roughly $V \cdot d$ output parameters per step, you update $(k+1) \cdot d$ of them.

The negative sampling objective for a single (center, context) pair looks like:

where $\sigma$ is the sigmoid function and $k$ is the number of negative samples (typically 5-20). The first term pushes the real pair's dot product up. The sum pushes the fake pairs' dot products down.

Levy and Goldberg later showed that skip-gram with negative sampling implicitly factorizes a shifted pointwise mutual information matrix under their analysis assumptions. This connects its predictive objective to count-and-factorize approaches such as LSA, though their losses and training procedures differ.

Subsampling and hierarchical softmax

Frequent-word subsampling. Words like "the," "of," and "a" are everywhere. They dominate every window but carry little distributional signal — almost any noun could be next to "the." Word2vec randomly discards each token during training with probability roughly $\max(0, 1 - \sqrt{t / f(w)})$, where $f(w)$ is the word's frequency and $t$ is a small constant (around $10^{-5}$). Common words get dropped often, rare words almost never. Discarding frequent tokens reduces training work and changes which words become neighbors within a window. Mikolov et al. reported improved accuracy as well as faster training with subsampling.

Hierarchical softmax. Negative sampling became the famous shortcut, but it wasn't the only one. The earlier word2vec implementation could also use hierarchical softmax, which arranges the vocabulary as a binary Huffman tree and predicts a path through the tree instead of scoring every word. That brings the per-step cost from $O(V)$ down to $O(\log V)$. The follow-up paper compares hierarchical softmax, NCE, and negative sampling head-to-head; negative sampling performed best on most tasks in that evaluation, especially with subsampling.

The analogy property

Some relationships between words appear as approximately parallel offsets in trained embeddings.

A well-known example subtracts "man" from "king" and adds "woman," then searches for a nearby vector such as "queen." The result depends on the trained vectors and nearest-neighbor procedure:

Mikolov et al. evaluated offsets for relationships including gender, country-capital pairs, and verb forms. Examples such as Paris − France + Italy ≈ Rome suggest that some relationships have consistent directions in the learned space. These offsets do not succeed for every pair.

Here's a hand-drawn 2D example of that geometry. Click the analogy buttons to see the vector arithmetic:

Notice how the royalty cluster sits in one region, the gender words in another, and the countries/capitals off to the right. The spacing within each cluster encodes structure too. The offset from "man" to "king" is roughly parallel to the offset from "woman" to "queen."

Implementation sketch

A skip-gram model with negative sampling needs two embedding tables and the sampled loss:

import torch
import torch.nn as nn
import torch.nn.functional as F
 
class SkipGram(nn.Module):
    def __init__(self, vocab_size: int, embed_dim: int):
        super().__init__()
        # These two matrices are the whole model
        self.center_embeddings = nn.Embedding(vocab_size, embed_dim)
        self.context_embeddings = nn.Embedding(vocab_size, embed_dim)
 
    def forward(self, center_ids, context_ids, negative_ids):
        # center_ids:   (batch,)
        # context_ids:  (batch,)
        # negative_ids: (batch, num_negatives)
 
        center = self.center_embeddings(center_ids)      # (batch, d)
        context = self.context_embeddings(context_ids)    # (batch, d)
        negatives = self.context_embeddings(negative_ids) # (batch, k, d)
 
        # Positive: dot product of real (center, context) pairs
        pos_score = (center * context).sum(dim=-1)        # (batch,)
        pos_loss = F.logsigmoid(pos_score)
 
        # Negative: dot product with random "wrong" context words
        neg_score = torch.bmm(
            negatives, center.unsqueeze(2)
        ).squeeze(2)                                      # (batch, k)
        neg_loss = F.logsigmoid(-neg_score).sum(dim=-1)
 
        # Maximize positive, minimize negative
        return -(pos_loss + neg_loss).mean()
 
# Usage:
# model = SkipGram(vocab_size=50000, embed_dim=300)
# loss = model(center_batch, context_batch, negative_batch)
# loss.backward()

After training, center_embeddings contains the input word vectors. Their dimension is configurable; the example uses 300.

The remaining training-loop work is to generate center-context pairs and draw negative samples; the module above computes their loss.

Misconceptions

"Word2vec understands meaning." Its objective captures distributional relationships from text, without grounding words in objects or experiences. "Cat" and "kitten" can have similar vectors because of overlapping contexts; cosine similarity alone does not establish what a model knows about either animal.

"The analogy trick works reliably." Table 1 of Mikolov et al. (2013) reports 59–61% total word-analogy accuracy for its 300-dimensional negative-sampling models, depending on the number of negative samples and subsampling. These figures describe that evaluation setup, not a universal accuracy for word vectors. Syntactic analogies (walk:walked :: swim:swam) tend to work better than semantic ones, and the method is sensitive to how you measure nearest neighbor and whether the input words are excluded from the search.

"You need huge datasets." The released Google News vectors were trained on about 100 billion words. Smaller domain corpora can also produce useful neighbors when words appear often enough in informative contexts. Evaluate the vectors on the intended task rather than using corpus size alone as a quality measure.

"One vector per word is enough." Word2vec is a static embedding model: every word type gets exactly one vector. "Bank" in "river bank" and "bank" in "investment bank" share the same point in space, even though they mean very different things. Contextual models like ELMo and BERT, and the hidden representations inside transformers, compute context-dependent vectors. A transformer still starts with a fixed token-embedding lookup; its later layers add context.

"Word2vec handles new words gracefully." It doesn't. If a word wasn't in the training vocabulary, there's no row for it in the embedding table and no way to construct one. fastText (the next post) addresses this by composing a word's vector from character n-grams, which lets it produce reasonable vectors for unseen words and morphological variants — useful for languages with richer morphology than English.

"The analogy property is purely a feature." The same geometry that lets you do king − man + woman ≈ queen also encodes social stereotypes present in the training corpus. Bolukbasi et al. (2016) showed that word2vec vectors trained on Google News encode gender stereotypes along the same kinds of directions that produce the clean analogies; Caliskan et al. (2017) showed that text corpora contain recoverable human-like biases that embeddings reproduce. These associations come from learning statistics from human text, and downstream systems can reproduce those associations unless their design or training addresses them.

What's next

The next post covers GloVe, FastText, and the embedding zoo, how GloVe fits word co-occurrence statistics, how FastText adds subword features that can construct vectors for words absent from the word vocabulary, and what survives into modern embedding models.

Additional reading (and watching)

• Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). Efficient Estimation of Word Representations in Vector Space. arXiv:1301.3781.
• Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013). Distributed Representations of Words and Phrases and their Compositionality. arXiv:1310.4546.
• Levy, O. & Goldberg, Y. (2014). Neural Word Embedding as Implicit Matrix Factorization. NeurIPS 2014.
• Alammar, J. The Illustrated Word2vec.
• Bengio, Y., Ducharme, R., Vincent, P., & Jauvin, C. (2003). A Neural Probabilistic Language Model. JMLR.
• Collobert, R. & Weston, J. (2008). A Unified Architecture for Natural Language Processing. ICML 2008.
• Vaswani, A., et al. (2017). Attention Is All You Need. NeurIPS 2017. The transformer paper; modern LLMs still begin with a learned token-embedding lookup, but those embeddings are trained end-to-end inside a much larger contextual model rather than by the word2vec objective.
• Karpukhin, V., et al. (2020). Dense Passage Retrieval for Open-Domain Question Answering. EMNLP 2020. One of the key papers showing that dense dual-encoder retrieval could beat strong sparse baselines like BM25 on open-domain QA, helping establish the dense-retrieval side of modern RAG systems.
• Barkan, O. & Koenigstein, N. (2016). Item2Vec: Neural Item Embedding for Collaborative Filtering. IEEE MLSP 2016. Showed that the same skip-gram-with-negative-sampling template applies to collaborative filtering by treating item co-occurrence the way word2vec treats word co-occurrence.
• Bolukbasi, T., et al. (2016). Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings. NeurIPS 2016. Showed that the same geometry behind the analogy results encodes gender stereotypes inherited from the training corpus.
• Caliskan, A., Bryson, J., & Narayanan, A. (2017). Semantics derived automatically from language corpora contain human-like biases. Science. Demonstrated that human-like biases (gender, race) are recoverable from word embeddings via implicit-association-style tests.
• Devlin, J., et al. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL 2019. The contextual-embedding turn: token vectors computed from the surrounding sentence in all layers, instead of one fixed vector per type.