Embeddings from Scratch: From Word2Vec to E5

This is the first post of Arc 9, which is the retrieval and memory arc of the series. Everything in here lives downstream of one question: how do we turn arbitrary text into a number that a vector store can search? The answer in 2026 is "an embedding model," and that phrase covers about a decade of quietly-revolutionary ideas that got folded into one API call.

I've already spent one post on word2vec and another on the embedding table inside a trained transformer. Those posts are about one vector per token. This one is about where the field went next. Namely, how we got from one-vector-per-word to one-vector-per-word-in-context to one-vector-per-sentence, and why every modern retrieval system ends up training its embeddings with a loss that looks nothing like next-token prediction.

The payoff at the end: a model you can hand a sentence and get back 768 (or 384, or 1024) numbers that behave usefully under cosine similarity. That's the atom of RAG, of semantic search, of "find me the most similar ticket in the last year." The rest of the arc is about what you do with those numbers.

Text is not geometry

Vector search needs vectors. The relational-database version of "find me related documents" is a keyword match, and keyword matching has one legendary weakness: synonyms. A query about "refund policy" misses documents that use the word "return," and a query about "my package arrived damaged" misses documents that say "broken on delivery." These pairs share no tokens. To a bag-of-words index, they're unrelated.

The hope is that if you embed every document as a point in some high-dimensional space, and you embed the query as a point in the same space, then the nearest documents by some distance metric (usually cosine similarity) are the ones that are about the same thing regardless of the specific words they use.

That hope is doing a lot of work. It requires that "about the same thing" actually becomes a geometric fact in the embedding space. Nothing about a random neural network guarantees that. You have to train for it.

So the story of the last decade of embedding models is the story of people figuring out how to bake that property in. Different models, different losses, different training data, but the target is the same: similar meaning should land in similar places.

Prereqs

I'm going to assume you've got the earlier-arc atoms in your head already: dot products and cosine similarity, the idea of an embedding table inside a transformer, and the rough shape of how BERT-style encoders work. Everything in this post stacks on top of those.

If you want the one-sentence refresher: cosine similarity is a dot product normalized by length, so it measures angle between two vectors and ignores magnitude. Two vectors pointing the same direction have cosine 1, perpendicular is 0, opposite is -1. In practice for modern embedding models, similarities tend to live in a compressed high-positive band because the embeddings cluster on a narrow cone in the space (the anisotropy I'll get to below), not because vectors are "positive" in any coordinate sense.

A quick history, in four generations

1. word2vec and friends (2013)

One vector per word type, learned by predicting co-occurrence. You pass "the cat sat on the" and the model tries to predict "mat." After training, vectors for words that appear in similar contexts end up close together. I covered this in its own post, so I won't rebuild it here.

The thing word2vec can't do: the word "bank" has one vector. River bank and investment bank share a point in space, and every system downstream gets to deal with the ambiguity. For a retrieval system that has to decide whether "bank fees" and "river bank" are similar, this is a real problem.

2. Contextual embeddings (ELMo, BERT — 2018)

The move was to stop pretending a word has a single meaning. Instead of looking up one vector per token, you run the whole sequence through a bidirectional encoder and pull out the hidden state at each position. Now "bank" in "bank fees" and "bank" in "river bank" produce different vectors, because the encoder saw the surrounding context.

This was a huge step, but if you're trying to do retrieval, you still have a problem: each token has a vector, but you probably want to embed a whole sentence or a whole passage. The naive fix is to average the token vectors, and for a while that was the default, and it worked okay-ish. Off-the-shelf BERT sentence embeddings are not great for retrieval, as it turned out. The model wasn't trained for that job.

3. Sentence-BERT (2019)

Reimers and Gurevych took a pretrained BERT, wrapped it in a siamese structure (run two copies in parallel, one for each sentence in a pair), and fine-tuned it so that pairs of sentences labeled as "similar" in a labeled dataset ended up close in embedding space. The output is a single fixed-size vector per sentence, and the distance between those vectors actually tracks human judgments of similarity.

The pre-trained BERT isn't the right tool on its own for retrieval; the fine-tuned siamese version is. The architecture barely changes, and the training objective changes completely. The frame I keep coming back to: the loss is usually the story, not the architecture.

4. Contrastive retrieval models (DPR, E5, GTE, Nomic — 2020+)

The current crop. Sentence-BERT depended on a modestly-sized labeled similarity dataset, which capped how good it could get. The next leap came from going much bigger on the data and changing the loss from "regress on a similarity score" to contrastive learning: given a (query, relevant passage) pair, train the model to make their embeddings closer to each other than to any of the other passages in the batch.

Karpukhin et al.'s Dense Passage Retrieval showed this worked on open-domain QA, dramatically beating BM25 (the classic sparse-keyword baseline) on Wikipedia retrieval benchmarks. E5 pushed the idea further by pretraining on weakly-supervised pairs scraped from the web (things like title/abstract pairs, question/answer pairs from Stack Exchange, and Reddit post/top-comment pairs), producing something that works across a huge range of retrieval tasks with no task-specific fine-tuning.

These are the models you're actually using when you call an embeddings API in 2026. The architecture is usually a BERT- or T5-sized encoder. The pretraining is usually contrastive. And the thing that makes one better than another is almost entirely about the quality and diversity of the pairs used during training.

The minimal math: InfoNCE

Contrastive learning, stripped down, is pretty simple. You have:

• A query $q$ (a sentence).
• A positive $k^+$ (a sentence that is "relevant" to $q$ — a paraphrase, an answer, a paired passage).
• A set of negatives $k_1^-, k_2^-, \ldots$ (random unrelated sentences, usually other things in the same batch).

You pass each through the encoder, get embeddings $\mathbf{v}_q$, $\mathbf{v}^+$, $\mathbf{v}_1^-$, etc. Then the loss for this one query is:

This is called the InfoNCE loss, and if you squint it's just a softmax cross-entropy over "which of these candidates is the right one." The "correct answer" is always the positive. Everything else is a distractor.

The $\tau$ is a temperature that sharpens or softens the distribution. Small $\tau$ means the loss cares a lot about which item is slightly more similar. Large $\tau$ smooths things out. In practice $\tau$ gets set to something like 0.05, and people don't mess with it much.

The geometric picture is easier than the equation:

Notice the negatives come from the same mini-batch. This is called in-batch negatives and it's the main reason modern embedding models want huge batch sizes during training. The larger the batch, the more negatives per step, the stronger the contrastive signal. People routinely train with batch sizes of 8,192 or more for this exact reason.

One forward pass, one vector

At inference time, all of this disappears. You ship a single copy of the encoder. For any input sentence you do one forward pass through the encoder, take the hidden state at a designated position (usually the [CLS] token for BERT-style, or a mean-pool across tokens for T5-style), normalize it to unit length, and that's your embedding. No siamese anything, no contrastive anything. The training setup has already done its work.

A few things about this forward pass that matter in practice:

Instruction prefixes. Models in the E5 family were trained with a specific wrapper on the input: "query: ..." for queries and "passage: ..." for the documents being indexed. The asymmetry lets one model handle both sides of the retrieval without confusing them. Forgetting the prefix drops retrieval quality by double-digit percentage points.

Normalization. Almost every retrieval embedding model normalizes the output to unit length before you compare. Once you do that, cosine similarity and dot product and Euclidean distance all become monotonically equivalent (they rank pairs in the same order), so it basically doesn't matter which one you use. Most vector databases default to cosine.

Embedding dimension. You'll see numbers like 384, 768, 1024, 1536, 3072, 4096. Higher-dimensional embeddings capture more information but cost more to store and slower to search. A million documents at 768 floats is 3 GB of float32. At 4096, it's 16 GB. At scale, that matters.

The dimension tradeoff, and Matryoshka's trick

Here's a thing that used to be annoying: you had to pick your embedding dimension once, at indexing time, for your entire corpus. If you wanted to try a smaller model to save money, you had to re-embed everything. If you wanted to start with cheap 128-dim vectors and upgrade to 768-dim later for a subset of high-traffic queries, you couldn't.

Matryoshka Representation Learning (Kusupati et al., 2022) fixed this in an almost suspiciously clean way. The training loss is applied not just to the full-dimensional embedding, but to multiple nested prefixes of it at once: the first 64 dims, the first 128, the first 256, and so on up to the full 768. Each prefix has to be a good retrieval embedding in its own right.

The result is that at inference time, you can truncate to any of the training-time prefix sizes and the result still works. You literally just drop the tail and re-normalize. No retraining, no second model, no approximation. One vector, many usable resolutions, like those stacking Russian dolls the name is borrowed from.

This is the kind of thing that sounds like a neat academic trick until you realize OpenAI shipped it as text-embedding-3-large and -3-small with tunable dimensions via an API parameter, and now every major provider offers it as a first-class feature. Matryoshka is one of a small number of papers from the last five years that quietly became infrastructure.

Worked example

Let's put it all together. I'll pick five sentences deliberately designed to have two semantic clusters plus some ambiguity. Then I'll embed each one with a model in the E5 family and compute all-pairs cosine similarity. The resulting 5×5 matrix should light up the clusters visually.

The five sentences:

1. "How do I return a broken laptop?"
2. "What is your refund policy?"
3. "My package arrived damaged."
4. "Cats purr when they are content."
5. "The kitten curled up on the couch."

The first three are all about returns and refunds but they share almost no vocabulary. The last two are about cats but they don't share words with each other either (besides the "the"). A bag-of-words system would find this set mostly unrelated. An embedding model shouldn't.

Here's what the matrix looks like:

The viz above uses a clean stylized matrix (within-cluster ~0.8, cross-cluster ~0.2) so the block structure reads at a glance. When I actually run this through E5-base-v2, the numbers are more compressed: within-cluster ~0.75–0.83, cross-cluster ~0.65–0.70. That's because modern encoder embeddings live on a narrow cone in the space, not spread evenly around the unit sphere. The anisotropy I hammered on in the embedding-table post shows up here too. Same underlying phenomenon. What matters is that within-cluster is consistently higher than cross-cluster, so the ranking is correct even though the absolute numbers all look "high."

The point isn't the exact numbers. The point is the shape: within-cluster cells strongly lit up, cross-cluster cells mostly dark, and a few soft off-diagonal cells where the sentences genuinely share some latent overlap (e.g., "refund policy" and "damaged package" both live in the customer-service world).

Implementation sketch

The actual code for what we just did is anticlimactic. sentence-transformers is the Python library that wraps this whole ecosystem, and it's been the path of least resistance since Sentence-BERT.

# pip install sentence-transformers torch
 
from sentence_transformers import SentenceTransformer
import torch
 
# E5-base-v2 is a solid, small, open-weights retrieval model.
# 768-dim, ~110M params, trained with contrastive weak supervision.
model = SentenceTransformer("intfloat/e5-base-v2")
 
# The E5 family expects these prefixes. "query:" for search queries,
# "passage:" for documents being indexed. Our five sentences are all
# "passages" since we're just computing pairwise similarity.
sentences = [
    "passage: How do I return a broken laptop?",
    "passage: What is your refund policy?",
    "passage: My package arrived damaged.",
    "passage: Cats purr when they are content.",
    "passage: The kitten curled up on the couch.",
]
 
# Encode returns a (5, 768) tensor of embeddings. normalize_embeddings=True
# divides each row by its L2 norm, so dot products become cosine sims.
emb = model.encode(sentences, normalize_embeddings=True, convert_to_tensor=True)
print(emb.shape)   # torch.Size([5, 768])
 
# All-pairs cosine similarity. Because rows are unit-norm, this is just
# emb @ emb.T — one matmul, no loops.
sim = emb @ emb.T
print(sim.round(decimals=2))
# tensor([[1.00, 0.83, 0.77, 0.66, 0.69],
#         [0.83, 1.00, 0.75, 0.68, 0.65],
#         [0.77, 0.75, 1.00, 0.66, 0.67],
#         [0.66, 0.68, 0.66, 1.00, 0.78],
#         [0.69, 0.65, 0.67, 0.78, 1.00]])
# Note: cross-cluster scores look "high" (~0.65) because E5 embeddings
# live on a narrow anisotropic cone. The *ranking* is still correct:
# every within-cluster pair scores higher than every cross-cluster pair.
 
# Matryoshka-style truncation: first 256 dims, re-normalized.
emb_256 = emb[:, :256]
emb_256 = emb_256 / emb_256.norm(dim=1, keepdim=True)
sim_256 = emb_256 @ emb_256.T
# The structure survives. E5-base-v2 isn't formally MRL-trained but the
# first 256 dims already carry most of the retrieval signal; formally
# MRL-trained models (like OpenAI's text-embedding-3) make this exact.

Three lines of real work: load a model, encode the sentences, multiply the matrix by its own transpose. The pretraining, the contrastive loss, the curated billion pairs of weak supervision are all somewhere back in 2022, already done, sitting in a checkpoint on Hugging Face.

Misconceptions

"Bigger embedding dimension is always better." Higher dimensions capture more information, up to a point, but they also cost proportionally more in storage, in RAM during search, and in inference latency for ANN indexes. On most real retrieval tasks the quality-vs-dimension curve saturates hard somewhere between 256 and 768. Matryoshka models make the tradeoff legible: you can literally measure your app's retrieval quality at each prefix length and pick the smallest one that meets your bar. A lot of production systems land at 256 and never regret it.

"If I swap embedding models, I just need to re-embed my corpus." You need to re-embed your corpus and your queries with the same model. The embeddings from model A and the embeddings from model B live in different coordinate systems. A query vector from one and a document vector from the other produces a number, but that number is meaningless. This sounds obvious written down, and the failure mode is still easy to hit: upgrading the embedder on the query side without re-indexing the documents leaves the two halves of the system in incompatible coordinate systems.

"Cosine similarity is a real measure of semantic similarity." Cosine similarity is a real measure of alignment in the embedding space you happen to have. If your embedding space is any good (big well-trained contrastive model), that alignment correlates with human judgments of similarity well enough to be useful. If your embedding space is bad (raw BERT mean-pooled, or a model fine-tuned on the wrong domain), the cosine numbers are still perfectly valid as a geometric quantity and still perfectly useless for retrieval. The number isn't the thing; the space is. Treat cosine similarity as a diagnostic that only works when you already trust the encoder.

"One embedding model is enough." For a pure semantic search over one domain, yes. For production retrieval, almost never. The arc from here (dense-sparse hybrids, rerankers, query rewriting) is largely about stacking additional signals on top of the dense embeddings because dense embeddings alone systematically miss certain query types. Exact-match queries ("model number XR-482B") are a good example: dense embeddings don't help, sparse BM25 dominates, and a hybrid of the two plus a reranker is what production systems actually run.

What's next

This post built the model. The next post builds the database. You've got vectors coming out of an encoder; now you need to search a billion of them in a millisecond, which is a completely different engineering problem. The next post covers vector search and approximate nearest neighbors, including HNSW, IVF, product quantization, and why exact nearest-neighbor search is almost never what production systems actually do.

Additional reading (and watching)

• Devlin, J., et al. (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL 2019. The paper that made contextual embeddings mainstream and set up the encoder backbone every modern retrieval model still builds on.

• Reimers, N. & Gurevych, I. (2019). Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. EMNLP 2019. The paper that turned BERT into a usable sentence encoder via siamese fine-tuning on NLI and STS data.

• Oord, A. v. d., Li, Y., & Vinyals, O. (2018). Representation Learning with Contrastive Predictive Coding. arXiv:1807.03748. Introduces the InfoNCE loss that underlies essentially every modern contrastive embedding model.

• Karpukhin, V., et al. (2020). Dense Passage Retrieval for Open-Domain Question Answering. EMNLP 2020. DPR: the paper that demonstrated contrastive dense retrieval beating BM25 on open-domain QA, kicking off the current era.

• Wang, L., et al. (2022). Text Embeddings by Weakly-Supervised Contrastive Pre-training. arXiv:2212.03533. The E5 paper. Web-scale weakly-supervised pairs, instruction prefixes, and the recipe a lot of 2023+ open embedding models imitate.

• Chen, T., et al. (2020). A Simple Framework for Contrastive Learning of Visual Representations. ICML 2020. SimCLR. The clearest treatment of the InfoNCE-with-in-batch-negatives loss in one place, originally for vision but the math transfers directly.

• Gao, T., Yao, X., & Chen, D. (2021). SimCSE: Simple Contrastive Learning of Sentence Embeddings. EMNLP 2021. Showed that contrastive sentence embeddings work even with trivially-constructed positives (dropout masks as augmentation), and made clear how much batch size matters for the loss.

• Kusupati, A., et al. (2022). Matryoshka Representation Learning. NeurIPS 2022. The MRL paper. Trains nested prefixes jointly so one vector supports many dimensions at inference.

• OpenAI. (2024). New embedding models and API updates. OpenAI blog. The first major hosted embedding API to expose tunable output dimensions via MRL-style training.

• Ethayarajh, K. (2019). How Contextual are Contextualized Word Representations?. EMNLP 2019. Demonstrates that contextual embeddings live on an anisotropic cone rather than isotropically on the sphere, which is why cosine similarities between unrelated sentences bottom out at 0.2-ish, not 0.

• Lin, J., Ma, X., et al. (2021). A Replication Study of Dense Passage Retrieval. arXiv:2104.05740. Careful empirical study that establishes when DPR beats BM25 and when it doesn't, grounding the dense-sparse hybrid intuition.

• Muennighoff, N., et al. (2023). MTEB: Massive Text Embedding Benchmark. EACL 2023. The benchmark. See also the live leaderboard on Hugging Face.

• Nussbaum, Z., et al. (2024). Nomic Embed: Training a Reproducible Long Context Text Embedder. arXiv:2402.01613. A fully open-weights, open-data, open-code embedding model competitive with closed alternatives on MTEB.