Embeddings from Scratch: From Word2Vec to E5

How do we turn a passage into a vector that search can use? This arc starts with the encoder: the model that makes a query and a relevant passage comparable even when they use different words.

Word2vec and the embedding table assign vectors to vocabulary entries. Retrieval needs a representation of a whole query or passage, including how its words are used together. Training objectives can reward similarities that help rank relevant passages.

Text is not geometry

Plain term matching can miss relevant passages that use different words. Without expansion, “refund policy” does not match “return,” and “my package arrived damaged” shares no terms with “broken on delivery.” An embedding model can learn to rank such pairs together.

The hope is that if you embed every document as a point in some high-dimensional space, and you embed the query as a point in the same space, then the nearest documents by some distance metric (usually cosine similarity) are the ones that are about the same thing regardless of the specific words they use.

Retrieval training rewards that relationship between relevance and geometry. The architecture alone does not establish it: the objective and training pairs determine which similarities the model learns.

Prereqs

This uses dot products and cosine similarity, the idea of an embedding table inside a transformer, and the rough shape of how BERT-style encoders work.

Cosine similarity is a dot product normalized by length: it measures alignment and ignores magnitude. Nonzero vectors pointing in the same direction have cosine 1, perpendicular vectors 0, and opposite vectors −1. The distribution of text-embedding scores depends on the model and its training; a high positive score is not a universal relevance threshold.

A quick history, in four generations

1. word2vec and friends (2013)

Word2vec learns one vector per word type from local co-occurrence. Skip-gram predicts neighboring words from a center word; CBOW predicts a center word from its surrounding window. Neither is the left-to-right next-token objective used by a causal language model.

The thing word2vec can't do: the word "bank" has one vector. River bank and investment bank share a point in space, and every system downstream gets to deal with the ambiguity. For a retrieval system that has to decide whether "bank fees" and "river bank" are similar, this is a real problem.

2. Contextual embeddings (ELMo, BERT — 2018)

To represent a word in context, you run the whole sequence through a bidirectional encoder and pull out the hidden state at each position. Now "bank" in "bank fees" and "bank" in "river bank" produce different vectors, because the encoder saw the surrounding context.

Retrieval often needs one vector for a whole sentence or passage. Averaging BERT’s token states produces such a vector, but BERT’s pretraining objective does not directly train the resulting sentence similarities for retrieval.

3. Sentence-BERT (2019)

Sentence-BERT uses a siamese structure: sentences pass separately through an encoder with shared weights, then a pooling step yields sentence vectors. Reimers and Gurevych study classification, regression and triplet objectives, including training on natural-language-inference data. The goal is a representation that can be compared without jointly encoding every possible sentence pair.

Encode the documents once; encode each query when it arrives. Fine-tuning determines whether the resulting geometry serves the task.

4. Contrastive retrieval models (DPR, E5, GTE, Nomic — 2020+)

DPR and E5 put query–passage contrastive training at the center of retrieval. A positive passage competes with negative candidates. The data can come from human relevance labels, mined text pairs or other supervision; a large labeled similarity dataset is not the only route.

DPR reported improvements over BM25 on several open-domain question-answering datasets. E5 studies large-scale weakly supervised text pairs followed by supervised fine-tuning. Those results are evidence for particular training and evaluation setups, not a guarantee that dense retrieval beats lexical search on a new corpus.

Embedding models now include both encoder and decoder-based architectures. Hosted APIs do not necessarily disclose their complete training recipe. Follow the model’s documented pooling, prefixes, normalization and input limit rather than assuming all embedding endpoints work alike.

The minimal math: InfoNCE

A contrastive training example contains:

• A query $q$ (a sentence).
• A positive $k^+$ (a sentence that is "relevant" to $q$ — a paraphrase, an answer, a paired passage).
• A set of negatives $k_1^-, k_2^-, \ldots$ (random unrelated sentences, usually other things in the same batch).

You pass each through the encoder, get embeddings $\mathbf{v}_q$, $\mathbf{v}^+$, $\mathbf{v}_1^-$, etc. Then the loss for this one query is:

This is called the InfoNCE loss, a softmax cross-entropy over "which of these candidates is the right one." The "correct answer" is always the positive. Everything else is a distractor.

The temperature $\tau$ sharpens or softens the distribution. It is a training hyperparameter, not a universal constant: E5-base-v2 documents 0.01.

With in-batch negatives, the other examples in a batch supply competing candidates. Larger batches offer more negatives, but can also contain false negatives: passages relevant to more than one query.

One forward pass, one vector

At inference, run the encoder and apply its documented pooling procedure. Some models use a designated token; others use masked mean pooling or the last token. Training has made that particular representation useful for comparison.

Instruction prefixes. E5-base-v2 expects query: and passage: for asymmetric query-to-passage retrieval. Its model card recommends query: on both inputs for symmetric similarity tasks. Omitting the required wrapper changes the input distribution.

Normalization. If both nonzero vectors have unit length, their dot product equals cosine and their squared Euclidean distance is $2-2\cos(\theta)$. These metrics then produce the same ordering. A model trained for unnormalized inner product may rely on magnitude, so check before normalizing.

Embedding dimension. More dimensions cost more storage and comparison work; they do not automatically improve retrieval. One million 768-dimensional float32 vectors require 3.072 GB of raw payload. At 4,096 dimensions that becomes 16.384 GB. Index links, metadata and allocator overhead come on top.

Embedding dimension and Matryoshka training

Changing encoders usually means re-embedding the corpus. A model trained to support several output dimensions offers another option: keep the encoder and shorten its vectors.

Matryoshka Representation Learning applies training losses to selected nested prefixes of a representation. A 768-dimensional vector might also be trained at 64, 128, 256 and 512 dimensions. The shorter prefixes are explicitly optimized rather than treated as arbitrary slices.

At a supported prefix length, truncate and normalize again for cosine search. This is lossy: quality can change, and the acceptable dimension depends on your task. Arbitrarily truncating an ordinary embedding model does not acquire Matryoshka’s training objective.

OpenAI’s text-embedding-3 models expose a dimensions parameter for shortening output vectors.

Worked example

Consider five sentences with two intended topic groups. The matrix below uses authored similarities to make those groups visible. The code afterward runs E5 and computes a separate matrix from its actual outputs.

The five sentences:

1. "How do I return a broken laptop?"
2. "What is your refund policy?"
3. "My package arrived damaged."
4. "Cats purr when they are content."
5. "The kitten curled up on the couch."

The first three concern related customer-support issues: returns, refunds and damaged deliveries. The last two concern cats. They share little vocabulary within each group, so term overlap alone gives a weak signal.

The visual uses deliberately separated scores so its block structure is easy to read. E5-base-v2’s model card notes that its cosine scores often lie around 0.7–1.0. Absolute thresholds therefore need calibration for the model and task.

Implementation sketch

The following code uses sentence-transformers to run the actual model.

# pip install sentence-transformers torch
 
from sentence_transformers import SentenceTransformer
import torch
 
# E5-base-v2 is a solid, small, open-weights retrieval model.
# 768-dim, ~110M params, trained with contrastive weak supervision.
model = SentenceTransformer("intfloat/e5-base-v2")
 
# For this symmetric sentence-similarity example, E5 recommends query: on both sides.
sentences = [
    "query: How do I return a broken laptop?",
    "query: What is your refund policy?",
    "query: My package arrived damaged.",
    "query: Cats purr when they are content.",
    "query: The kitten curled up on the couch.",
]
 
# Encode returns a (5, 768) tensor of embeddings. normalize_embeddings=True
# divides each row by its L2 norm, so dot products become cosine sims.
emb = model.encode(sentences, normalize_embeddings=True, convert_to_tensor=True)
print(emb.shape)   # torch.Size([5, 768])
 
# Unit-normalized rows make a dot product equal cosine similarity.
sim = emb @ emb.T
print(sim.round(decimals=2))
 
# E5-base-v2 is not documented as Matryoshka-trained.
# Do not assume that slicing it to 256 dimensions preserves retrieval quality.

The code loads a model, encodes five sentences and multiplies the matrix by its transpose. It downloads model weights; the output depends on the model and library versions.

Misconceptions

"Bigger embedding dimension is always better." Storage grows linearly with dimension. Retrieval quality does not. Compare models and supported dimensions on held-out queries from your corpus.

"If I swap embedding models, I just need to re-embed my corpus." Queries and documents must use a compatible encoder configuration. Some systems share one model; others train distinct query and passage encoders into a common space. Upgrading one side independently can break that compatibility, even if the vector dimensions still match. Re-index and test the matched configuration together.

"Cosine similarity is a real measure of semantic similarity." It measures alignment in the chosen embedding space. Whether that alignment predicts relevance depends on the encoder, domain and task. An unadapted BERT pooling baseline may rank differently from a retrieval-trained encoder, but neither the architecture nor a plausible cosine score establishes quality. Check rankings against relevance labels.

"One embedding model is enough." It can be. If evaluation reveals misses on exact identifiers or on subtle query–passage relevance, lexical retrieval or a reranker may help. Additional stages earn their place through measured improvements.

What's next

The next post turns from producing vectors to searching them. Exact search is a useful baseline; larger workloads may justify an approximate index.

Additional reading (and watching)

• Devlin, J., et al. (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL 2019. A foundation for bidirectional encoder models, including many retrieval encoders.

• Reimers, N. & Gurevych, I. (2019). Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. EMNLP 2019. The paper that turned BERT into a usable sentence encoder via siamese fine-tuning on NLI and STS data.

• Oord, A. v. d., Li, Y., & Vinyals, O. (2018). Representation Learning with Contrastive Predictive Coding. arXiv:1807.03748. Introduces the InfoNCE objective used in many contrastive representation models.

• Karpukhin, V., et al. (2020). Dense Passage Retrieval for Open-Domain Question Answering. EMNLP 2020. DPR: a contrastive dense retriever evaluated against BM25 on open-domain QA.

• Wang, L., et al. (2022). Text Embeddings by Weakly-Supervised Contrastive Pre-training. arXiv:2212.03533. The E5 paper. Web-scale weakly-supervised pairs, instruction prefixes, and the recipe a lot of 2023+ open embedding models imitate.

• Chen, T., et al. (2020). A Simple Framework for Contrastive Learning of Visual Representations. ICML 2020. SimCLR. Describes contrastive training with in-batch negatives for visual representations.

• Gao, T., Yao, X., & Chen, D. (2021). SimCSE: Simple Contrastive Learning of Sentence Embeddings. EMNLP 2021. Showed that contrastive sentence embeddings work even with trivially-constructed positives (dropout masks as augmentation), and made clear how much batch size matters for the loss.

• Kusupati, A., et al. (2022). Matryoshka Representation Learning. NeurIPS 2022. The MRL paper. Trains nested prefixes jointly so one vector supports many dimensions at inference.

• OpenAI. New embedding models and API updates. Documents shortening text-embedding-3 outputs with the dimensions parameter.

• Ethayarajh, K. (2019). How Contextual are Contextualized Word Representations?. Studies contextual word representations and anisotropy; it does not establish a universal cosine threshold for retrieval.

• Lin, J., Ma, X., et al. (2021). A Replication Study of Dense Passage Retrieval. arXiv:2104.05740. Careful empirical study that establishes when DPR beats BM25 and when it doesn't, grounding the dense-sparse hybrid intuition.

• Muennighoff, N., et al. (2023). MTEB. A multi-task embedding evaluation suite.

• E5-base-v2 model card. Prefixes, masked mean pooling, normalization, 512-token limit and score-range guidance.