Norms, Dot Products, and Similarity

In the last post we talked about vectors: ordered lists of numbers that represent things. A word embedding is a vector. An activation inside a neural network is a vector. A row in a weight matrix is a vector. Great. We have all these vectors floating around in high-dimensional space.

But now what? Having vectors isn't useful on its own. You need to do things with them. Specifically, you need to answer two questions that come up constantly in machine learning:

1. How big is this vector? (Its magnitude, or norm.)
2. How similar are these two vectors? (Some kind of comparison.)

That's what this post is about. We'll build up from the L2 norm to the dot product to cosine similarity. The dot product is the operation. Attention runs on it, embeddings are compared with it, and every linear layer is a batch of them. If there's one mathematical idea from this series worth internalizing, this is it.

How big is a vector?

Let's start simple. You have a vector $\mathbf{v} = [3, 4]$. How "big" is it?

If you think back to middle school geometry, this is just the Pythagorean theorem. The vector $[3, 4]$ is a point in 2D space, and its distance from the origin is $\sqrt{3^2 + 4^2} = \sqrt{9 + 16} = \sqrt{25} = 5$.

That distance-from-the-origin measurement is called the L2 norm (also called the Euclidean norm, also written $\|\mathbf{v}\|$). The general formula for any number of dimensions is:

So for a 768-dimensional word embedding, the L2 norm is just "square every component, add them up, take the square root." It tells you the magnitude of the vector. How far it is from the origin. How "big" the signal is.

Why does this matter? When you're looking at activations inside a neural network, the norm tells you about the scale of the signal flowing through the network. Layer normalization, which shows up everywhere in transformers, is related to norms and scale control, but it isn't literally "make this vector have norm 1." It subtracts the feature mean and divides by the feature standard deviation across the feature dimensions of a single example, then applies a learned scale and bias. The effect is that each normalized feature vector has controlled mean and variance before the learned affine step, which indirectly controls scale. Without some way to talk about the size of vectors, you can't reason about any of this. We'll get into LayerNorm properly in a later post.

One special case: a unit vector is any vector with norm equal to 1. You can turn any vector into a unit vector by dividing it by its norm:

This operation is called normalization, and it shows up again in about thirty seconds when we get to cosine similarity.

Other ways to measure "big"

A quick aside, because the word "norm" shows up in contexts where it does not mean the L2 version. A norm is any function that measures "how big" a vector is while obeying a short list of rules (positive, scales cleanly, triangle inequality). The L2 norm is one specific choice. Two others show up often enough that they're worth having a picture for.

The L1 norm is the sum of absolute values, $\|\mathbf{v}\|_1 = \sum_i |v_i|$. Think of it as taxicab distance: you can only move along the axes, so the length of the trip is the total number of blocks. L1 is what regularizers like Lasso use, and it's what encourages sparsity in the weights. Under the right optimizer and objective it can drive many of them all the way to zero.

The L∞ norm is the largest absolute component, $\|\mathbf{v}\|_\infty = \max_i |v_i|$. It tells you about the biggest single spike in the vector, regardless of the others. Useful for bounding worst-case behavior, and it's exactly what gradient clipping on a per-element basis is doing.

The cleanest way to see the difference is to look at the unit ball for each norm: the set of all vectors whose norm is at most 1. Its boundary, where the norm equals exactly 1, is the unit sphere. Each norm gives that ball a different shape, and the shape tells you everything about what that norm rewards and punishes.

In deep learning you'll use the L2 norm most of the time: attention scores, embedding distances, layer-norm statistics, gradient magnitudes. But when you see a paper talking about "L1 regularization" or a framework clipping gradients "by value" (L∞) versus "by norm" (L2), the difference is exactly which of these pictures applies.

The dot product

Okay, so we can measure how big a single vector is. But the more interesting question is: given two vectors, how related are they? Are they pointing in roughly the same direction? Are they perpendicular? Are they pointing in opposite directions?

The dot product answers this. Given two vectors $\mathbf{a}$ and $\mathbf{b}$, their dot product is:

Multiply corresponding components, add them up. Let me do one by hand so we can see what's happening.

Take $\mathbf{a} = [3, 4]$ and $\mathbf{b} = [4, 1]$:

The result is a single number: 16. Not a vector, just a scalar. But that scalar contains a surprising amount of information, because of this identity:

where $\theta$ is the angle between the two vectors.

This is the geometric interpretation. The dot product is telling you three things at once: how big $\mathbf{a}$ is, how big $\mathbf{b}$ is, and how much they're pointing in the same direction ($\cos\theta$). Let me unpack that.

When two vectors point in exactly the same direction, $\theta = 0°$ and $\cos\theta = 1$. For fixed vector lengths, the dot product is at its maximum: $\|\mathbf{a}\| \|\mathbf{b}\|$. (If you let the lengths grow, of course the dot product can grow without bound; the maximum is over directions, not magnitudes.) When they're perpendicular, $\theta = 90°$ and $\cos\theta = 0$. The dot product is zero. When they point in opposite directions, $\theta = 180°$ and $\cos\theta = -1$. The dot product is negative.

So the sign and magnitude of the dot product give you a quick read on alignment:

• Positive: vectors generally point the same way
• Zero: vectors are orthogonal, meaning this dot-product comparison sees no linear alignment between them. (That doesn't necessarily mean they're "unrelated" in some semantic sense; it just means they don't project onto each other under this particular inner product.)
• Negative: vectors point in opposite directions

Here's that identity made physical. Drag either vector and watch both sides of the equation compute themselves in real time. The coordinate formula on the left only knows about $a_1 b_1 + a_2 b_2$. The geometry formula on the right only knows about magnitudes and angle. They always land on the same number.

Starting position is our running example: $\mathbf{a} = [3, 4]$, $\mathbf{b} = [4, 1]$. The dot product is 16, which is positive but less than the maximum possible value of $\|\mathbf{a}\| \|\mathbf{b}\| = 5 \times \sqrt{17} \approx 20.6$. Drag $\mathbf{a}$ until it points roughly away from $\mathbf{b}$ and watch the number flip to negative.

To really internalize the sign-and-magnitude story, play with the interactive below. You're rotating one unit vector around another and watching the dot product swing from $+1$ (aligned) to $0$ (perpendicular) to $-1$ (opposite) and back. The bar at the bottom is the same quantity that ends up inside every attention score.

Projection: "how much of a points in b's direction?"

There's another way to think about the dot product that becomes critical in attention mechanisms. You can ask: if I shine a flashlight straight down onto the line defined by $\mathbf{b}$, where does the shadow of $\mathbf{a}$ fall?

That shadow is called the projection of $\mathbf{a}$ onto $\mathbf{b}$. There are actually two different objects here, and it helps to keep the names straight. The signed length of the shadow (the scalar projection) is:

And the actual projection vector (the shadow itself, lying along $\mathbf{b}$) is:

Let's compute it for our vectors. We already know $\mathbf{a} \cdot \mathbf{b} = 16$ and $\mathbf{b} \cdot \mathbf{b} = 4^2 + 1^2 = 17$. So $\text{comp}_{\mathbf{b}}(\mathbf{a}) = 16/\sqrt{17} \approx 3.88$, and $\text{proj}_{\mathbf{b}}(\mathbf{a}) = (16/17)\,[4, 1] \approx [3.76, 0.94]$.

Why does this matter for neural networks? In attention, when a query vector $\mathbf{q}$ is dotted with a key vector $\mathbf{k}$, it's essentially asking: "how much does this query align with this key?" The larger the dot product, the more the query is pointing in the direction of the key, and the more attention that token receives. Projection is the geometric backbone of that computation.

Cosine similarity: stripping out magnitude

Here's a problem with using raw dot products to measure similarity. Consider these two vectors:

• $\mathbf{a} = [1, 0]$
• $\mathbf{b} = [1000, 0]$

Their dot product is $1 \times 1000 + 0 \times 0 = 1000$. And the dot product of $\mathbf{a}$ with $\mathbf{c} = [0.5, 0.866]$ (which points 60 degrees away) is $1 \times 0.5 + 0 \times 0.866 = 0.5$.

So the dot product says $\mathbf{a}$ is way more "similar" to $\mathbf{b}$ (score of 1000) than to $\mathbf{c}$ (score of 0.5). But $\mathbf{a}$ and $\mathbf{b}$ point in exactly the same direction, while $\mathbf{a}$ and $\mathbf{c}$ point in different directions. The dot product is correct in the sense that $\mathbf{b}$ really is more aligned, but the scores are being dominated by magnitude, not direction.

In many applications, especially embedding search, you care about direction, not magnitude. Two word embeddings should be "similar" if they point the same way in the learned space, regardless of how long the vectors happen to be.

The picture below makes the disagreement concrete. Same query, same three candidates, two metrics. The "long" candidate is mostly off-axis but enormous; the "aligned" one is short but pointing right at the query. Watch the rankings shuffle when the metric flips.

Cosine similarity fixes this by normalizing:

You take the dot product and divide by the product of the norms. This gives you a number between $-1$ and $1$:

• $1$ means identical direction
• $0$ means perpendicular
• $-1$ means opposite direction

For our example: $\cos(\mathbf{a}, \mathbf{b}) = 1000 / (1 \times 1000) = 1.0$ (identical direction, as expected). And $\cos(\mathbf{a}, \mathbf{c}) = 0.5 / (1 \times 1) = 0.5$ (60 degrees apart).

There's an equivalent way to think about this: cosine similarity is just the dot product of the unit vectors. Normalize both vectors first, then dot them:

Same result. When you see a system that "normalizes embeddings before comparing," this is exactly what's happening. Normalize to unit vectors, then dot product. That's cosine similarity.

One small caveat: cosine similarity is undefined when either vector has zero norm, since you'd be dividing by zero. In practice, libraries either skip zero vectors or add a small epsilon to the denominator. Not usually a problem with trained embeddings, but it does come up.

Cosine is a good default when vector magnitude is accidental or doesn't carry signal you care about. But it's not always the right metric. If the embedding model was trained so that length encodes something meaningful (popularity, frequency, confidence), then raw dot product is the metric you actually want, because cosine throws that information away. The right move is to check how the embedding model was trained and what its docs recommend. OpenAI's embeddings, for example, are returned pre-normalized to unit length, so dot product, cosine similarity, and Euclidean ranking all agree on the ordering for those vectors.

A worked example

Let me walk through all three operations on our running example with concrete numbers.

Vectors: $\mathbf{a} = [3, 4]$ and $\mathbf{b} = [4, 1]$

Norms: $\|\mathbf{a}\| = \sqrt{9 + 16} = \sqrt{25} = 5$ $\|\mathbf{b}\| = \sqrt{16 + 1} = \sqrt{17} \approx 4.12$

Dot product: $\mathbf{a} \cdot \mathbf{b} = (3)(4) + (4)(1) = 16$

Cosine similarity: $\cos\theta = \frac{16}{5 \times 4.12} \approx \frac{16}{20.6} \approx 0.776$

Angle: $\theta = \arccos(0.776) \approx 39°$

Projection of a onto b: $\text{comp}_{\mathbf{b}}(\mathbf{a}) = \frac{16}{\sqrt{17}} \approx 3.88$ (the signed length of the shadow) $\text{proj}_{\mathbf{b}}(\mathbf{a}) = \frac{16}{17}\,[4, 1] \approx [3.76, 0.94]$ (the shadow as a vector)

So these two vectors are about 39 degrees apart. They're reasonably aligned (cosine similarity of 0.776 out of a maximum of 1.0), and most of $\mathbf{a}$'s magnitude projects onto $\mathbf{b}$'s direction.

Now imagine doing this in 768 dimensions instead of 2. The formulas are identical. You're still just multiplying components and summing. The numbers are bigger, but the geometry works the same way.

Where this lives in the stack

So we have these three tools: norms, dot products, cosine similarity. Where do they actually show up in an LLM? Pretty much everywhere, and honestly the list below is a partial one. Let me trace through the major ones.

Attention. Attention starts by computing dot products between query and key vectors. In transformers it's scaled dot-product attention: those raw scores get divided by $\sqrt{d_k}$, optionally masked, and then passed through softmax to become weights over the value vectors. So the dot product isn't the whole attention mechanism, but it's the primitive compatibility score that everything else is built on. The machinery around it gets its own post later in the series.

Here's a feel for what ranking by dot product gets you. Below is a tiny vocabulary where each word is a hand-tuned vector. We pick one word as the query, compute its similarity with every other word, and sort by the score. The closest match pops to the top. Swap the query and watch the ranking rearrange. This is the same shape of operation that lets an attention head decide which earlier tokens matter, and that lets a vector database surface the closest documents to your query.

Embedding retrieval. When you search a vector database, you encode the query as an embedding and compare it against the stored embeddings using cosine similarity, normalized dot product, raw inner product, or L2 distance, depending on how the embedding model and index are configured. At small scale you can brute-force the comparison against every vector. At large scale, vector databases like FAISS use indexing structures and approximate nearest neighbor algorithms (IVF, HNSW, product quantization) to avoid scanning the whole corpus on every query. The metric is still a dot product or a distance; the work is in not computing all of them.

Linear layers. Every nn.Linear(in, out) in PyTorch computes $\mathbf{x} W^T + \mathbf{b}$. Each output neuron is the dot product of the input with one row of the weight matrix. A linear layer with 768 inputs and 3072 outputs computes 3072 dot products in parallel.

Implementation

In code, all of this is one-liners.

import numpy as np
import torch
import torch.nn.functional as F
 
a = np.array([3.0, 4.0])
b = np.array([4.0, 1.0])
 
# Norms
np.linalg.norm(a)                        # 5.0
torch.norm(torch.tensor(a))              # 5.0
 
# Dot product
np.dot(a, b)                             # 16.0
torch.dot(torch.tensor(a), torch.tensor(b))  # 16.0
 
# Cosine similarity
cos_sim = np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))
# 0.776...
 
# PyTorch has a built-in (expects batched inputs)
F.cosine_similarity(
    torch.tensor(a).unsqueeze(0),
    torch.tensor(b).unsqueeze(0)
)  # tensor([0.7761])

The thing to notice is that these are all extremely fast operations individually. A dot product on two 768-dimensional vectors is 768 multiplications and 767 additions, and on a GPU one of those in isolation is essentially nothing. The catch is that attention computes one dot product for every query–key pair, so the score matrix grows quadratically with sequence length. At long context, attention is one of the central scaling costs, not a free operation. Transformers stay practical because all those dot products get batched into highly optimized matrix multiplications that GPUs are absurdly good at, not because the total amount of work is small. Later posts on FlashAttention, KV caches, and serving systems are all about managing exactly this cost.

Common misconceptions

"Cosine similarity measures semantic meaning." Not exactly. Cosine similarity measures the geometric angle between two vectors. Whether that angle corresponds to semantic meaning depends entirely on how the embedding space was trained. If the training objective pushed similar words to have similar vectors (like Word2Vec or a sentence transformer did), then yes, a high cosine similarity correlates with semantic relatedness. But cosine similarity itself has no idea what the vectors mean. You could compute cosine similarity between two random vectors and get a number. It just wouldn't mean anything semantically. The geometry is meaningful only because the training process made it so.

"A bigger dot product always means more similar." Only if the vectors have the same magnitude. The dot product conflates magnitude and direction. Vector $[1000, 0]$ has a huge dot product with almost anything, not because it's similar to everything, but because it's long. If you want pure directional similarity, you need cosine similarity (which normalizes out the magnitude). This is why most embedding search systems normalize their vectors before storing them.

"Euclidean distance and cosine similarity always agree." They don't, at least not for unnormalized vectors. Two vectors can be close in Euclidean distance but point in very different directions (imagine two short vectors near the origin, pointing away from each other). And two vectors can have high cosine similarity but be far apart in Euclidean space (imagine two very long vectors pointing the same way but with different magnitudes). However, if you normalize all your vectors to unit length first, then Euclidean distance and cosine similarity become monotonically related. This is why many vector databases normalize embeddings at index time: it lets you use either metric interchangeably.

What's next

We've got vectors, and now we know how to measure and compare them. The next post covers probability distributions, the mathematical foundations of how neural networks express uncertainty, which sets up cross-entropy, softmax, and everything that lets a model say "I'm 87% sure the next token is cat."

Additional reading (and watching)

• Vaswani, A., et al. (2017). Attention Is All You Need. NeurIPS 2017. The paper that introduced scaled dot-product attention as $\mathrm{softmax}(QK^T/\sqrt{d_k})V$, which is why every attention score is a dot product.
• Alammar, J. (2019). The Illustrated Word2vec. A visual walkthrough of how embedding spaces learn directional meaning through training.
• Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning, Chapter 2: Linear Algebra. MIT Press. Covers the relationship between norms, distances, and inner products at the level needed for deep learning.
• Kusupati, A., et al. (2022). Matryoshka Representation Learning. NeurIPS 2022. Introduced nested embeddings that let you truncate the vector to any prefix and still get useful similarities.
• Johnson, J., Douze, M., & Jégou, H. (2017). Billion-scale similarity search with GPUs. IEEE Transactions on Big Data. The FAISS paper; made approximate nearest-neighbor search practical at web scale.