Norms, Dot Products, and Similarity

In the last post we talked about vectors: ordered lists of numbers that represent things. A word embedding is a vector. An activation inside a neural network is a vector. A row in a weight matrix is a vector.

To work with those vectors, we need ways to measure and compare them:

1. How big is this vector? (Its magnitude, or norm.)
2. How similar are these two vectors? (Some kind of comparison.)

We'll build up from the L2 norm to the dot product to cosine similarity. These operations let us measure activation scale, compute attention scores, and compare embeddings.

How big is a vector?

Let's start simple. You have a vector $\mathbf{v} = [3, 4]$. How "big" is it?

The Pythagorean theorem gives the distance. The vector $[3, 4]$ is a point in 2D space, and its distance from the origin is $\sqrt{3^2 + 4^2} = \sqrt{9 + 16} = \sqrt{25} = 5$.

That distance-from-the-origin measurement is called the L2 norm (also called the Euclidean norm, also written $\|\mathbf{v}\|$). The general formula for any number of dimensions is:

So for a 768-dimensional word embedding, the L2 norm is just "square every component, add them up, take the square root." That is its distance from the origin.

For activations, the norm measures signal scale. Dividing by that norm is different from layer normalization, which centers and rescales features before a learned affine step.

One special case: a unit vector is any vector with norm equal to 1. You can turn any nonzero vector into a unit vector by dividing it by its norm:

Dividing a nonzero vector by its norm preserves its direction and sets its length to one.

Other ways to measure "big"

A norm is any function that measures "how big" a vector is while obeying a short list of rules (positive, scales cleanly, triangle inequality). The L2 norm is one specific choice. Two other common choices are L1 and L∞.

The L1 norm is the sum of absolute values, $\|\mathbf{v}\|_1 = \sum_i |v_i|$. Think of it as taxicab distance: you can only move along the axes, so the length of the trip is the total number of blocks. L1 is what regularizers like Lasso use, and it's what encourages sparsity in the weights. Under the right optimizer and objective it can drive many of them all the way to zero.

The L∞ norm is the largest absolute component, $\|\mathbf{v}\|_\infty = \max_i |v_i|$. It tells you about the biggest single spike in the vector, regardless of the others. Useful for bounding worst-case behavior, and it's exactly what gradient clipping on a per-element basis is doing.

The cleanest way to see the difference is to look at the unit ball for each norm: the set of all vectors whose norm is at most 1. Its boundary, where the norm equals exactly 1, is the unit sphere. Each norm gives that ball a different shape, and the shape tells you everything about what that norm rewards and punishes.

In deep learning you'll use the L2 norm most of the time: attention scores, embedding distances, layer-norm statistics, gradient magnitudes. But when you see a paper talking about "L1 regularization" or a framework clipping gradients "by value" (L∞) versus "by norm" (L2), the difference is exactly which of these pictures applies.

The dot product

Okay, so we can measure how big a single vector is. But the more interesting question is: given two vectors, how related are they? Are they pointing in roughly the same direction? Are they perpendicular? Are they pointing in opposite directions?

The dot product answers this. Given two vectors $\mathbf{a}$ and $\mathbf{b}$, their dot product is:

Multiply corresponding components, then add the products.

Take $\mathbf{a} = [3, 4]$ and $\mathbf{b} = [4, 1]$:

The result is a scalar, 16. We can also compute it from the vectors' lengths and the angle between them:

where $\theta$ is the angle between the two vectors.

The dot product is telling you three things at once: how big $\mathbf{a}$ is, how big $\mathbf{b}$ is, and how much they're pointing in the same direction ($\cos\theta$).

When two vectors point in exactly the same direction, $\theta = 0°$ and $\cos\theta = 1$. For fixed vector lengths, the dot product is at its maximum: $\|\mathbf{a}\| \|\mathbf{b}\|$. When they're perpendicular, $\theta = 90°$ and $\cos\theta = 0$. The dot product is zero. When they point in opposite directions, $\theta = 180°$ and $\cos\theta = -1$. The dot product is negative.

So the sign and magnitude of the dot product give you a quick read on alignment:

• Positive: the angle is less than 90°
• Zero: vectors are orthogonal, meaning this dot-product comparison sees no linear alignment between them.
• Negative: the angle between nonzero vectors is greater than 90°

Drag either vector and watch both sides of the equation compute themselves in real time. The coordinate formula on the left only knows about $a_1 b_1 + a_2 b_2$. The geometry formula on the right only knows about magnitudes and angle. They always land on the same number.

Starting position is our running example: $\mathbf{a} = [3, 4]$, $\mathbf{b} = [4, 1]$. The dot product is 16, which is positive but less than the maximum possible value of $\|\mathbf{a}\| \|\mathbf{b}\| = 5 \times \sqrt{17} \approx 20.6$. Drag $\mathbf{a}$ until it points roughly away from $\mathbf{b}$ and watch the number flip to negative.

Rotating one unit vector around another makes the dot product vary from $+1$ (aligned) through $0$ (perpendicular) to $-1$ (opposite).

Projection: "how much of a points in b's direction?"

There's another way to think about the dot product that becomes critical in attention mechanisms. You can ask: if I shine a flashlight straight down onto the line defined by $\mathbf{b}$, where does the shadow of $\mathbf{a}$ fall?

That shadow is called the projection of $\mathbf{a}$ onto $\mathbf{b}$. There are actually two different objects here, and it helps to keep the names straight. For nonzero $\mathbf{b}$, the signed length of the shadow (the scalar projection) is:

And the actual projection vector (the shadow itself, lying along $\mathbf{b}$) is:

Let's compute it for our vectors. We already know $\mathbf{a} \cdot \mathbf{b} = 16$ and $\mathbf{b} \cdot \mathbf{b} = 4^2 + 1^2 = 17$. So $\text{comp}_{\mathbf{b}}(\mathbf{a}) = 16/\sqrt{17} \approx 3.88$, and $\text{proj}_{\mathbf{b}}(\mathbf{a}) = (16/17)\,[4, 1] \approx [3.76, 0.94]$.

In attention, when a query vector $\mathbf{q}$ is dotted with a key vector $\mathbf{k}$, it's essentially asking: "how much does this query align with this key?" The score depends on both angle and length. Holding other scores and masks fixed, increasing it raises that token's softmax attention weight.

Cosine similarity: stripping out magnitude

Here's a problem with using raw dot products to measure similarity. Consider these two vectors:

• $\mathbf{a} = [1, 0]$
• $\mathbf{b} = [1000, 0]$

Their dot product is $1 \times 1000 + 0 \times 0 = 1000$. And the dot product of $\mathbf{a}$ with $\mathbf{c} = [0.5, 0.866]$ (which points 60 degrees away) is $1 \times 0.5 + 0 \times 0.866 = 0.5$.

So the dot product says $\mathbf{a}$ is way more "similar" to $\mathbf{b}$ (score of 1000) than to $\mathbf{c}$ (score of 0.5). But $\mathbf{a}$ and $\mathbf{b}$ point in exactly the same direction, while $\mathbf{a}$ and $\mathbf{c}$ point in different directions. The dot product is correct in the sense that $\mathbf{b}$ really is more aligned, but the scores are being dominated by magnitude, not direction.

For an embedding model trained to use directional similarity, normalizing removes length differences from the comparison.

The picture below makes the disagreement concrete. Same query, same three candidates, two metrics. The "long" candidate is 20° off-axis and longer; the "aligned" one is short but pointing right at the query. Watch the rankings shuffle when the metric flips.

Cosine similarity fixes this by normalizing:

You take the dot product and divide by the product of the norms. This gives you a number between $-1$ and $1$:

• $1$ means identical direction
• $0$ means perpendicular
• $-1$ means opposite direction

For our example: $\cos(\mathbf{a}, \mathbf{b}) = 1000 / (1 \times 1000) = 1.0$ (identical direction, as expected). And $\cos(\mathbf{a}, \mathbf{c}) = 0.5 / (1 \times 1) = 0.5$ (60 degrees apart).

There's an equivalent way to think about this: cosine similarity is just the dot product of the unit vectors. Normalize both vectors first, then dot them:

Cosine similarity is undefined for a zero vector. Libraries may skip such vectors or stabilize the denominator with epsilon; check that behavior when comparing results.

Cosine is a good default when vector magnitude is accidental or doesn't carry signal you care about. But it's not always the right metric. If the embedding model was trained so that length encodes something meaningful (popularity, frequency, confidence), then raw dot product is the metric you actually want, because cosine throws that information away. The right move is to check how the embedding model was trained and what its docs recommend. For unit-length embeddings, dot product, cosine similarity, and Euclidean nearest-neighbor ranking agree: $\|a-b\|^2 = 2-2(a\cdot b)$.

A worked example

Vectors: $\mathbf{a} = [3, 4]$ and $\mathbf{b} = [4, 1]$

Norms: $\|\mathbf{a}\| = \sqrt{9 + 16} = \sqrt{25} = 5$ $\|\mathbf{b}\| = \sqrt{16 + 1} = \sqrt{17} \approx 4.12$

Dot product: $\mathbf{a} \cdot \mathbf{b} = (3)(4) + (4)(1) = 16$

Cosine similarity: $\cos\theta = \frac{16}{5 \times 4.12} \approx \frac{16}{20.6} \approx 0.776$

Angle: $\theta = \arccos(0.776) \approx 39°$

Projection of a onto b: $\text{comp}_{\mathbf{b}}(\mathbf{a}) = \frac{16}{\sqrt{17}} \approx 3.88$ (the signed length of the shadow) $\text{proj}_{\mathbf{b}}(\mathbf{a}) = \frac{16}{17}\,[4, 1] \approx [3.76, 0.94]$ (the shadow as a vector)

So these two vectors are about 39 degrees apart. They're reasonably aligned (cosine similarity of 0.776 out of a maximum of 1.0), and most of $\mathbf{a}$'s magnitude projects onto $\mathbf{b}$'s direction.

Now imagine doing this in 768 dimensions instead of 2. The formulas are identical. You're still just multiplying components and summing. There are more terms in each sum, but the same identities apply.

Where this lives in the stack

Norms, dot products, and cosine similarity appear in several parts of an LLM:

Attention. Attention starts by computing dot products between query and key vectors. In transformers it's scaled dot-product attention: those raw scores get divided by $\sqrt{d_k}$, optionally masked, and then passed through softmax to become weights over the value vectors. So the dot product isn't the whole attention mechanism, but it's the primitive compatibility score that everything else is built on. The machinery around it gets its own post later in the series.

The hand-tuned vectors below let you rank a tiny vocabulary by cosine similarity. Change the query to see which neighbors move up or down.

Embedding retrieval. When you search a vector database, you encode the query as an embedding and compare it against the stored embeddings using cosine similarity, normalized dot product, raw inner product, or L2 distance, depending on how the embedding model and index are configured. At small scale you can brute-force the comparison against every vector. At large scale, libraries such as FAISS use indexing structures and approximate nearest neighbor algorithms (IVF, HNSW, product quantization) to avoid scanning the whole corpus on every query. The metric is still a dot product or a distance; the work is in not computing all of them.

Linear layers. Every nn.Linear(in, out) in PyTorch computes $\mathbf{x} W^T + \mathbf{b}$. Each output neuron is the dot product of the input with one row of the weight matrix. A linear layer with 768 inputs and 3072 outputs computes 3072 dot products in parallel.

Implementation

import numpy as np
import torch
import torch.nn.functional as F
 
a = np.array([3.0, 4.0])
b = np.array([4.0, 1.0])
 
# Norms
np.linalg.norm(a)                        # 5.0
torch.linalg.vector_norm(torch.tensor(a))              # 5.0
 
# Dot product
np.dot(a, b)                             # 16.0
torch.dot(torch.tensor(a), torch.tensor(b))  # 16.0
 
# Cosine similarity
cos_sim = np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))
# 0.776...
 
# Compare along the default feature dimension, dim=1
F.cosine_similarity(
    torch.tensor(a).unsqueeze(0),
    torch.tensor(b).unsqueeze(0)
)  # tensor([0.7761])

A dot product on two 768-dimensional vectors requires 768 multiplications and 767 additions. Attention computes one for every query–key pair, so the score matrix grows quadratically with sequence length. Batching these products into matrix multiplications uses GPUs efficiently, but the total work still grows. FlashAttention, KV caches, and serving systems address different parts of that cost.

Common misconceptions

"Cosine similarity measures semantic meaning." It measures the angle between vectors. Its usefulness for semantic comparison depends on the representation: a model trained to bring related texts together may produce useful cosine rankings, while arbitrary vectors need not.

"A bigger dot product always means more similar." For a fixed query, dot-product ranking agrees with directional similarity when the candidate vectors have equal nonzero norms. Otherwise length affects the score. For example, $[1000,0]$ has dot product 1000 with $[1,0]$, zero with $[0,1]$, and −1000 with $[-1,0]$. A longer vector amplifies the signed projection; it does not make every score positive or large.

"Euclidean distance and cosine similarity always agree." They don't, at least not for unnormalized vectors. Two vectors can be close in Euclidean distance but point in very different directions (imagine two short vectors near the origin, pointing away from each other). And two vectors can have high cosine similarity but be far apart in Euclidean space (imagine two very long vectors pointing the same way but with different magnitudes). However, if you normalize all your vectors to unit length first, then Euclidean distance and cosine similarity become monotonically related. This is why many vector databases normalize embeddings at index time: it lets you use either metric interchangeably.

What's next

The next post covers probability distributions: turning a vector of scores into next-token probabilities with softmax.

Additional reading (and watching)

• Vaswani, A., et al. (2017). Attention Is All You Need. NeurIPS 2017. The paper that introduced scaled dot-product attention as $\mathrm{softmax}(QK^T/\sqrt{d_k})V$, which is why every attention score is a dot product.
• Alammar, J. (2019). The Illustrated Word2vec. A visual walkthrough of how embedding spaces learn directional meaning through training.
• Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning, Chapter 2: Linear Algebra. MIT Press. Covers the relationship between norms, distances, and inner products at the level needed for deep learning.
• Kusupati, A., et al. (2022). Matryoshka Representation Learning. NeurIPS 2022. Introduced nested embeddings that let you train selected vector prefixes to remain useful at different embedding sizes.
• Johnson, J., Douze, M., & Jégou, H. (2017). Billion-scale similarity search with GPUs. IEEE Transactions on Big Data. The FAISS paper; made approximate nearest-neighbor search practical at web scale.