Vectors, Matrices, and the Spaces They Live In

This is the first post in a series called Holding the LLM Stack in Your Head. The goal is to build up the mental models for the entire modern LLM stack, from the ground floor to agent protocols, in dependency order. Nothing gets used before the intuition for it has been earned. If you want to know what the series is about and why I'm writing it, there's a longer explanation on the about page. Otherwise, let's just start.

And we're starting with linear algebra. Almost everything that happens inside a neural network can be expressed as operations on vectors and matrices. Building an intuition for those operations, not just knowing the rules but actually feeling what a matrix multiply does to a vector, turns out to be a prerequisite for every layer of understanding above this one.

Personally, I really benefited from revisiting this topic. As basic as it sounds, when I sat down to work through what these operations mean geometrically, I realized my intuition was cloudier than I expected. If you were introduced to this math in college and haven't actively engaged with the geometric interpretations since (which I'm guessing describes a lot of the people reading this), there's a good chance yours has faded too. So let's rebuild it from the bottom.

What is a "space"?

Before we get to vectors and matrices, I want to define a word that comes up constantly in machine learning and almost never gets explained: space.

You'll hear people say "embedding space," "latent space," "the model projects into a higher-dimensional space." But what does "space" actually mean here?

In mathematics, a space is a set of objects together with rules about how those objects can combine. The set tells you what lives in the space. The rules tell you what you're allowed to do with those objects and guarantee that the results stay in the space.

A familiar example: the 2D plane. The objects are points (or equivalently, pairs of numbers like $(3, -1)$). The rules say you can add two points and get another point in the plane. You can scale a point by a number and get another point in the plane. Those two operations, addition and scalar multiplication, never throw you out of the plane. That closure property is what makes the plane a vector space in the mathematical sense.

Why does this matter? Because once you establish that a collection of objects forms a space with these rules, you inherit an entire toolkit of operations that are guaranteed to work: distance, similarity, projections, transformations. All of linear algebra is the study of what you can do inside spaces that have these properties. And neural networks live and breathe inside these spaces. Every layer takes vectors from one space and moves them to another.

Vectors

So what exactly is a vector?

A vector is an element of a vector space. That might sound circular, but it's the right starting point. A vector space is a set whose elements satisfy specific rules about addition and scaling (technically called the vector space axioms). The important ones for us:

1. You can add two vectors and get another vector in the same space.
2. You can multiply a vector by a scalar (a single number) and still get a vector in the same space.
3. There's a zero vector that acts as the identity for addition.

Why do we call these things "vectors" instead of just "arrays" or "lists of numbers"? Because the word "vector" carries meaning that "array" doesn't. When you call something a vector, you're saying it lives in a space with algebraic structure. It can be added, scaled, combined, and transformed in ways that are guaranteed to behave well. A plain array of bytes doesn't promise any of that. You can concatenate arrays, zip them, reverse them. But "add two arrays" and "scale an array" aren't operations with guaranteed mathematical properties unless you specifically define them to be. The word "vector" is a contract about what operations are legal and what promises those operations keep.

In practice, in deep learning, a vector is a list of numbers like $[0.5, -1.2, 3.0, 0.8]$. Four numbers, in a specific order, where each position encodes some dimension of meaning. When people say a language model has an "embedding dimension of 768," they mean every token in the vocabulary gets represented as a vector with 768 numbers. Each of those 768 slots encodes some learned aspect of the token's meaning. We don't get to pick what each slot means; the model figures that out during training. But the point is: a token's identity, as far as the network is concerned, is that vector.

In a neural network context, these vectors are sometimes called activations to distinguish them from the model's weights. Activations are the data flowing through the network; they change for every input. This is in contrast to parameters (or weights), which are the learned values that stay fixed during inference. The distinction between activations and parameters will matter throughout the series, so it's worth flagging early: vectors flowing through the network are activations, matrices defining the transformations are parameters.

In 2D we can draw vectors and see the geometry. In 768 dimensions, obviously we can't. But the algebra works exactly the same way. Every operation we define in 2D (addition, scaling, dot products, projections) generalizes to $\mathbb{R}^{768}$ without modification. That's the power of working within a vector space: the rules are dimension-agnostic.

Matrices

If a vector is a list of numbers, a matrix is a grid of numbers with rows and columns.

But here's what took me a while to really appreciate: a matrix isn't just a container for data. A matrix is a function. Specifically, it's a linear map, a function that takes a vector from one space and delivers it to another space (or the same space), while preserving the addition and scaling structure along the way.

Why were matrices invented? Precisely to represent these transformations compactly. Instead of describing a linear function as a long list of rules ("dimension 1 of the output equals 2 times dimension 1 of the input plus 3 times dimension 2 of the input, plus..."), you write the coefficients in a grid and multiply. The matrix is the transformation, encoded as a table of coefficients.

In a neural network, matrices show up as weights (or parameters). When people say a layer "has 3 million parameters," those parameters are the numbers sitting in the layer's weight matrices. The network learned those values during training, and they stay fixed at inference time. They sit there, defining a transformation, waiting to be applied to whatever input flows through.

Here's a small example. A weight matrix $W$ with 2 rows and 4 columns:

This matrix defines a function from 4-dimensional space to 2-dimensional space. Hand it a vector with 4 numbers and it gives you back a vector with 2 numbers. In a real network, this matrix might have millions or billions of entries, but the structure is the same: a grid of learned numbers that defines a transformation between spaces.

What makes this different from just a 2D array? The fact that a matrix comes with a defined operation: you can multiply it by a vector or another matrix. A 2D array is a data structure. A matrix is that data structure plus the algebraic rules for how it interacts with vectors and other matrices. Those rules (matrix-vector multiplication and matrix-matrix multiplication) are what make it a mathematical object rather than just a grid.

Tensors: the generalization

You'll hear the word "tensor" constantly in deep learning, mostly because PyTorch calls its basic data type a Tensor. Here's the hierarchy:

• A scalar is a single number. Zero dimensions.
• A vector is a list of numbers. One dimension. Shape: $[d]$.
• A matrix is a grid of numbers. Two dimensions. Shape: $[m, n]$.
• A tensor is the generalization to any number of dimensions. Shape: $[d_1, d_2, \ldots, d_n]$.

In practice, when someone in deep learning says "tensor," they usually just mean "a multi-dimensional array of numbers." A 3D tensor with shape $[B, T, d]$ represents a batch of $B$ sequences, each with $T$ tokens, each token encoded as a $d$-dimensional vector. That's the shape you'll encounter constantly in transformer code.

The mathematical definition of a tensor is richer than "multi-dimensional array." In mathematics and physics, a tensor isn't just a grid of numbers; it's a grid of numbers plus rules for how those numbers change when you switch coordinate systems. The idea is that the thing a tensor represents (a stress field in a material, a curvature of spacetime) is real and coordinate-independent, even though the numbers you write down depend on which axes you chose. That's a deep and beautiful idea, but it's not one we need right now. For building intuition about neural networks, the "multi-dimensional array" interpretation is the right one to carry around. We'll pick up the mathematical perspective later in the series if it becomes useful.

Understanding shapes turns out to be enormously practical. When something goes wrong in your model code, it's almost always a shape mismatch. And when you're reading a paper, the shapes tell you what's actually happening more clearly than the prose does. A few shapes to get comfortable with:

• $[d]$: a single vector. An embedding of one token.
• $[B, d]$: a batch of $B$ vectors. $B$ inputs, each with $d$ features.
• $[B, T, d]$: a batch of sequences. $B$ sequences, each $T$ tokens long, each token a $d$-dimensional vector. This is the bread-and-butter shape of transformer code.

Two kinds of multiplication

Before we get to the main event: there are two fundamentally different ways to "multiply" vectors and matrices, and they do completely different things.

Element-wise multiplication (also called the Hadamard product, written $\odot$) does exactly what it sounds like. Take two objects of the same shape and multiply corresponding entries:

Same shape in, same shape out. Each entry only interacts with its counterpart. This is the ordinary kind of multiplication that feels natural.

Matrix multiplication is completely different. It mixes information across dimensions. The shapes don't even need to match the same way; what matters is that the inner dimensions agree. A $[2, 4]$ matrix times a $[4, 3]$ matrix gives a $[2, 3]$ matrix. The output shape is assembled from the outer dimensions of the inputs.

Matrix multiplication is the operation that matters for neural networks. It's how a layer transforms its input, how attention computes relevance scores, how embeddings get projected into new spaces. Almost every interesting computation in a neural network is a matrix multiply or something built on top of one.

So how does it actually work?

Matrix multiplication: the operation that runs everything

The mechanical rule: each element of the output is a dot product between a row of the first matrix and a column of the second. A dot product means you multiply corresponding elements and add them up.

Say matrix $A$ is $2 \times 4$ and matrix $B$ is $4 \times 3$. The output $C$ will be $2 \times 3$. To compute the entry at row $i$, column $j$:

Take row $i$ from $A$, take column $j$ from $B$, multiply element-wise, sum. That's one dot product, and it produces one number in the output. Repeat for every $(i, j)$ pair.

Watch the animation step through each cell. Once you see the pattern (row from $A$, column from $B$, multiply and sum) you see it forever. You can pause and inspect any step.

But zoom out for a second. What does it mean when two full matrices get multiplied, not just a matrix times a vector?

We defined a matrix as a linear map, a function that takes a vector from one space to another. That was the matrix-vector story. For matrix-matrix, there are two ways to read what's happening, and both come up constantly.

Each row is a separate vector. Look at the left matrix $A$. Row 0 is a vector in $\mathbb{R}^4$. Row 1 is another vector in $\mathbb{R}^4$. The matrix is just those vectors stacked. When you compute $A \times B$, the matrix $B$ transforms each row of $A$ independently. Row 0 goes in, gets transformed, comes out as row 0 of the output. Row 1, same deal. The transformation is the same for every row; the input vectors are different.

In a neural network, the left matrix is your data (a stack of token embeddings, one row per token) and the right matrix is a learned weight matrix. Multiplying them applies the same transformation to every token simultaneously. A single @ replaces what would otherwise be a for-loop, and the parallelism maps directly to GPU hardware.

Composing two transformations. There's a second reading. When both matrices represent transformations, their product is the composed transformation. If $A$ maps from space $\mathcal{U}$ to $\mathcal{V}$ and $B$ maps from $\mathcal{V}$ to $\mathcal{W}$, then $BA$ maps from $\mathcal{U}$ directly to $\mathcal{W}$, collapsing two steps into one matrix. We'll see this in the next section, where a neural network chains weight matrices with nonlinearities between them.

Both readings are the same arithmetic under the hood. What changes is which matrix you think of as "the data" and which as "the function."

Matrices as transformations

The mechanics tell you how to compute a matrix multiply. But the deeper insight is in understanding what it means.

When you multiply a matrix by a vector, you're applying a linear transformation. You're rotating, scaling, shearing, projecting, or some combination of these. The matrix defines the transformation, and the vector is what gets transformed.

Here's the beautiful part: the columns of a matrix tell you exactly where the basis vectors land. If your 2×2 matrix is:

Then column 1, $[a, c]$, is where the basis vector $\hat{e}_1 = [1, 0]$ gets sent. Column 2, $[b, d]$, is where $\hat{e}_2 = [0, 1]$ gets sent. Every other vector in the space transforms accordingly, because every vector is a linear combination of the basis vectors. Once you know where the basis vectors go, you know where everything goes.

This is the insight that makes matrix multiplication concrete. A rotation matrix rotates the basis vectors. A scaling matrix stretches them. A shear matrix pushes one axis along the other. A projection matrix collapses a dimension to zero. Play with the interactive visualization below to see each one:

A few things worth noticing as you experiment:

• Rotation preserves the shape of the parallelogram (the area stays the same, det ≈ 1.00). The grid lines remain perpendicular. Nothing gets stretched or squashed, just turned.
• Shear deforms the square into a slanted parallelogram, but the area is preserved. One axis stays fixed while the other slides.
• Scale 2× doubles both basis vectors. The parallelogram is 4× the area (det = 4.00) because area scales with the determinant.
• Squeeze stretches one axis while compressing the other. The parallelogram changes shape dramatically, but the determinant (area factor) is 1.00. Area is preserved even though the shape isn't.
• Project → collapses the $y$-axis entirely. The red basis vector goes to zero. The determinant is 0, meaning the transformation destroys a dimension. Information is lost and you can't undo it.

That last one is especially worth thinking about. When a matrix has a zero determinant, it squashes the space down to a lower dimension. In a neural network, some layers deliberately do this (projection layers that reduce dimensionality), and others deliberately don't. The determinant tells you whether a transformation is information-preserving.

In a neural network, every weight matrix defines one of these transformations. The network has learned, through training, exactly which combination of rotation, scaling, and shearing to apply at each layer to turn raw input into useful representations. The Q, K, V projections in attention, the feedforward layers, the output projection: every one of them is a matrix multiply moving data from one representational space to another.

A neural network is a chain of matmuls

Here's what a forward pass through a simple two-layer network looks like:

Input vector $\mathbf{x}$ gets multiplied by weight matrix $W_1$ (a linear transformation), a bias vector $\mathbf{b}_1$ gets added (a translation), and the result passes through a nonlinear function (ReLU, which zeros out negative values). That gives us a hidden vector $\mathbf{h}$. Then $\mathbf{h}$ gets transformed by another matrix $W_2$, another bias gets added, and we get the output $\mathbf{y}$.

Two matmuls, two bias additions, one nonlinearity. A transformer is more complex in its wiring, but every individual operation in it (the attention projections for Q, K, and V, the output projection, the feedforward layers) is the same thing: a matrix multiply followed by some nonlinear processing.

This is why we started the entire series here. If you understand what $W\mathbf{x}$ does (a linear transformation that rotates, scales, and projects a vector from one space into another) you have the atom that everything else is built from. Attention? A particular pattern of matmuls with softmax in the middle. Feedforward layers? A matmul that expands dimensionality, a nonlinearity, then a matmul that compresses it back. Embedding lookups? Selecting a row from a weight matrix by index, which is equivalent to multiplying by a one-hot vector. The entire architecture is matmuls arranged in clever patterns, with nonlinearities between them to add the curves and bends that linear maps alone can't provide.

How practitioners interact with these objects

If you're going to work with neural networks, you're going to write code that manipulates tensors. PyTorch is the dominant framework for deep learning research and increasingly for production. The math maps to actual code, and the mapping is beautifully direct. Here is how simple it is:

import torch
 
# --- Creating tensors ---
x = torch.tensor([[1.0, 2.0, 0.0, 3.0],
                   [4.0, 1.0, 5.0, 2.0]])   # shape (2, 4)
W = torch.tensor([[2.0, 0.0, 1.0],
                   [1.0, 3.0, 0.0],
                   [0.0, 1.0, 2.0],
                   [3.0, 2.0, 1.0]])          # shape (4, 3)
 
y = x @ W   # shape (2, 3) — the @ operator is matmul
print(y)
# tensor([[11.,  8.,  4.],
#         [13., 10., 13.]])
 
# --- Fun with 2D transformations ---
# A 45-degree rotation matrix
import math
theta = math.pi / 4
R = torch.tensor([[math.cos(theta), -math.sin(theta)],
                   [math.sin(theta),  math.cos(theta)]])
 
# Some 2D points: a unit square
points = torch.tensor([[1.0, 0.0],
                        [1.0, 1.0],
                        [0.0, 1.0],
                        [0.0, 0.0]])  # shape (4, 2)
 
rotated = points @ R.T   # rotate all 4 points at once
print(rotated.round(decimals=3))
# Each point is now rotated 45° counterclockwise
 
# --- Scaling ---
S = torch.tensor([[2.0, 0.0],
                   [0.0, 0.5]])  # stretch x, squash y
scaled = points @ S.T
print(scaled)

Both NumPy and PyTorch use the @ operator for matrix multiplication. PyTorch also provides torch.mm (strictly 2D) and torch.matmul (handles arbitrary batch dimensions). In practice you'll see @ in most modern code. There's little ceremony here. The conceptual work is in understanding what the matmul does. The code is one character.

Misconceptions

"Vectors are arrows." This isn't wrong. It's one perfectly valid way to think about them, and in 2D and 3D it's an excellent picture. 3Blue1Brown's Essence of Linear Algebra series builds spectacular geometric intuition using exactly this representation. But "arrow" is one particular way to visualize vectors, not the definition. The deeper idea is an element of a space with algebraic structure: you can add them, scale them, and the results obey certain rules. The arrow picture breaks down once you're in 768 dimensions. The algebraic properties don't. Hold onto both: arrows when the dimension allows it, abstract algebra when it doesn't.

"You need to deeply understand linear algebra to understand neural networks." Let me be specific about what "deep understanding" means here, because it helps to know what you don't (yet) need. Deep linear algebra means eigendecomposition, singular value decomposition, Jordan normal form, change of basis proofs, rank-nullity arguments. These are real and useful tools. We'll touch on eigendecomposition when we get to PCA and interpretability, and SVD shows up in LoRA. But none of them are prerequisites for understanding how a transformer processes text right now. What you actually need at this point: what a vector is, what a matrix is, how matmul works mechanically, what shapes mean, and the idea that a matrix defines a transformation between spaces. That's what this post has been building. The rest of linear algebra is available on-demand when the series needs it, and we'll pick it up then.

What's next

This post covered the foundations: what spaces, vectors, matrices, and tensors are; why we use those words and not just "arrays"; the two kinds of multiplication and why matmul is the one that matters; and the geometric insight that a matrix defines a transformation of space. We watched matrices rotate, scale, shear, and project, and traced the connection from those 2D visualizations to the weight matrices inside neural networks.

The next post covers norms, dot products, and similarity, how the dot product becomes a measure of how similar two vectors are, and why that idea is the beating heart of attention.

Additional reading (and watching)

• Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning, Chapter 2: Linear Algebra. MIT Press. The standard textbook treatment of vectors, matrices, and linear maps in the deep learning context.
• Boyd, S. & Vandenberghe, L. (2018). Introduction to Applied Linear Algebra. Cambridge University Press. An excellent applied treatment that defines vectors and matrices from the ground up with practical motivation.
• Strang, G. (2019). Linear Algebra and Learning from Data. Wellesley-Cambridge Press. Bridges classic linear algebra with modern data science and deep learning applications.
• Zhang, A., et al. (2023). Dive into Deep Learning, Section 2.3: Linear Algebra. An interactive online textbook that covers tensor fundamentals with runnable code examples.
• 3Blue1Brown. (2016). Linear transformations and matrices. Essence of Linear Algebra series, Chapter 3. The best visual explanation of the "columns are transformed basis vectors" insight.
• He, K., et al. (2015). Deep Residual Learning for Image Recognition. arXiv:1512.03385. Demonstrated that depth (many layers of matmuls with skip connections) dramatically improves what networks can learn.
• 3Blue1Brown. (2016). Vectors, what even are they?. Essence of Linear Algebra series, Chapter 1. Covers the three perspectives on vectors (physics, CS, math) and why the "list of numbers" view is the most general.
• NVIDIA. (2017). NVIDIA Tesla V100 GPU Architecture. Whitepaper introducing Tensor Cores and the mixed-precision matrix multiply-accumulate operation.
• Dao, T., et al. (2022). FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. NeurIPS 2022. Reshaped how attention matmuls are executed by fusing them into a single memory-efficient kernel.
• Dettmers, T., et al. (2022). LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale. arXiv:2208.07339. Showed that large language models can be quantized to 8-bit integers with minimal quality loss.