Vectors, Matrices, and the Spaces They Live In

This is the first post in a series called Holding the LLM Stack in Your Head. The goal is to build up the mental models for the entire modern LLM stack, from the ground floor to agent protocols, in dependency order. If you want to know what the series is about and why I'm writing it, there's a longer explanation on the about page.

We start with linear algebra because neural networks transform their inputs through operations on vectors and matrices. The geometric view helps explain what those operations do.

Personally, I really benefited from revisiting this topic. As basic as it sounds, when I sat down to work through what these operations mean geometrically, I realized my intuition was cloudier than I expected. If you were introduced to this math in college and haven't actively engaged with the geometric interpretations since (which I'm guessing describes a lot of the people reading this), there's a good chance yours has faded too. So let's rebuild it from the bottom.

What is a "space"?

In phrases such as "embedding space" and "latent space," what does space mean?

In mathematics, a space is a set of objects with some specified structure. Different kinds of spaces provide different structure. Here we care about addition and scalar multiplication: the operations of a vector space.

A familiar example: the 2D plane. The objects are points (or equivalently, pairs of numbers like $(3, -1)$). The rules say you can add two points and get another point in the plane. You can scale a point by a number and get another point in the plane. Those two operations, addition and scalar multiplication, never throw you out of the plane. That is closure. Together with the usual algebraic rules for addition and scaling, it makes the plane a vector space.

These rules let us define linear transformations and reason about compositions of them. Adding a norm or inner product gives us tools for distance, angle, and projection. The real coordinate spaces used in neural networks usually come with those familiar choices.

Vectors

So what exactly is a vector?

A vector is an element of a vector space. A vector space is a set whose elements satisfy specific rules about addition and scaling (technically called the vector space axioms). The important ones for us:

1. You can add two vectors and get another vector in the same space.
2. You can multiply a vector by a scalar (a single number) and still get a vector in the same space.
3. There's a zero vector that acts as the identity for addition.

Why do we call these things "vectors" instead of just "arrays" or "lists of numbers"? Because the word "vector" carries meaning that "array" doesn't. When you call something a vector, you're saying it lives in a space with algebraic structure. It can be added, scaled, combined, and transformed in ways that are guaranteed to behave well. A plain array of bytes doesn't promise any of that. You can concatenate arrays, zip them, reverse them. But "add two arrays" and "scale an array" aren't operations with guaranteed mathematical properties unless you specifically define them to be. Calling it a vector specifies the algebra we use with its entries.

In practice, in deep learning, a vector is a list of numbers like $[0.5, -1.2, 3.0, 0.8]$. Four numbers, in a specific order, with one coordinate at each position. When people say a language model has an "embedding dimension of 768," they mean every token in the vocabulary gets represented as a vector with 768 numbers. Information is distributed across those coordinates; a single slot need not have an identifiable meaning. We don't get to pick what each slot means; the model figures that out during training. But the point is: a token's identity, as far as the network is concerned, is that vector.

In a neural network context, these vectors are sometimes called activations to distinguish them from the model's weights. Activations are the data flowing through the network; they change for every input. This is in contrast to parameters (or weights), which are the learned values that stay fixed during inference. Activations and parameters differ in their role, not their shape. Activations can be matrices or higher-rank tensors, and parameters can include vectors such as biases.

In 2D we can draw vectors and see the geometry. We cannot draw the full geometry of 768 dimensions. But the algebra works exactly the same way. Every operation we define in 2D (addition, scaling, dot products, projections) generalizes to $\mathbb{R}^{768}$ without modification.

Matrices

If a vector is a list of numbers, a matrix is a grid of numbers with rows and columns.

But here's what took me a while to really appreciate: a matrix isn't just a container for data. A matrix is a function. Specifically, it's a linear map, a function that takes a vector from one space and delivers it to another space (or the same space), while preserving the addition and scaling structure along the way.

A matrix represents a linear transformation compactly once we choose coordinates. Instead of describing a linear function as a long list of rules ("dimension 1 of the output equals 2 times dimension 1 of the input plus 3 times dimension 2 of the input, plus..."), you write the coefficients in a grid and multiply. The matrix is the transformation, encoded as a table of coefficients.

In a neural network, matrices show up as weights (or parameters). When people say a layer "has 3 million parameters," those parameters include the entries of its weight matrices and, where present, its biases or other learned tensors. The network learned those values during training, and they stay fixed at inference time. They sit there, defining a transformation, waiting to be applied to whatever input flows through.

Here's a small example. A weight matrix $W$ with 2 rows and 4 columns:

This matrix defines a function from 4-dimensional space to 2-dimensional space. Hand it a vector with 4 numbers and it gives you back a vector with 2 numbers. In a real network, this matrix might have millions or billions of entries, but the structure is the same: a grid of learned numbers that defines a transformation between spaces.

Tensors: the generalization

You'll hear the word "tensor" constantly in deep learning, mostly because PyTorch calls its basic data type a Tensor. Here's the hierarchy:

• A scalar is a single number. Zero dimensions.
• A vector is a list of numbers. One dimension. Shape: $[d]$.
• A matrix is a grid of numbers. Two dimensions. Shape: $[m, n]$.
• A tensor is the generalization to any number of dimensions. Shape: $[d_1, d_2, \ldots, d_n]$.

In deep-learning code, "tensor" usually means a multi-dimensional array, without requiring the coordinate-transformation rules of tensors in mathematics and physics. A 3D tensor with shape $[B, T, d]$ represents a batch of $B$ sequences, each with $T$ tokens, each token encoded as a $d$-dimensional vector. That's the shape you'll encounter constantly in transformer code.

Shapes are useful for debugging and reading model code. A shape mismatch often reveals that two operations expect different representations. A few shapes to get comfortable with:

• $[d]$: a single vector. An embedding of one token.
• $[B, d]$: a batch of $B$ vectors. $B$ inputs, each with $d$ features.
• $[B, T, d]$: a batch of sequences. $B$ sequences, each $T$ tokens long, each token a $d$-dimensional vector. Transformer layers commonly operate on tensors with this shape.

Two kinds of multiplication

Element-wise multiplication and matrix multiplication combine entries in different ways.

Element-wise multiplication (also called the Hadamard product, written $\odot$) does exactly what it sounds like. Take two objects of the same shape and multiply corresponding entries:

The output has the same shape; each entry interacts only with its counterpart.

Matrix multiplication is completely different. It mixes information across dimensions. The shapes don't even need to match the same way; what matters is that the inner dimensions agree. A $[2, 4]$ matrix times a $[4, 3]$ matrix gives a $[2, 3]$ matrix. The output shape is assembled from the outer dimensions of the inputs.

Matrix multiplication

The mechanical rule: each element of the output is a dot product between a row of the first matrix and a column of the second. A dot product means you multiply corresponding elements and add them up.

Say matrix $A$ is $2 \times 4$ and matrix $B$ is $4 \times 3$. The output $C$ will be $2 \times 3$. To compute the entry at row $i$, column $j$:

Take row $i$ from $A$, take column $j$ from $B$, multiply element-wise, sum. That's one dot product, and it produces one number in the output. Repeat for every $(i, j)$ pair.

But zoom out for a second. What does it mean when two full matrices get multiplied, not just a matrix times a vector?

We defined a matrix as a linear map, a function that takes a vector from one space to another. That was the matrix-vector story. For matrix-matrix, there are two ways to read what's happening, and both come up constantly.

Each row is a separate vector. Look at the left matrix $A$. Row 0 is a vector in $\mathbb{R}^4$. Row 1 is another vector in $\mathbb{R}^4$. The matrix is just those vectors stacked. When you compute $A \times B$, the matrix $B$ transforms each row of $A$ independently. Row 0 goes in, gets transformed, comes out as row 0 of the output. Row 1, same deal. The transformation is the same for every row; the input vectors are different.

In a neural network, the left matrix is your data (a stack of token embeddings, one row per token) and the right matrix is a learned weight matrix. Multiplying them applies the same transformation to every token simultaneously. A single @ replaces what would otherwise be a for-loop, and the parallelism maps directly to GPU hardware.

Composing two transformations. There's a second reading. When both matrices represent transformations, their product is the composed transformation. Using column vectors, if $A$ maps from space $\mathcal{U}$ to $\mathcal{V}$ and $B$ maps from $\mathcal{V}$ to $\mathcal{W}$, then $BA$ maps from $\mathcal{U}$ directly to $\mathcal{W}$, collapsing two steps into one matrix. We'll see this in the next section, where a neural network chains weight matrices with nonlinearities between them.

With row vectors, applying $A$ and then $B$ is instead written $xAB$. Keep that convention separate from the column-vector expression $BAx$. Both readings use the same arithmetic. What changes is which matrix you think of as "the data" and which as "the function."

Matrices as transformations

When you multiply a matrix by a vector, you're applying a linear transformation. You're rotating, scaling, shearing, projecting, or some combination of these. The matrix defines the transformation, and the vector is what gets transformed.

The columns of a matrix tell you exactly where the basis vectors land. If your 2×2 matrix is:

Then column 1, $[a, c]$, is where the basis vector $\hat{e}_1 = [1, 0]$ gets sent. Column 2, $[b, d]$, is where $\hat{e}_2 = [0, 1]$ gets sent. Every other vector in the space transforms accordingly, because every vector is a linear combination of the basis vectors. Once you know where the basis vectors go, you know where everything goes.

A rotation matrix rotates the basis vectors. A scaling matrix stretches them. A shear matrix pushes one axis along the other. A projection matrix collapses a dimension to zero. Play with the interactive visualization below to see each one:

The presets show different effects on shape and area:

• Rotation preserves the shape of the parallelogram (the area stays the same, det ≈ 1.00). The grid lines remain perpendicular. Nothing gets stretched or squashed, just turned.
• Shear deforms the square into a slanted parallelogram, but the area is preserved. One axis stays fixed while the other slides.
• Scale 2× doubles both basis vectors. The parallelogram is 4× the area (det = 4.00) because area scales with the determinant.
• Squeeze stretches one axis while compressing the other. The parallelogram changes shape dramatically, but the determinant (area factor) is 1.00. Area is preserved even though the shape isn't.
• Project → collapses the $y$-axis entirely. The red basis vector goes to zero. The determinant is 0, meaning the transformation destroys a dimension. Information is lost and you can't undo it.

When a square matrix has a zero determinant, it squashes the space down to a lower dimension. In a neural network, some layers deliberately do this (projection layers that reduce dimensionality), and others deliberately don't. For square matrices, a nonzero determinant means the map is invertible. Rectangular maps have no determinant; their rank tells us whether different inputs can collapse to the same output.

In a neural network, every weight matrix defines one of these transformations. The network has learned, through training, exactly which combination of rotation, scaling, and shearing to apply at each layer to turn raw input into useful representations. The Q, K, V projections in attention, the feedforward layers, the output projection: every one of them is a matrix multiply moving data from one representational space to another.

A neural network is a chain of matmuls

Here's what a forward pass through a simple two-layer network looks like:

Input vector $\mathbf{x}$ gets multiplied by weight matrix $W_1$ (a linear transformation), a bias vector $\mathbf{b}_1$ gets added (a translation), and the result passes through a nonlinear function (ReLU, which zeros out negative values). That gives us a hidden vector $\mathbf{h}$. Then $\mathbf{h}$ gets transformed by another matrix $W_2$, another bias gets added, and we get the output $\mathbf{y}$.

Two matmuls, two bias additions, one nonlinearity. Transformer feedforward layers use this pattern, with different dimensions and activation functions.

How practitioners interact with these objects

PyTorch represents these arrays as tensors. The code below creates vectors and matrices, multiplies them, and applies transformations to a set of points.

import torch
 
# --- Creating tensors ---
x = torch.tensor([[1.0, 2.0, 0.0, 3.0],
                   [4.0, 1.0, 5.0, 2.0]])   # shape (2, 4)
W = torch.tensor([[2.0, 0.0, 1.0],
                   [1.0, 3.0, 0.0],
                   [0.0, 1.0, 2.0],
                   [3.0, 2.0, 1.0]])          # shape (4, 3)
 
y = x @ W   # shape (2, 3) — the @ operator is matmul
print(y)
# tensor([[13., 12.,  4.],
#         [15., 12., 16.]])
 
# --- Fun with 2D transformations ---
# A 45-degree rotation matrix
import math
theta = math.pi / 4
R = torch.tensor([[math.cos(theta), -math.sin(theta)],
                   [math.sin(theta),  math.cos(theta)]])
 
# Some 2D points: a unit square
points = torch.tensor([[1.0, 0.0],
                        [1.0, 1.0],
                        [0.0, 1.0],
                        [0.0, 0.0]])  # shape (4, 2)
 
rotated = points @ R.T   # rotate all 4 points at once
print(rotated.round(decimals=3))
# Each point is now rotated 45° counterclockwise
 
# --- Scaling ---
S = torch.tensor([[2.0, 0.0],
                   [0.0, 0.5]])  # stretch x, squash y
scaled = points @ S.T
print(scaled)

Both NumPy and PyTorch use the @ operator for matrix multiplication. PyTorch also provides torch.mm (strictly 2D) and torch.matmul (handles arbitrary batch dimensions).

Misconceptions

"Vectors are arrows." This isn't wrong. It's one perfectly valid way to think about them, and in 2D and 3D it's an excellent picture. 3Blue1Brown's Essence of Linear Algebra series builds spectacular geometric intuition using exactly this representation. But "arrow" is one particular way to visualize vectors, not the definition. The deeper idea is an element of a space with algebraic structure: you can add them, scale them, and the results obey certain rules. The arrow picture breaks down once you're in 768 dimensions. The algebraic properties don't. Hold onto both: arrows when the dimension allows it, abstract algebra when it doesn't.

"You need to deeply understand linear algebra to understand neural networks." The immediate prerequisites are vectors, matrices, matrix multiplication, tensor shapes, and linear maps. Later topics draw on more algebra: LoRA uses low-rank factorization, for example, although implementing it does not require an SVD. The series introduces those tools where they are needed.

What's next

The next post covers norms, dot products, and similarity, how the dot product becomes a measure of how similar two vectors are, and how attention uses that comparison.

Additional reading (and watching)

• Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning, Chapter 2: Linear Algebra. MIT Press. The standard textbook treatment of vectors, matrices, and linear maps in the deep learning context.
• Boyd, S. & Vandenberghe, L. (2018). Introduction to Applied Linear Algebra. Cambridge University Press. An excellent applied treatment that defines vectors and matrices from the ground up with practical motivation.
• Strang, G. (2019). Linear Algebra and Learning from Data. Wellesley-Cambridge Press. Bridges classic linear algebra with modern data science and deep learning applications.
• Zhang, A., et al. (2023). Dive into Deep Learning, Section 2.3: Linear Algebra. An interactive online textbook that covers tensor fundamentals with runnable code examples.
• 3Blue1Brown. (2016). Linear transformations and matrices. Essence of Linear Algebra series, Chapter 3. The best visual explanation of the "columns are transformed basis vectors" insight.
• He, K., et al. (2015). Deep Residual Learning for Image Recognition. arXiv:1512.03385. Demonstrated that depth (many layers of matmuls with skip connections) dramatically improves what networks can learn.
• 3Blue1Brown. (2016). Vectors, what even are they?. Essence of Linear Algebra series, Chapter 1. Covers the three perspectives on vectors (physics, CS, math) and why the "list of numbers" view is the most general.
• NVIDIA. (2017). NVIDIA Tesla V100 GPU Architecture. Whitepaper introducing Tensor Cores and the mixed-precision matrix multiply-accumulate operation.
• Dao, T., et al. (2022). FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. NeurIPS 2022. Reshaped how attention matmuls are executed by fusing them into a single memory-efficient kernel.
• Dettmers, T., et al. (2022). LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale. arXiv:2208.07339. Showed that large language models can be quantized to 8-bit integers with minimal quality loss.