Gradients and How Machines Learn

Okay so I keep hearing "gradient descent." Every ML tutorial, every paper, every talk. "We compute the gradient, then we descend." Great. But what is a gradient? Like, the slope of a hill? Sort of, but not exactly. And what does "descending" it mean when you have millions of parameters?

In the last post we built up the idea of a loss function. We have some model, it makes predictions, those predictions are wrong by some measurable amount, and the loss function puts a number on that wrongness. Lower loss means better predictions. The question now is: how do we make the loss go down?

The naive answer is "just try random stuff and keep what works." And honestly, that is not a terrible intuition for small problems. But neural networks have millions of parameters. Billions, now. Trying random changes would take longer than the heat death of the universe. We need a way to figure out which direction to adjust each parameter, and by how much.

That is what gradients give us.

What a gradient actually is

Let's start simple. Say you have a function of one variable, like $f(x) = x^2$. The derivative $f'(x) = 2x$ tells you the slope at any point. If you're at $x = 3$, the slope is 6, meaning the function is increasing steeply to the right. If you want to decrease $f$, you should move to the left, in the opposite direction of the slope.

That's one dimension. Now what if your function has two inputs? Say $f(x, y) = x^2 + y^2$. This is a bowl shape. At any point on the surface, you could ask "how steep is it in the x direction?" and "how steep is it in the y direction?" separately. Those are partial derivatives:

The gradient is just those partial derivatives bundled together into a vector:

That's it. The gradient is a vector with one component per input variable. Each component tells you "if I nudge this variable a tiny bit, holding the others fixed, how much does the output change?"

The key property: the gradient points in the direction of steepest ascent. If you're standing on a hilly landscape and the gradient tells you "go northeast," then going northeast increases $f$ the fastest. To decrease $f$, you walk the opposite way, against the gradient.

This extends to any number of dimensions. A neural network with 10 million parameters has a loss function $L(\theta_1, \theta_2, \ldots, \theta_{10{,}000{,}000})$. The gradient of $L$ is a vector with 10 million components, one per parameter. Each component says "how much does the loss change if I nudge this particular weight?"

You never actually visualize a 10-million-dimensional landscape. But the math works the same way. The gradient still points uphill, and for a small enough step, going against it decreases the function locally. (Step too far and you can overshoot; we'll see this with learning rates in a moment.)

The update rule: gradient descent

So here's the algorithm. You have parameters $\theta$ (a big vector of all the weights in your network). You have a loss $L(\theta)$. You want to find the $\theta$ that makes $L$ small.

Step 1: Compute $\nabla L(\theta)$, the gradient of the loss with respect to every parameter.

Step 2: Update:

Where $\eta$ (eta) is the learning rate, a small positive number like 0.01 or 0.001. It controls how big a step you take.

Step 3: Repeat thousands or millions of times.

That is gradient descent. The entire deep learning revolution runs on this loop, plus a lot of engineering to make it fast.

The learning rate matters a lot. Too big and you overshoot the minimum, bouncing wildly around the landscape. Too small and you crawl toward the minimum so slowly that training takes forever. Finding a good learning rate is one of the oldest and most persistent headaches in machine learning.

Let's see this in action. Here's a simple bowl-shaped loss surface with three different learning rates trying to reach the minimum at the center:

The middle panel (lr = 0.2) is the one you want — a brisk, monotonic walk into the minimum. The left panel (lr = 0.02) is moving in the right direction but at a crawl; you'd need hundreds more steps to get to the bottom. The right panel (lr = 0.95) is the cautionary tale: each step overshoots so badly that the path zig-zags across the minimum, getting close but never settling. With a slightly larger learning rate it would diverge entirely.

Backpropagation: computing the gradient

So gradient descent says "compute the gradient, walk downhill." Simple enough as a concept. But computing the gradient of a neural network with millions of parameters is the hard part. How do you figure out how each weight contributes to the final loss?

This is where backpropagation comes in. The word "backprop" gets thrown around like it's a separate piece of mathematics. It isn't. Backpropagation is the chain rule from calculus, applied systematically to a computation graph, with some bookkeeping so you don't redo work.

The chain rule says: if $y = f(g(x))$, then

You decompose the derivative of a composed function into a product of simpler derivatives. A neural network is just a very long chain of composed functions. Layer 1 feeds into Layer 2, which feeds into Layer 3, which feeds into the loss. Each layer is a relatively simple operation (a matrix multiply, an activation function, etc.), and the chain rule lets you compute the gradient of the loss with respect to any weight in the network by multiplying together the local derivatives along the path from that weight to the output.

Here's the mechanic on a toy graph, $L = (w \cdot x)^2$ with $x=2, w=3$:

Watch one cycle: first the values fill in going forward, then gradients propagate backward. The gradient at z is 2z = 12. That single number then gets multiplied by the other input to get each weight's gradient: ∂L/∂w = 12·x = 24, ∂L/∂x = 12·w = 36. That's the chain rule doing its job — the gradient at the output gets multiplied through the graph, collecting local derivatives as it goes.

Think of the network as a pipeline:

Input $\rightarrow$ Layer 1 $\rightarrow$ Layer 2 $\rightarrow$ Layer 3 $\rightarrow$ Loss

The forward pass pushes data through left to right, computing the output and the loss.

The backward pass goes right to left. Starting from the loss, it asks: how does the loss change with respect to Layer 3's output? Given that, how does it change with respect to Layer 3's weights? And with respect to Layer 2's output? Each step applies the chain rule to propagate the gradient one layer deeper. By the time you reach Layer 1, you have the gradient for every single parameter in the network.

The reason this is efficient is that you compute each local derivative once and reuse it. The alternative, forward-mode differentiation, would require a separate pass per parameter. That's fine when there are a few inputs and one output, but catastrophic when there are millions of parameters and one scalar loss. Reverse-mode autodiff (which is what backpropagation is) gets the entire gradient in one backward pass, usually within a small constant factor of the forward pass in compute, at the cost of storing or recomputing intermediate activations. That memory cost is real and is the reason gradient checkpointing exists, but the asymptotic story is unbeatable for "scalar loss, lots of parameters."

Rumelhart, Hinton, and Williams published the paper that made backpropagation famous in 1986, though the idea had been discovered and rediscovered multiple times before that. Their key contribution was showing that hidden units could learn useful internal representations and that multi-layer networks could be trained effectively in practice. Before that, many researchers were skeptical that multi-layer networks could be trained at scale. Nielsen's walk-through of the four backprop equations and 3Blue1Brown's geometric version are the two references I keep coming back to.

Worked example: learning y = 3x + 1

Let me make this concrete with the smallest possible example. We have a model with two parameters:

We want it to learn the function $y = 3x + 1$. So the true values are $w = 3$ and $b = 1$. We'll use mean squared error as our loss:

Let's pick a single training example: $x = 1$, $y = 4$ (because $3 \cdot 1 + 1 = 4$). Initialize both weights to zero: $w = 0$, $b = 0$. Learning rate $\eta = 0.1$.

Step 1:

Forward pass: $\hat{y} = 0 \cdot 1 + 0 = 0$. Loss: $(0 - 4)^2 = 16$.

Gradients (using the chain rule):

Update: $w = 0 - 0.1 \cdot (-8) = 0.8$. $b = 0 - 0.1 \cdot (-8) = 0.8$.

Step 2:

Forward pass: $\hat{y} = 0.8 \cdot 1 + 0.8 = 1.6$. Loss: $(1.6 - 4)^2 = 5.76$.

Gradients: $\frac{\partial L}{\partial w} = -4.8$, $\frac{\partial L}{\partial b} = -4.8$. Update: $w = 1.28$, $b = 1.28$.

Step 3:

Forward pass: $\hat{y} = 2.56$. Loss: $2.07$.

See it? The loss went $16 \rightarrow 5.76 \rightarrow 2.07$. Each step, the gradient told us which direction to move the weights, and the loss decreased. With enough steps and sufficiently varied clean data, the weights converge to the true values. With noisy data, they converge to the best-fitting line under MSE: close to the true parameters, but biased by whatever noise happens to be in the sample. Here's that full process:

The line rotates from wrong (negative slope) to roughly right (slope ≈ 3) over about 50 steps. The faint red segments from each point to the line are the residuals, which is what MSE sums the squares of. As the line aligns with the data, those segments shrink and the loss drops.

Same loop. Same math. Just more data points and more steps than the arithmetic above. The conceptual loop in a 175-billion-parameter GPT is still this one: forward pass, compute loss, backward pass, update weights, repeat. The scaled-up version adds cross-entropy next-token loss, AdamW-style optimizers, gradient clipping, mixed precision, distributed data and model parallelism, activation checkpointing, and learning-rate schedules, but those are all engineering layered on top of the same forward/backward/update spine.

The training loop in practice

In PyTorch, the training loop looks almost disappointingly simple:

import torch
import torch.nn as nn
 
# A tiny model: one linear layer
model = nn.Linear(1, 1)
 
# MSE loss and SGD optimizer
criterion = nn.MSELoss()
optimizer = torch.optim.SGD(model.parameters(), lr=0.01)
 
# Training data: y = 3x + 1
x = torch.tensor([[1.0], [2.0], [3.0], [4.0]])
y = torch.tensor([[4.0], [7.0], [10.0], [13.0]])
 
# The loop
for epoch in range(200):
    optimizer.zero_grad()        # Clear stale gradients first
    pred = model(x)              # Forward pass
    loss = criterion(pred, y)    # Compute loss
    loss.backward()              # Backward pass (compute gradients)
    optimizer.step()             # Update weights

That's fourteen lines of actual code. Let me walk through what's happening.

model(x) runs the forward pass. Data goes in, a prediction comes out. Internally, PyTorch builds a computation graph as it goes, recording every operation.

loss.backward() is where the magic happens. PyTorch walks the computation graph backwards, applying the chain rule at each node, and stores the gradient for every parameter. When this call finishes, model.weight.grad and model.bias.grad contain the gradients.

optimizer.step() takes those gradients and applies the update rule: $\theta \leftarrow \theta - \eta \cdot \nabla L$. Each parameter gets nudged in the direction that decreases the loss.

optimizer.zero_grad() clears out the stored gradients so they don't accumulate from the previous step. PyTorch accumulates gradients by default, which is useful for gradient accumulation across micro-batches, but in a plain loop you need to zero them out exactly once per iteration. The conventional placement is at the top of the loop (as above); placing it after optimizer.step() is also valid as long as it happens before the next backward().

One small precision note: nn.MSELoss() defaults to mean squared error, not sum, so the gradient magnitude is normalized by batch size. The earlier hand calculation used a single example, where mean and sum agree. With the four-point batch above the gradients are 1/4 of what the sum-form would give, which is part of why the same learning rate that worked in the hand example might want tuning here.

Everything else (data loading, batching, logging, checkpointing, distributed training) is infrastructure built around this loop.

Stochastic gradient descent

One thing I glossed over: in practice, you don't compute the gradient over your entire dataset at once. If you have a million training examples, computing the loss and gradient for all of them before taking a single step would be extremely slow.

Instead, you sample a small random batch (say 32 or 64 examples), compute the gradient on just that batch, and update. This is stochastic gradient descent (SGD). "Stochastic" just means random. If the batch is sampled properly, the mini-batch gradient is an unbiased estimate of the full-dataset gradient: it points in the right direction on average, even though any single batch can be noisy or even point the wrong way. In exchange for that noise, you get many more steps per second.

The noise can be beneficial. In many settings, it helps the optimizer explore the landscape and biases training away from sharp, brittle minima toward flatter regions that tend to generalize better. But it's a tradeoff: too much noise (too small a batch, too high a learning rate, badly correlated sampling) can destabilize training. The "noise as regularizer" view is real, just not unconditional.

The landscape gets harder

The bowl-shaped loss surface above was a friendly example. In real neural networks, the landscape is much more complicated. Valleys are elongated. There are saddle points where the gradient is nearly zero but you're not at a minimum. Some directions are steep and others are almost flat.

Here's what vanilla gradient descent looks like on an elongated valley, $f(x, y) = x^2 + 10y^2$:

See that zig-zag pattern? The gradient is steep in the y direction (the narrow axis of the valley) and shallow in the x direction (the long axis). So the optimizer takes big steps up and down in y, overshooting back and forth, while barely making progress in x. It eventually converges, but painfully slowly.

This is where plain gradient descent starts to look inefficient. SGD with momentum can mitigate it, but the cleanest fix is the family of fancier optimizers we'll build up in the next post: momentum, RMSProp, Adam. They all exist to handle variations of this problem. The post also briefly touched on saddle points (places where the gradient is near zero but the surface is neither a max nor a min), which are a real obstacle in high-dimensional nonconvex landscapes, and another reason adaptive optimizers tend to win at scale.

Common misconceptions

"Gradient = slope." Close, but not quite. A slope is a scalar (a single number). A gradient is a vector, with one component per dimension. In one dimension they're the same thing, which is where the confusion comes from. But in the multi-dimensional parameter spaces of neural networks, the gradient is a direction in that high-dimensional space, not just a steepness.

"Backpropagation is some special, mysterious algorithm." Under the hood, it's the chain rule, applied to a computation graph, with some bookkeeping to avoid redundant computation. The "discovery" was realizing that you could apply it efficiently to train multi-layer neural networks, not inventing new math. Andrej Karpathy's micrograd project implements the entire thing in about 100 lines of Python, and his video walkthrough builds it up from nothing. Worth reading if you want to see the whole thing laid bare.

"You need deep calculus to understand this." You need three things from calculus: the concept of a derivative (how much does the output change when I nudge an input?), partial derivatives (which are just regular derivatives where you hold some variables constant), and the chain rule (how do derivatives compose?). If you understand those three ideas, you have the mathematical foundation for all of gradient-based optimization.

What's next

We now have the core loop: forward, compute loss, backward, update, repeat. But vanilla SGD has problems, as we just saw on the elongated valley. It oscillates in the steep directions, it crawls in the shallow ones, and it's painfully sensitive to the learning rate. The next post fixes those problems.

Optimizers: momentum, Adam, and learning rate schedules.

Additional reading (and watching)

• Rumelhart, D., Hinton, G., & Williams, R. (1986). Learning representations by back-propagating errors. Nature, 323, 533-536. The paper that made backprop famous, though the algorithm had been independently rediscovered several times before that.
• Karpathy, A. (2020). micrograd. A scalar-valued autograd engine and tiny neural net library in about 100 lines of Python. The clearest possible read of how reverse-mode autodiff actually works.
• Nielsen, M. (2015). Neural Networks and Deep Learning, Chapter 2: How the backpropagation algorithm works. The most patient walk-through of the four backprop equations I know of.
• 3Blue1Brown. (2017). Backpropagation, intuitively and Backpropagation calculus. Neural Networks series, Chapters 3 and 4. The geometric story for the chain rule applied to a network.
• Karpathy, A. (2022). The spelled-out intro to neural networks and backpropagation: building micrograd. YouTube. A patient code-along that builds reverse-mode autodiff from nothing.
• Bottou, L., Curtis, F. E., & Nocedal, J. (2018). Optimization Methods for Large-Scale Machine Learning. SIAM Review, 60(2), 223-311. The careful treatment of why SGD with mini-batches works as well as it does, including the noise-as-regularizer view.
• Chen, T., et al. (2016). Training Deep Nets with Sublinear Memory Cost. arXiv:1604.06174. The original gradient checkpointing paper. Frames the activation-vs-recompute tradeoff that every modern LLM trainer relies on.