Gradients and How Machines Learn

What does it mean to move downhill when a model has millions of parameters? A gradient gives us a direction in that parameter space.

In the last post we built up the idea of a loss function. We have some model, it makes predictions, those predictions are wrong by some measurable amount, and the loss function puts a number on that wrongness. Lower loss means better predictions. The question now is: how do we make the loss go down?

We could try random parameter changes and keep improvements. For a model with millions or billions of parameters, a gradient offers a much more directed approach: estimate how each parameter affects the loss, then use that information to choose an update.

What a gradient actually is

Let's start simple. Say you have a function of one variable, like $f(x) = x^2$. The derivative $f'(x) = 2x$ tells you the slope at any point. If you're at $x = 3$, the slope is 6, meaning the function is increasing steeply to the right. If you want to decrease $f$, you should move to the left, in the opposite direction of the slope.

That's one dimension. Now what if your function has two inputs? Say $f(x, y) = x^2 + y^2$. This is a bowl shape. At any point on the surface, you could ask "how steep is it in the x direction?" and "how steep is it in the y direction?" separately. Those are partial derivatives:

The gradient is just those partial derivatives bundled together into a vector:

The gradient is a vector with one component per input variable. Each component tells you "if I nudge this variable a tiny bit, holding the others fixed, how much does the output change?"

The gradient points in the direction of steepest ascent under Euclidean distance. If you're standing on a hilly landscape and the gradient tells you "go northeast," then going northeast increases $f$ the fastest. To decrease $f$, you walk the opposite way, against the gradient.

This extends to any number of dimensions. A neural network with 10 million parameters has a loss function $L(\theta_1, \theta_2, \ldots, \theta_{10{,}000{,}000})$. The gradient of $L$ is a vector with 10 million components, one per parameter. Each component says "how much does the loss change if I nudge this particular weight?"

You never actually visualize a 10-million-dimensional landscape. But the math works the same way. The gradient still points uphill, and for a small enough step, going against it decreases the function locally. (Step too far and you can overshoot; we'll see this with learning rates in a moment.)

The update rule: gradient descent

So here's the algorithm. You have parameters $\theta$ (a big vector of all the weights in your network). You have a loss $L(\theta)$. You want to find the $\theta$ that makes $L$ small.

Step 1: Compute $\nabla L(\theta)$, the gradient of the loss with respect to every parameter.

Step 2: Update:

Where $\eta$ (eta) is the learning rate, a small positive number like 0.01 or 0.001. It controls how big a step you take.

Step 3: Repeat thousands or millions of times.

A large learning rate can overshoot a minimum or make updates diverge. A very small one can require many more updates to reduce the loss.

Here's a simple bowl-shaped loss surface with three different learning rates trying to reach the minimum at the center:

On this bowl, lr = 0.2 converges monotonically and faster than lr = 0.02. At lr = 0.95, each coordinate is multiplied by −0.9 per step, so the path alternates across the minimum while converging. At lr = 1 it would oscillate without shrinking; above 1 it would diverge. These thresholds depend on this loss surface.

Backpropagation: computing the gradient

How do we compute a loss gradient with respect to millions of parameters?

Backpropagation applies the chain rule systematically to a computation graph, reusing intermediate results as it works backward from the loss.

The chain rule says: if $y = f(g(x))$, then

You decompose the derivative of a composed function into a product of simpler derivatives. A neural network is just a very long chain of composed functions. Layer 1 feeds into Layer 2, which feeds into Layer 3, which feeds into the loss. Each layer is a relatively simple operation (a matrix multiply, an activation function, etc.), and the chain rule lets you compute the gradient of the loss with respect to any weight in the network by multiplying local derivatives along each path from that weight to the output and summing contributions where paths branch.

Here's the mechanic on a toy graph, $L = (w \cdot x)^2$ with $x=2, w=3$:

Watch one cycle: first the values fill in going forward, then gradients propagate backward. The gradient at z is 2z = 12. That single number then gets multiplied by the other input to get each weight's gradient: ∂L/∂w = 12·x = 24, ∂L/∂x = 12·w = 36.

Input $\rightarrow$ Layer 1 $\rightarrow$ Layer 2 $\rightarrow$ Layer 3 $\rightarrow$ Loss

The forward pass pushes data through left to right, computing the output and the loss.

The backward pass goes right to left. Starting from the loss, it asks: how does the loss change with respect to Layer 3's output? Given that, how does it change with respect to Layer 3's weights? And with respect to Layer 2's output? Each step applies the chain rule to propagate the gradient one layer deeper. By the time you reach Layer 1, you have the gradient for every single parameter in the network.

For a scalar loss with many parameters, reverse-mode autodiff computes the full gradient in one backward pass, usually within a small constant factor of the forward-pass cost. It needs stored or recomputed intermediate activations; gradient checkpointing trades that storage for additional computation. Forward-mode differentiation instead propagates one input direction at a time, which is useful with few inputs or when only a directional derivative is needed.

Rumelhart, Hinton, and Williams published the paper that made backpropagation famous in 1986, though the idea had been discovered and rediscovered multiple times before that. Their key contribution was showing that hidden units could learn useful internal representations and that multi-layer networks could be trained effectively in practice. Before that, many researchers were skeptical that multi-layer networks could be trained at scale. Nielsen’s walk-through and 3Blue1Brown’s geometric explanation offer two ways to follow the calculation.

Worked example: learning y = 3x + 1

Consider a model with two parameters:

We want it to learn the function $y = 3x + 1$. So the true values are $w = 3$ and $b = 1$. We'll use mean squared error as our loss:

Let's pick a single training example: $x = 1$, $y = 4$ (because $3 \cdot 1 + 1 = 4$). Initialize both weights to zero: $w = 0$, $b = 0$. Learning rate $\eta = 0.1$.

Step 1:

Forward pass: $\hat{y} = 0 \cdot 1 + 0 = 0$. Loss: $(0 - 4)^2 = 16$.

Gradients (using the chain rule):

Update: $w = 0 - 0.1 \cdot (-8) = 0.8$. $b = 0 - 0.1 \cdot (-8) = 0.8$.

Step 2:

Forward pass: $\hat{y} = 0.8 \cdot 1 + 0.8 = 1.6$. Loss: $(1.6 - 4)^2 = 5.76$.

Gradients: $\frac{\partial L}{\partial w} = -4.8$, $\frac{\partial L}{\partial b} = -4.8$. Update: $w = 1.28$, $b = 1.28$.

Step 3:

Forward pass: $\hat{y} = 2.56$. Loss: $2.07$.

The loss went $16 \rightarrow 5.76 \rightarrow 2.07$. Repeating only this example would make $w$ and $b$ approach 2: all it tells us is $w+b=4$. To identify slope and intercept separately, we need varied inputs. The animation fits a noisy sample, whose best-fitting parameters need not be exactly 3 and 1.

The training loop in practice

In PyTorch, the loop computes predictions and loss, then applies the gradients:

import torch
import torch.nn as nn
 
# A tiny model: one linear layer
model = nn.Linear(1, 1)
 
# MSE loss and SGD optimizer
criterion = nn.MSELoss()
optimizer = torch.optim.SGD(model.parameters(), lr=0.01)
 
# Training data: y = 3x + 1
x = torch.tensor([[1.0], [2.0], [3.0], [4.0]])
y = torch.tensor([[4.0], [7.0], [10.0], [13.0]])
 
# The loop
for epoch in range(200):
    optimizer.zero_grad()        # Clear stale gradients first
    pred = model(x)              # Forward pass
    loss = criterion(pred, y)    # Compute loss
    loss.backward()              # Backward pass (compute gradients)
    optimizer.step()             # Update weights

The loop has four operations:

model(x) runs the forward pass. Data goes in, a prediction comes out. Internally, PyTorch builds a computation graph as it goes, recording every operation.

loss.backward() computes the gradients. PyTorch walks the computation graph backwards, applying the chain rule at each node, and stores the gradient for every parameter. When this call finishes, model.weight.grad and model.bias.grad contain the gradients.

optimizer.step() takes those gradients and applies the update rule: $\theta \leftarrow \theta - \eta \cdot \nabla L$.

optimizer.zero_grad() clears out the stored gradients so they don't accumulate from the previous step. PyTorch accumulates gradients by default, which is useful for gradient accumulation across micro-batches, but in a plain loop you need to zero them out exactly once per iteration. The conventional placement is at the top of the loop (as above); placing it after optimizer.step() is also valid as long as it happens before the next backward().

nn.MSELoss() averages squared errors by default. In the earlier single-example calculation, mean and sum agree. With this four-point batch, the mean produces gradients one-quarter the size of the sum, changing the effect of a given learning rate.

Stochastic gradient descent

Computing a gradient over a large dataset before every update is expensive. Minibatch training uses a subset of examples for each step.

Instead, you sample a small random batch (say 32 or 64 examples), compute the gradient on just that batch, and update. This is stochastic gradient descent (SGD). "Stochastic" just means random. If the batch is sampled properly, the mini-batch gradient is an unbiased estimate of the full-dataset gradient: it points in the right direction on average, even though any single batch can be noisy or even point the wrong way. In exchange for that noise, you get many more steps per second.

The landscape gets harder

The bowl-shaped loss surface above was a friendly example. In real neural networks, the landscape is much more complicated. Valleys are elongated. There are saddle points where the gradient is nearly zero but you're not at a minimum. Some directions are steep and others are almost flat.

Here's what vanilla gradient descent looks like on an elongated valley, $f(x, y) = x^2 + 10y^2$:

Common misconceptions

"Gradient = slope." Close, but not quite. A slope is a scalar (a single number). A gradient is a vector, with one component per dimension. In one dimension they're the same thing, which is where the confusion comes from. But in the multi-dimensional parameter spaces of neural networks, the gradient is a direction in that high-dimensional space, not just a steepness.

"Backpropagation is some special, mysterious algorithm." It applies the chain rule to a computation graph, reusing intermediate derivatives to compute parameter gradients efficiently. Andrej Karpathy's micrograd project implements reverse-mode differentiation in about 100 lines of Python, with a video walkthrough of its construction.

"You need deep calculus to understand this." The calculations here use derivatives, partial derivatives, and the chain rule. A derivative measures local sensitivity; a partial derivative holds other inputs fixed; the chain rule combines sensitivities through composed functions.

What's next

The elongated valley showed how one learning rate can produce oscillation in one direction and slow progress in another. Optimizers: momentum, Adam, and learning rate schedules compares ways to modify those updates.

Additional reading (and watching)

• Rumelhart, D., Hinton, G., & Williams, R. (1986). Learning representations by back-propagating errors. Nature, 323, 533-536. The paper that made backprop famous, though the algorithm had been independently rediscovered several times before that.
• Karpathy, A. (2020). micrograd. A scalar-valued autograd engine and tiny neural net library in about 100 lines of Python. The clearest possible read of how reverse-mode autodiff actually works.
• Nielsen, M. (2015). Neural Networks and Deep Learning, Chapter 2: How the backpropagation algorithm works. The most patient walk-through of the four backprop equations I know of.
• 3Blue1Brown. (2017). Backpropagation, intuitively and Backpropagation calculus. Neural Networks series, Chapters 3 and 4. The geometric story for the chain rule applied to a network.
• Karpathy, A. (2022). The spelled-out intro to neural networks and backpropagation: building micrograd. YouTube. A patient code-along that builds reverse-mode autodiff from nothing.
• Bottou, L., Curtis, F. E., & Nocedal, J. (2018). Optimization Methods for Large-Scale Machine Learning. SIAM Review, 60(2), 223-311. The careful treatment of why SGD with mini-batches works as well as it does, including the noise-as-regularizer view.
• Chen, T., et al. (2016). Training Deep Nets with Sublinear Memory Cost. arXiv:1604.06174. The original gradient checkpointing paper. Frames the activation-vs-recompute tradeoff that every modern LLM trainer relies on.