Cross-Entropy, KL Divergence, and What Loss Functions Measure

Every training loop you've ever seen has a line that looks something like this:

loss = criterion(logits, targets)
loss.backward()

Two lines. You call a loss function, you backpropagate the result. And if you're training a language model, criterion is almost certainly cross-entropy loss. Every GPT, every LLaMA, every BERT pretraining run uses it.

So if cross-entropy is the loss function for language modeling, what exactly is it measuring? Like, what IS entropy? Why not just compute accuracy (percentage of correct predictions) and minimize that?

I operated with a vague mental model of this for a long time. "It measures how different two distributions are." Sure. But that's hand-wavy enough to be useless when you're trying to debug a training run or figure out why your perplexity is stuck at 45.

Let me try to build the intuition from scratch.

Why not just use accuracy?

This is worth addressing upfront, because it's the obvious question. If the model predicts the next token, and we know the actual next token, why not just score it as "right or wrong" and try to maximize the percentage of correct predictions?

Two problems. First, accuracy isn't differentiable; it's a step function. The model either got it right or it didn't, and there's no gradient to flow backwards. You can't do calculus on "right or wrong."

Second, accuracy throws away information. Suppose the correct next token is "the." Model A assigns 51% probability to "the" and 49% to "a." Model B assigns 99% probability to "the." Accuracy says both models scored a perfect 1.0 on this example. But Model B clearly has a better grip on the distribution, and we need a loss function that can see the difference.

Let me just show that. The viz below flips between the two situations. Accuracy doesn't move. Cross-entropy does.

Cross-entropy solves both of these problems. It's smooth, differentiable, and it cares about the probabilities the model assigns, not just whether the top prediction was correct.

Entropy: the floor

Before we get to cross-entropy, we need entropy itself. Shannon invented it in 1948, and honestly the core idea is simpler than it sounds.

Entropy measures the irreducible uncertainty in a probability distribution. How surprised should you be, on average, when you observe an outcome?

For each possible outcome, multiply its probability by the log of its probability, sum them up, flip the sign.

Play with the slider for a second before we do the coin-and-dice math. Drag concentration from uniform toward one-hot and watch the needle fall.

Some quick examples to build intuition. A fair coin has two outcomes, each with probability 0.5:

One bit of entropy. You need exactly one bit of information to describe the outcome of a fair coin flip. That checks out.

A fair six-sided die:

More outcomes means more uncertainty means more entropy. A loaded die that lands on 6 about 90% of the time has much lower entropy. You're rarely surprised.

The key idea: entropy is low when the distribution is peaked (one outcome dominates), and high when the distribution is spread out (many outcomes are likely). A perfectly uniform distribution over $n$ outcomes has entropy $\log_2 n$, which is the maximum possible.

Why does this matter for language? Because language has structure. The next word after "the United" is almost certainly "States" or "Kingdom" or "Nations," and the entropy of the true next-token distribution in that context is low. After "I want to", the entropy is much higher because there are lots of reasonable continuations. A good language model should internalize this: be confident when the context demands it, uncertain when it doesn't.

Cross-entropy: how surprised is the model?

Now we get to the thing we actually compute during training. Cross-entropy measures something slightly different from entropy. Entropy asks "how uncertain is the true distribution?" Cross-entropy asks "how surprised is model $q$ when reality follows distribution $p$?"

Notice the subtle difference from regular entropy. We still weight by the true probabilities $p_i$ (because reality determines which outcomes happen), but we take the log of the model's probabilities $q_i$ (because the model's predictions determine how surprised it is).

If the model perfectly matches reality ($q = p$), then cross-entropy equals entropy: $H(p, q) = H(p)$. That's the best you can do. You can't be less surprised than the inherent uncertainty of the data.

If the model is wrong ($q \neq p$), cross-entropy is always higher than entropy. The model is more surprised than it needs to be, and that extra surprise is what we're trying to eliminate during training.

Let me make this concrete. Say the true distribution over the next word is $p = [0.7, 0.2, 0.1]$ for tokens ["the", "a", "one"]. And our model predicts $q = [0.5, 0.3, 0.2]$.

True entropy:

Cross-entropy:

The model is paying an extra ~0.12 bits of surprise because its distribution doesn't match reality. It's assigning too little probability to "the" and too much to the rarer words.

Here's what that looks like visually. The component below computes all the metrics live:

Now what happens if the model improves? It studies more data, adjusts its weights, and gets closer to the true distribution:

That's what training does. In expectation, gradient updates push the red bars toward the blue bars; on any single step, minibatch noise and shared parameters can move some probabilities the wrong way before the average pulls things back.

KL divergence: the gap

KL divergence is going to show up constantly in the ML literature, especially once we get to RLHF and alignment, so it deserves its own section.

KL divergence is just the difference between cross-entropy and entropy:

Or, equivalently:

It answers a very specific question: how much extra surprise does model $q$ cause, beyond the irreducible uncertainty in $p$?

KL divergence has a beautiful property. It's always $\geq 0$, and it equals zero if and only if $p = q$. This is called Gibbs' inequality, and it's what makes cross-entropy work as a loss function. The minimum of the loss surface is exactly where the model distribution matches the true distribution.

When you minimize cross-entropy, you're also minimizing KL divergence (because $H(p)$ is a constant that doesn't depend on the model). So minimizing cross-entropy, minimizing KL divergence, and maximizing log-likelihood are all the same optimization objective, just written in different ways.

One important gotcha: KL divergence is not symmetric. $D_{\text{KL}}(p \| q) \neq D_{\text{KL}}(q \| p)$ in general. This means it's not a "distance" in the mathematical sense. The direction matters. $D_{\text{KL}}(p \| q)$ asks "how well does $q$ approximate $p$?" while $D_{\text{KL}}(q \| p)$ asks the reverse. Language model training uses the first form: how well does our model approximate reality.

The asymmetry isn't a math curiosity. It changes what kind of model you end up with. Watch the same $q$ get scored both ways:

Perplexity: making it interpretable

Cross-entropy in bits is hard to intuit. Is 3.2 bits good? Bad? Depends on the vocabulary size and the task. Perplexity converts cross-entropy into something more tangible:

That formula assumes cross-entropy is in bits (log base 2). If your loss is reported in nats (natural log), as PyTorch's CrossEntropyLoss does, the equivalent is $\text{Perplexity} = e^{\text{loss}}$. Either way, the answer is the same number; it's just a base change.

Perplexity is the effective number of choices the model is deciding between. If a model has a perplexity of 10 on English text, it's as uncertain, on average, as if it were uniformly choosing between 10 equally likely next tokens at each position.

A perfect model that fully captured English text statistics might have a perplexity in the low single digits for many contexts. GPT-2 (1.5B) reported a perplexity of about 17.5 on WikiText-103 zero-shot, which is roughly 4.13 bits per token. Numbers like that are tied to a specific benchmark, tokenizer, and evaluation setup, and the comparison falls apart the moment any of those change. Frontier models in 2026 don't always publish clean perplexity numbers in apples-to-apples form; loss curves are usually reported in bits-per-byte or against a fixed evaluation harness. Lower perplexity means less confusion on that benchmark, not a universally better model.

Keep one thing in mind, though: perplexity is an average over positions. Some positions are easy (the word "States" after "United") and some are genuinely hard (the first word of a new paragraph). A model with perplexity 10 isn't wrong 90% of the time; it means that on average the model's probability distribution is as wide as a uniform distribution over 10 options. At easy positions it might be choosing between 2 options, at hard positions maybe 50.

What actually happens during training

Let's zoom into what happens at a single training step, because this is where cross-entropy goes from theory to practice.

The model receives a sequence of tokens as context. For each position, it outputs a vector of logits, one number per vocabulary token. Those logits get turned into a probability distribution via softmax. That gives us $q$, the model's predicted distribution over the next token.

There's a subtle thing happening here. The "true distribution" $p$ in the conceptual story above was something like $[0.7, 0.2, 0.1]$, a real conditional distribution over plausible next tokens. But the dataset doesn't hand us that distribution. It hands us one sampled token: the actual next word in this particular sentence. We represent that single observation as a one-hot vector and treat that as $p$ for this training example. Over many examples drawn from the same context distribution, the average loss we get is an empirical estimate of the cross-entropy between the true conditional distribution and the model's distribution. The one-hot target is a per-example sample of the underlying distribution, not a claim that the underlying distribution is one-hot.

With a one-hot target, the cross-entropy formula collapses:

That's it. Cross-entropy with a one-hot target is just the negative log probability of the correct token. $-\log q_{\text{correct}}$.

If the model assigns probability 0.9 to the correct token, the loss is $-\log_2(0.9) \approx 0.15$ bits. Small loss, the model is confident and correct. If it assigns probability 0.01 to the correct token, the loss is $-\log_2(0.01) \approx 6.64$ bits. Huge loss. The model was surprised by what actually happened.

Notice the shape of that curve. If the model assigns probability 1.0 to the correct token, loss is zero. Perfect. If it assigns probability approaching 0, loss goes to infinity. The log function provides this natural asymmetry for free.

There's one more piece I want to pull out here, because it makes the whole training story click. When softmax and cross-entropy are stacked together, the gradient on each logit is surprisingly clean:

Predicted probability minus target. For the softmax example above, $q = [0.603, 0.222, 0.135, 0.030, 0.011]$ with target class 0:

The gradient on the correct logit is negative (so the optimizer pushes that logit up); the gradient on every wrong logit is positive (so those logits get pushed down), in proportion to how much probability mass they were wrongly carrying. The exponential, the log, and the sum all cancel exactly when softmax and cross-entropy are paired. This is one of the deeper reasons that pairing is everywhere in deep learning.

Zoom out and the same per-token loss curve, averaged over millions of tokens, is what you watch drop during training:

Implementation: it's four lines

In PyTorch, cross-entropy loss takes raw logits (pre-softmax), not probabilities. It applies softmax internally for numerical stability. The log-sum-exp trick avoids the overflow and underflow that can happen if you softmax first and take the log separately.

import torch
import torch.nn as nn
 
# Raw logits from the model (batch=1, vocab_size=5)
# Same numbers as the SoftmaxExplorer above
logits = torch.tensor([[2.0, 1.0, 0.5, -1.0, -2.0]])
 
# True label: index 0 is the correct token ("the")
target = torch.tensor([0])
 
# PyTorch handles softmax + negative log-likelihood internally
criterion = nn.CrossEntropyLoss()
loss = criterion(logits, target)
 
print(f"Loss: {loss.item():.4f}")
# Loss: 0.5063
 
# Equivalent manual computation:
probs = torch.softmax(logits, dim=-1)
manual_loss = -torch.log(probs[0, target[0]])
print(f"Manual: {manual_loss.item():.4f}")
# Manual: 0.5063
 
# Note: PyTorch works in nats (natural log), not bits.
# To convert nats -> bits, divide by ln(2):
print(f"In bits: {loss.item() / torch.log(torch.tensor(2.0)).item():.4f}")
# In bits: 0.7304

In a real training loop the logits tensor has shape (batch_size * sequence_length, vocab_size), and the target tensor has shape (batch_size * sequence_length), so you're computing this loss across every position in every sequence in the batch, then averaging.

Common variants

Cross-entropy as written above is the baseline. A few variants show up often enough that you'll see them in real code.

Label smoothing. Instead of a strict one-hot target, mix in a small amount of uniform mass: target = $(1-\epsilon)\,\text{one-hot} + \epsilon/V$. The loss now penalizes the model for being too confident on the training token, which acts as a regularizer. Originally introduced for Inception-v3, and supported in PyTorch via CrossEntropyLoss(label_smoothing=...).

Class weighting. When some target classes are much rarer than others, you can pass per-class weights so the loss doesn't get dominated by the common classes. PyTorch supports this via the weight argument to CrossEntropyLoss.

Focal loss. Reshapes cross-entropy by multiplying it with $(1 - q_{\text{correct}})^\gamma$, which down-weights examples the model is already getting confidently right and concentrates learning on the hard ones. Originally motivated by extreme class imbalance in dense object detection.

Binary / multi-label cross-entropy. When labels are independent rather than mutually exclusive (a tag predictor, a multi-label classifier), you don't want softmax across all labels. You want a sigmoid per label and binary cross-entropy on each. PyTorch's BCEWithLogitsLoss is the numerically stable version.

For language modeling specifically, plain softmax cross-entropy with one-hot targets is still the dominant pretraining objective. The variants above mostly come up in classifier heads, fine-tuning recipes, and adjacent tasks.

Misconceptions

"Lower loss means better generation quality." Cross-entropy measures how well the model's distribution matches the training data distribution. Log loss is what statisticians call a strictly proper scoring rule, which is a fancy way of saying it's only minimized when the model's reported probabilities actually match the true conditional probabilities. So a well-trained model has well-calibrated next-token distributions, in expectation. But calibrated next-token probabilities and good generation aren't the same thing. Generation involves sampling from that distribution, decoding strategies, and a system prompt that shapes the conditional. A model with great pretraining loss can still produce repetitive or unhelpful long-form text. This is part of why the field moved toward RLHF and DPO for alignment, which we'll cover in Arc 7. Loss measures distribution fit; quality is what you get after a decoding policy and an alignment objective have had their say.

"KL divergence is a distance metric." It's not. A distance metric has to be symmetric, so the distance from A to B equals the distance from B to A. KL divergence doesn't satisfy that. $D_{\text{KL}}(p \| q)$ can be wildly different from $D_{\text{KL}}(q \| p)$, and this isn't just a technicality. The two directions have different practical behavior. Forward KL ($D_{\text{KL}}(p \| q)$, what we use in pretraining) tends to produce models that are mean-seeking: they try to cover all the modes of the true distribution. Reverse KL tends to produce models that are mode-seeking: they collapse to a single mode. This distinction becomes critical when we get to variational inference and policy optimization.

"Perplexity of 10 means the model is wrong 90% of the time." No. Perplexity is about the width of the model's probability distribution, not its accuracy. A perplexity of 10 means the model's average uncertainty is equivalent to choosing uniformly from 10 options. It might be getting the top-1 prediction correct 60% of the time while spreading the remaining probability mass across 9 alternatives. Perplexity and accuracy are correlated but not interchangeable.

"Cross-entropy values are comparable across models." Only if the tokenizers are the same. A model that tokenizes text into longer pieces will naturally have higher per-token loss than one that splits into shorter pieces, because each token is carrying more information. When you see a perplexity number for a model, the tokenizer is part of the number. Comparing perplexities across models with different vocabularies is apples and oranges unless someone normalizes by bits-per-byte or bits-per-character.

What's next

Cross-entropy is a simple idea with a lot of baggage. Take the true distribution, take the model's distribution, ask how surprised the model is by reality, minimize that surprise. The model predicts a distribution, reality reveals an answer, cross-entropy measures the gap, and gradients close it.

The next post covers gradients and how machines learn, the calculus that turns "this loss is too high" into actual updates to the weight matrices we covered in the first post.

Additional reading (and watching)

• Shannon, C.E. (1948). A Mathematical Theory of Communication. Bell System Technical Journal, 27(3), 379-423. The founding paper for information theory and the definition of entropy as average surprise.
• Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning, Ch. 3.13. Formal equivalence between minimizing cross-entropy, minimizing KL divergence, and maximizing log-likelihood. Cover & Thomas, Elements of Information Theory (2006), Ch. 2, gives the information-theoretic foundations.
• Jurafsky, D. & Martin, J.H. Speech and Language Processing (3rd ed.), Ch. 3. Perplexity as the "effective branching factor" interpretation originates here.
• Murphy, K.P. Machine Learning: A Probabilistic Perspective (2012), Ch. 2. See also Voita, L. NLP Course: Language Modeling for an accessible treatment of forward vs. reverse KL.
• Kaplan, J. et al. (2020). Scaling Laws for Neural Language Models. arXiv:2001.08361. First large-scale empirical study of loss-vs-compute power laws.
• Hoffmann, J. et al. (2022). Training Compute-Optimal Large Language Models. The Chinchilla paper. Revised the token-parameter ratio and showed most frontier models up to that point were under-trained.
• Rafailov, R. et al. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. NeurIPS 2023. Recasts preference optimization as a cross-entropy-style loss directly on the policy.