Cross-Entropy, KL Divergence, and What Loss Functions Measure

A PyTorch training step often includes these two lines:

loss = criterion(logits, targets)
loss.backward()

Cross-entropy is a standard next-token training loss. But what does it measure, and why use it instead of accuracy?

Accuracy counts how often the most likely token matches the observation. Cross-entropy also measures the probability assigned to that observation.

Why not just use accuracy?

If the model predicts the next token, and we know the actual next token, why not just score it as "right or wrong" and try to maximize the percentage of correct predictions?

Two problems. First, top-1 accuracy is constant while small logit changes leave the winning class unchanged, then jumps when the winner changes. Its gradient is zero almost everywhere and undefined at those boundaries, so it gives gradient descent no useful local direction.

Second, accuracy throws away information. Suppose the correct next token is "the." Model A assigns 51% probability to "the" and 49% to "a." Model B assigns 99% probability to "the." Accuracy says both models scored a perfect 1.0 on this example. Model B assigns higher probability to this observed token, so it gets lower log loss on this example.

Cross-entropy solves both of these problems. It's smooth, differentiable, and it cares about the probabilities the model assigns, not just whether the top prediction was correct.

Entropy: the floor

Before we get to cross-entropy, we need entropy itself. Shannon introduced it in his 1948 work on information theory.

Entropy measures the irreducible uncertainty in a probability distribution. How surprised should you be, on average, when you observe an outcome?

For each possible outcome, multiply its probability by the log of its probability, sum them up, flip the sign.

The slider varies the distribution's concentration. Drag concentration from uniform toward one-hot and watch the needle fall.

A fair coin has two outcomes, each with probability 0.5:

One bit of entropy. You need exactly one bit of information to describe the outcome of a fair coin flip.

A fair six-sided die:

A uniform distribution has more entropy when it has more outcomes. A loaded die that lands on 6 about 90% of the time has much lower entropy. You're rarely surprised.

Entropy is low when the distribution is peaked (one outcome dominates), and high when the distribution is spread out (many outcomes are likely). A perfectly uniform distribution over $n$ outcomes has entropy $\log_2 n$, which is the maximum possible.

Why does this matter for language? Because language has structure. The next word after "the United" is almost certainly "States" or "Kingdom" or "Nations," and the entropy of the true next-token distribution in that context is low. After "I want to", the entropy is much higher because there are lots of reasonable continuations. A good language model should internalize this: be confident when the context demands it, uncertain when it doesn't.

Cross-entropy: how surprised is the model?

Entropy averages surprise using the distribution's own probabilities. Cross-entropy averages the surprise assigned by model $q$ when outcomes follow distribution $p$:

The expectation uses $p_i$, the probabilities governing which outcomes occur, while the logarithm uses the model probabilities $q_i$.

When $q = p$, cross-entropy reaches its minimum, $H(p, q) = H(p)$. The uncertainty in $p$ remains even with a matching model.

When $q \neq p$, cross-entropy is higher. Minimizing it encourages the model distribution to match $p$.

Suppose the true distribution over the next word is $p = [0.7, 0.2, 0.1]$ for tokens ["the", "a", "one"]. And our model predicts $q = [0.5, 0.3, 0.2]$.

True entropy:

Cross-entropy:

The model is paying an extra ~0.12 bits of surprise because its distribution doesn't match reality. It's assigning too little probability to "the" and too much to the rarer words.

Keeping the target fixed, consider a second model distribution closer to it:

KL divergence: the gap

KL divergence is just the difference between cross-entropy and entropy:

Or, equivalently:

KL divergence is always $\geq 0$, and it equals zero if and only if $p = q$. This is called Gibbs' inequality, and it's what makes cross-entropy work as a loss function. Over unrestricted probability distributions, the minimum is at $q=p$. A finite model family may not be able to represent that distribution.

When you minimize cross-entropy, you're also minimizing KL divergence (because $H(p)$ is a constant that doesn't depend on the model). So minimizing cross-entropy, minimizing KL divergence, and maximizing log-likelihood are all the same optimization objective, just written in different ways.

KL divergence is not symmetric. $D_{\text{KL}}(p \| q) \neq D_{\text{KL}}(q \| p)$ in general. This means it's not a "distance" in the mathematical sense. The direction matters. $D_{\text{KL}}(p \| q)$ asks "how well does $q$ approximate $p$?" while $D_{\text{KL}}(q \| p)$ asks the reverse. Language model training uses the first form: how well does our model approximate reality.

The two directions weight errors differently. Compare the scores for the same candidate $q$:

Perplexity: making it interpretable

Cross-entropy in bits is hard to intuit. Is 3.2 bits good? Bad? Depends on the vocabulary size and the task. Perplexity converts cross-entropy into something more tangible:

That formula assumes cross-entropy is in bits (log base 2). If your loss is reported in nats (natural log), as PyTorch's CrossEntropyLoss does, the equivalent is $\text{Perplexity} = e^{\text{loss}}$. Either way, the answer is the same number; it's just a base change.

Perplexity is the number of equally likely choices that would produce the same average log loss on the observed tokens. A perplexity of 10 corresponds to a geometric-mean probability of 0.1 for those tokens. It does not measure how many tokens the model actually considers plausible.

Perplexity comparisons need the same evaluation text, tokenization, context policy, and scoring procedure. A lower value means better log loss under that evaluation; it does not establish broader model quality.

A sharp distribution can still have high perplexity if it confidently predicts the wrong tokens. Conversely, a model can have low average loss while making occasional costly mistakes. Perplexity averages those observed losses; it is neither error rate nor distribution width.

What actually happens during training

The model receives a sequence of tokens as context. For each position, it outputs a vector of logits, one number per vocabulary token. Those logits get turned into a probability distribution via softmax. That gives us $q$, the model's predicted distribution over the next token.

The dataset supplies one observed next token per position, not the full conditional distribution $p$. We encode that observation as a one-hot target. Averaging these sampled losses estimates cross-entropy under the data distribution; the underlying distribution can still assign probability to many continuations.

With a one-hot target, the cross-entropy formula collapses:

If the model assigns probability 0.9 to the correct token, the loss is $-\log_2(0.9) \approx 0.15$ bits. Small loss, the model is confident and correct. If it assigns probability 0.01 to the correct token, the loss is $-\log_2(0.01) \approx 6.64$ bits. Huge loss. The model was surprised by what actually happened.

When softmax and cross-entropy are combined, the gradient on each logit simplifies to:

For loss in nats, this is predicted probability minus target. Loss in bits adds a factor of $1/\ln 2$. For the softmax example above, $q = [0.603, 0.222, 0.135, 0.030, 0.011]$ with target class 0:

The gradient on the correct logit is negative (so the optimizer pushes that logit up); the gradient on every wrong logit is positive (so those logits get pushed down), in proportion to how much probability mass they were wrongly carrying. The exponential, the log, and the sum all cancel exactly when softmax and cross-entropy are paired.

Zoom out and the same per-token loss curve, averaged over millions of tokens, is what you watch drop during training:

Implementation

In PyTorch, cross-entropy loss takes raw logits (pre-softmax), not probabilities. It computes log probabilities internally in a numerically stable way. The log-sum-exp trick avoids the overflow and underflow that can happen if you softmax first and take the log separately.

import torch
import torch.nn as nn
 
# Raw logits from the model (batch=1, vocab_size=5)
# Same numbers as the SoftmaxExplorer above
logits = torch.tensor([[2.0, 1.0, 0.5, -1.0, -2.0]])
 
# True label: index 0 is the correct token ("the")
target = torch.tensor([0])
 
# PyTorch handles softmax + negative log-likelihood internally
criterion = nn.CrossEntropyLoss()
loss = criterion(logits, target)
 
print(f"Loss: {loss.item():.4f}")
# Loss: 0.5063
 
# Equivalent manual computation:
probs = torch.softmax(logits, dim=-1)
manual_loss = -torch.log(probs[0, target[0]])
print(f"Manual: {manual_loss.item():.4f}")
# Manual: 0.5063
 
# Note: PyTorch works in nats (natural log), not bits.
# To convert nats -> bits, divide by ln(2):
print(f"In bits: {loss.item() / torch.log(torch.tensor(2.0)).item():.4f}")
# In bits: 0.7304

In a real training loop the logits tensor has shape (batch_size * sequence_length, vocab_size), and the target tensor has shape (batch_size * sequence_length), so you're computing this loss across every position in every sequence in the batch, then averaging.

Common variants

Label smoothing. Instead of a strict one-hot target, mix in a small amount of uniform mass: target = $(1-\epsilon)\,\text{one-hot} + \epsilon/V$. The loss now penalizes the model for being too confident on the training token, which acts as a regularizer. Used in Inception-v3, and supported in PyTorch via CrossEntropyLoss(label_smoothing=...).

Class weighting. When some target classes are much rarer than others, you can pass per-class weights so the loss doesn't get dominated by the common classes. PyTorch supports this via the weight argument to CrossEntropyLoss.

Focal loss. Reshapes cross-entropy by multiplying it with $(1 - q_{\text{correct}})^\gamma$, which down-weights examples the model is already getting confidently right and concentrates learning on the hard ones. Originally motivated by extreme class imbalance in dense object detection.

Binary / multi-label cross-entropy. When labels are not mutually exclusive (a tag predictor, a multi-label classifier), you don't want softmax across all labels. You want a sigmoid per label and binary cross-entropy on each. PyTorch's BCEWithLogitsLoss is the numerically stable version.

For language modeling specifically, plain softmax cross-entropy with one-hot targets is still the dominant pretraining objective. The variants above mostly come up in classifier heads, fine-tuning recipes, and adjacent tasks.

Misconceptions

"Lower loss means better generation quality." Log loss is strictly proper: its expected value is minimized by reporting the true distribution. That does not guarantee calibration for a finite, imperfectly trained model, or helpful behavior after decoding. Loss evaluates prediction probabilities; generation quality also depends on the task and decoding policy.

"KL divergence is a distance metric." It is not symmetric and does not satisfy the triangle inequality. The two directions weight errors differently. Descriptions such as forward KL being mode-covering and reverse KL being mode-seeking refer to particular restricted approximation families; both have the same unrestricted optimum, $q=p$.

"Perplexity of 10 means the model is wrong 90% of the time." It means the geometric mean of the probabilities assigned to observed tokens is 0.1. Top-1 accuracy cannot be recovered from that number. A model can put nearly all its mass on the wrong token and receive very high perplexity despite its narrow distribution.

"Cross-entropy values are comparable across models." Only under a matched evaluation protocol, including text, tokenizer, context lengths, masking, and averaging. A model that tokenizes text into longer pieces can have higher per-token loss than one that splits into shorter pieces, because each token is carrying more information. When you see a perplexity number for a model, the tokenizer is part of the number. Across tokenizers, report a common unit such as bits per byte or bits per character on matched text.

What's next

The next post covers gradients and how machines learn, the calculus that turns "this loss is too high" into actual updates to the weight matrices we covered in the first post.

Additional reading (and watching)

• Shannon, C.E. (1948). A Mathematical Theory of Communication. Bell System Technical Journal, 27(3), 379-423. The founding paper for information theory and the definition of entropy as average surprise.
• Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning, Ch. 3.13. Formal equivalence between minimizing cross-entropy, minimizing KL divergence, and maximizing log-likelihood. Cover & Thomas, Elements of Information Theory (2006), Ch. 2, gives the information-theoretic foundations.
• Jurafsky, D. & Martin, J.H. Speech and Language Processing (3rd ed.), Ch. 3. Perplexity as the "effective branching factor" interpretation originates here.
• Murphy, K.P. Machine Learning: A Probabilistic Perspective (2012), Ch. 2. See also Voita, L. NLP Course: Language Modeling for an accessible treatment of forward vs. reverse KL.
• Kaplan, J. et al. (2020). Scaling Laws for Neural Language Models. arXiv:2001.08361. First large-scale empirical study of loss-vs-compute power laws.
• Hoffmann, J. et al. (2022). Training Compute-Optimal Large Language Models. The Chinchilla paper. Revised the token-parameter ratio and showed most frontier models up to that point were under-trained.
• Rafailov, R. et al. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. NeurIPS 2023. Recasts preference optimization as a cross-entropy-style loss directly on the policy.