Language Modeling as Next-Token Prediction

Welcome to Arc 2. In Arc 1 we built up the mathematical tools: vectors, matrices, probability distributions, cross-entropy, gradients, optimizers, floating point. Now we get to use them. This arc covers how people modeled language before transformers, because every design choice in modern LLMs is a reaction to something that came before. If you skipped Arc 1, you can probably still follow along, but the cross-entropy post and the probability distributions post are the most directly relevant prerequisites.

So. What is a language model?

A language model is a system that assigns a probability to the next token, given all the tokens that came before it. Given a sequence of words (or subwords, or bytes), it outputs a probability distribution over what comes next.

Every GPT, every LLaMA, every Claude, every n-gram model from the 1990s does some version of this. The architectures are wildly different, but the objective is the same: predict the next token from context. (BERT and its descendants use a related but distinct objective, masked language modeling, where you predict held-out tokens from their bidirectional surroundings rather than predicting strictly left-to-right. We'll come back to that in Arc 4.)

That objective is doing more work than it looks like.

The Chain Rule: Why This Works

Why does predicting the next token let you model entire sequences? Because of a fact from probability theory that we covered in Post 1.3: the chain rule.

Any joint probability over a sequence of tokens can be decomposed into a product of conditionals:

Or more compactly:

This is an identity, not an approximation. It's true for any distribution over sequences, no matter how complex. And it says something important: if you can model each conditional $P(w_t \mid w_{<t})$ accurately, you can reconstruct the full joint distribution over any sequence. The chain rule guarantees that a next-token predictor is, in principle, sufficient to capture any pattern in language.

Here's what that factorization looks like in practice, for the first few words of a sentence:

The task "predict the next word" sounds like autocomplete, like the suggestion bar on your phone. The chain rule tells us it's actually a universal density estimator for sequences. If your model is expressive enough and you train it on enough data, minimizing the next-token prediction loss should in theory learn every statistical regularity in the training distribution. Syntax, semantics, world knowledge, reasoning patterns. All of it shows up as patterns in what comes next.

Shannon's Guessing Game

Claude Shannon understood this in 1948. His paper "A Mathematical Theory of Communication" is where entropy, cross-entropy, and most of information theory come from. But he also played a revealing game. In a 1951 follow-up paper, he had human subjects try to guess the next letter in English text, one character at a time.

People could predict the next letter with high accuracy, especially after seeing a few characters of context. After "the United Sta", the next letter is almost certainly "t". After a space following a period, it's probably a capital letter. After "q", it's almost certainly "u".

Shannon used these experiments to estimate the entropy of English at roughly 0.6 to 1.3 bits per character. If you already have enough context, the average English character carries only about 1 bit of genuine surprise. The rest is redundant, predictable from what came before.

That redundancy is what language models exploit. A perfect language model would compress English text to about 1 bit per character. A bad one wastes bits on things it should have predicted. The gap between the model's predictions and reality is exactly what cross-entropy measures, as we covered in Post 1.4.

Here's an animated version of Shannon's game, one letter at a time. Watch the distribution get peakier (and the "bits this step" number fall) as the context grows. After "the q" the next letter is essentially forced, so the entropy drops to near zero. After a space, we're back near 3-4 bits, because any word could start next. The running average settles in the neighborhood Shannon estimated from his human subjects.

If you want to feel what "bits of entropy" actually mean in the abstract, here's the distribution-to-entropy gauge from Arc 1 again. Drag the concentration slider (or press play and watch it breathe): a uniform distribution over 5 outcomes has $\log_2 5 \approx 2.32$ bits of entropy, and a one-hot distribution has zero. Everything a language model does is a push along that axis, from "I have no idea what comes next" toward "I know exactly which token comes next."

Perplexity: Making the Numbers Interpretable

We introduced perplexity briefly in the cross-entropy post, but let's revisit it. It's been the standard evaluation metric for language models since the 1970s IBM speech-recognition days, and the intuition is cleaner than cross-entropy.

Perplexity is just the exponentiated cross-entropy:

where $H(p, q)$ is the cross-entropy of the model $q$ with respect to the true data distribution $p$, measured on some held-out test set. In practice, since we compute cross-entropy using natural log in PyTorch, you'll often see:

where "loss" is the average cross-entropy loss per token.

The intuition is that perplexity is the effective number of tokens the model is choosing between at each step. If a model has perplexity 10 on English text, it's as uncertain, on average, as if it were uniformly guessing among 10 equally likely next tokens.

Slide the cross-entropy below to see where different regimes land:

Some rough landmarks to calibrate your intuition:

A uniform random model over a 50,000-word vocabulary has perplexity 50,000. It has no idea what comes next. A model with perplexity 30 has narrowed it down to "about 30 plausible options" at each position. That's a lot of learned structure. Perplexity for the latest closed models isn't really a published number anymore; the concept still matters, but the comparable numbers stop somewhere around the open-weight LMs you can actually run a held-out evaluation on.

A Worked Example

Let me make this concrete. Consider the sentence:

"the cat sat on the mat"

Let's compute the probability of this sentence under three different models: a unigram model, a bigram model, and a hypothetical "good" model. Then we'll compare their perplexities.

Unigram model. A unigram model ignores all context. It just knows how frequent each word is. Let's say in our training corpus:

The unigram probability of the sentence is:

Per-token cross-entropy (using log base 2):

Perplexity: $2^{7.107} \approx 137.5$

Bigram model. A bigram model conditions on the previous word. With reasonable bigram probabilities from a corpus:

Per-token cross-entropy:

Perplexity: $2^{3.917} \approx 15.1$

The bigram model's perplexity dropped from 137 to 15 just by looking at the previous word. That single word of context made a huge difference. "sat" after "cat" is much more likely than "sat" in isolation.

A "good" model. A stronger language model can assign higher probabilities because it uses the full context, not just the previous token:

Per-token cross-entropy:

Perplexity: $2^{2.240} \approx 4.7$

Three models, same sentence, wildly different numbers. The viz below stacks them together. The blue bars show the probability each model assigns to the correct next token at every position, and the red bars show the per-token surprise ($-\log_2 P$). Watch the red bars shrink and flatten as the model improves. That shrinkage is where the perplexity drop comes from.

Here's what those probability assignments look like side by side at a single position. The blue bars show how a "good" model distributes probability at position 5 (predicting "the" after "cat sat on"), and the red bars show what a bigram model does with only the previous word "on":

The good model assigns 60% to "the" because it has seen the full phrase "the cat sat on" and knows what typically follows. The bigram model can only see "on" and has to guess more broadly. That difference in confidence is exactly what perplexity captures.

Implementation: A Minimal Bigram LM

Here's a complete bigram language model with perplexity evaluation. This is the simplest possible language model that actually conditions on context.

import math
from collections import defaultdict
 
class BigramLM:
    def __init__(self):
        self.counts = defaultdict(lambda: defaultdict(int))
        self.totals = defaultdict(int)
        self.vocab = set()
    
    def train(self, sentences):
        """Train on a list of tokenized sentences."""
        for tokens in sentences:
            tokens = ["<s>"] + tokens + ["</s>"]
            for i in range(1, len(tokens)):
                self.counts[tokens[i-1]][tokens[i]] += 1
                self.totals[tokens[i-1]] += 1
                self.vocab.add(tokens[i])
    
    def prob(self, word, context):
        """P(word | context) with add-1 smoothing."""
        V = len(self.vocab)
        return (self.counts[context][word] + 1) / (self.totals[context] + V)
    
    def perplexity(self, sentences):
        """Compute perplexity on held-out data."""
        log_prob_sum = 0
        token_count = 0
        
        for tokens in sentences:
            tokens = ["<s>"] + tokens + ["</s>"]
            for i in range(1, len(tokens)):
                p = self.prob(tokens[i], tokens[i-1])
                log_prob_sum += math.log2(p)
                token_count += 1
        
        avg_cross_entropy = -log_prob_sum / token_count
        return 2 ** avg_cross_entropy
 
# Train on a small corpus
corpus = [
    "the cat sat on the mat".split(),
    "the cat ate the fish".split(),
    "the dog sat on the rug".split(),
    "a dog chased the cat".split(),
    "the cat slept on a mat".split(),
]
 
lm = BigramLM()
lm.train(corpus)
 
# Test probabilities
print(f"P(cat | the) = {lm.prob('cat', 'the'):.3f}")
print(f"P(sat | cat) = {lm.prob('sat', 'cat'):.3f}")
print(f"P(on  | sat) = {lm.prob('on', 'sat'):.3f}")
 
# Evaluate perplexity on held-out sentence
test = ["the cat sat on the rug".split()]
print(f"\nPerplexity on 'the cat sat on the rug': {lm.perplexity(test):.1f}")

The whole thing fits on a screen. Training is just counting pairs, and evaluation just looks up conditional probabilities, averages the log-probs, and exponentiates. The +1 in the numerator is add-one (Laplace) smoothing, the crudest way to handle unseen bigrams. We'll see much better smoothing in the next post.

Why Not Just Use Accuracy?

Why train language models with cross-entropy loss on the predicted distribution, rather than just scoring accuracy ("did the model guess the right word")?

Two reasons. First, accuracy is not differentiable. It's a step function: right or wrong, 1 or 0. No gradient to flow backward, so you can't train a neural network with it.

Second, accuracy throws away information about confidence. If the true next word is "mat" and Model A assigns it 51% while Model B assigns it 99%, accuracy scores both as perfect. Cross-entropy sees the difference. It rewards high confidence in the correct answer and heavily penalizes confident wrong answers (because $-\log(0.01) \gg -\log(0.5)$). That makes it a much richer training signal.

The viz below morphs between the two models. Both are "correct" under an accuracy metric. The barely-winning distribution carries over 1 bit of loss per token; the confident one is near zero. That gap is the gradient signal that makes training possible.

The Sufficiency Argument

One more point about next-token prediction, because it's easy to understate.

The chain rule decomposition above means that any probability distribution over sequences of tokens can be exactly represented as a product of next-token conditionals. No loss of generality. You're not approximating anything by decomposing a sequence distribution into per-position predictions.

So the task "predict the next token as well as possible" is sufficient to learn everything learnable about the distribution. Syntax, grammar, facts about the world, reasoning patterns, stylistic conventions, code structure. If any of these regularities help predict what token comes next in a large enough corpus, a sufficiently powerful model trained on next-token prediction will learn to exploit them.

This is why GPT-style models can do so many things without being explicitly trained for any of them. Translation? It helps predict what comes after "Translate the following French to English:". Arithmetic? It helps predict what comes after "What is 347 + 829? The answer is". Code completion? Same principle. The diversity of capabilities emerges because the training data contains diverse patterns, and next-token prediction incentivizes the model to internalize all of them.

Here's that idea made literal. Same objective, same model, different context prefixes. Watch the peak of the next-token distribution land on the "right answer" for each task, no fine-tuning required.

Of course, "sufficiently powerful" and "large enough corpus" are doing a lot of work in those sentences. The objective is not nearly as narrow as "autocomplete" makes it sound: in principle, next-token prediction can represent any distribution over sequences. As a sequence-modeling objective, it is far more general than it first appears. In practice, the bottlenecks move to data, capacity, optimization, architecture, context, and post-training. Universality of the decomposition is not the same as sufficiency for robust reasoning, truthfulness, agency, alignment, or out-of-distribution generalization. Those are real constraints that the rest of the series will keep coming back to.

Misconceptions

"Next-token prediction is just autocomplete." Your phone's autocomplete uses a small n-gram or shallow neural model that predicts the most likely next word. GPT-4 is also predicting the next token. Same objective in both cases. What differs is model expressiveness and training data scale. At sufficient scale, "autocomplete" becomes a system that has internalized enough statistical structure of language to do translation, reasoning, and code generation.

"Perplexity tells you how good a model is at real tasks." Perplexity measures how well the model's distribution matches the test data distribution. That's correlated with downstream task performance, but the correlation is noisy. A model can have excellent perplexity and still generate bland, repetitive text. A model with slightly worse perplexity might generate more interesting and useful responses after RLHF fine-tuning. Perplexity is the canonical metric for evaluating next-token modeling quality on a held-out text distribution. It's a useful proxy for base-model quality, but not a complete measure of general capability, which also depends on data mix, reasoning robustness, tool use, instruction following, calibration, code and math benchmarks, and post-training behavior. We'll dig into this more in Arc 8 when we cover evaluation.

"You need a different objective for each NLP task." This was the standard assumption before the GPT era. If you wanted a model for sentiment analysis, you trained a sentiment classifier. If you wanted translation, you trained a seq2seq model. GPT-2 and then GPT-3 showed that a single model trained on next-token prediction could do many tasks via prompting, without task-specific training. The field moved from "one objective per task" to "one objective, many tasks." That shift flows directly from the sufficiency of next-token prediction.

What's next

The next post covers n-gram models and the curse of sparsity, what happens when you try to do next-token prediction with nothing but counting, why the count table is almost entirely zeros no matter how much data you throw at it, and the smoothing tricks that kept statistical NLP alive until neural models arrived.

Additional reading (and watching)

• Shannon, C.E. (1948). A Mathematical Theory of Communication. Bell System Technical Journal, 27(3), 379-423. The foundational paper for information theory, entropy, and the mathematical framework for thinking about language as a probabilistic source.
• Shannon, C.E. (1951). Prediction and Entropy of Printed English. Bell System Technical Journal, 30(1), 50-64. Shannon's human guessing-game experiments that estimated the entropy of English at ~1.0-1.3 bits per character.
• Kaplan, J., et al. (2020). Scaling Laws for Neural Language Models. Showed that loss scales as a power law with model size, dataset size, and compute, with predictable returns.
• Hoffmann, J., et al. (2022). Training Compute-Optimal Large Language Models. The "Chinchilla" paper. Showed that previous models were significantly undertrained relative to their size, and that data and parameters should scale together.
• Wei, J., et al. (2022). Emergent Abilities of Large Language Models. Documents capabilities that appear suddenly at certain model scales, though see Schaeffer et al. (2023), "Are Emergent Abilities of Large Language Models a Mirage?" for a counter-argument.
• Radford, A., et al. (2019). Language Models are Unsupervised Multitask Learners. The GPT-2 paper. Showed that a single next-token-prediction model could do reading comprehension, translation, and summarization zero-shot.
• Brown, T., et al. (2020). Language Models are Few-Shot Learners. The GPT-3 paper. Demonstrated that in-context learning from next-token prediction scaled smoothly with model size across a wide range of benchmarks.
• Jelinek, F., Mercer, R. L., Bahl, L. R., & Baker, J. K. (1977). Perplexity — a measure of the difficulty of speech recognition tasks. JASA. The original introduction of perplexity as an evaluation metric for language models in the speech recognition community.