Scaling Laws: What They Predict and Where They Bend

For a few years around 2020, one of the more striking results in machine learning was hiding in a single figure. Plot the test loss of a language model against the compute you spent training it, both on log scales, and you get a straight line. Over seven orders of magnitude, across different model sizes, different dataset sizes, different hyperparameters, everything collapses onto the same downward line.

A straight line on log-log is a power law. Bigger models, more data, more compute: loss keeps falling, and it falls in a way that's predictable from the numbers you already have. You could look at the slope on a small run and project where a run a thousand times larger would land. In a field that had spent decades arguing about which architecture to use, scaling laws said something fairly blunt: most of what matters is how big you go.

This post is about those laws. What they actually predict, what they don't, why Kaplan's 2020 paper turned out to be wrong in a specific and consequential way, why Llama 3 then trained "past" the corrected optimum on purpose, and what's happening at the frontier where the straight line starts to bend.

The setup: what is a scaling law?

A scaling law, in the sense that matters here, is a fit. You train a lot of language models at a lot of scales. You record, for each one, three numbers: the number of parameters $N$, the number of training tokens $D$, and the final test loss $L$ (usually in nats per token, which is just cross-entropy loss). You plot $L$ against whatever you think is driving it. If the plot is a straight line on log-log, you fit a power law:

where $x$ is $N$, $D$, or compute $C$; $\alpha$ is the slope on the log-log plot; $A$ is a constant; and $E$ is the irreducible loss, the floor set by the entropy of language itself. No model does better than $E$, because $E$ is the information content of the distribution. You can't predict what's genuinely random.

The Kaplan et al. 2020 paper was the first big careful fit of this form across modern transformers, and it made two claims that shaped the field. First, the power law is real and striking: across more than seven orders of magnitude of compute, loss falls in a line, with no sign of diminishing returns past some visible horizon. Second, when you're given a fixed compute budget, the best way to spend it is to make $N$ bigger much faster than you make $D$ bigger. Scale the model aggressively; the data will come along for the ride.

That second claim is where it gets interesting.

Compute, parameters, data, and the 6ND approximation

To talk about compute-optimal allocation, we need a compute model. The one everyone uses is disarmingly simple:

Training compute (in FLOPs) is roughly six times the number of parameters times the number of tokens seen. The six is a rough constant that comes from counting the forward pass (2 FLOPs per parameter per token), the backward pass (another 4), and a few accounting details. For dense transformers in the regime we care about, this is accurate to within about 20%, which is good enough for allocation arguments.

So compute, parameters, and data are linked by one equation. If you fix $C$, you get to trade off $N$ against $D$. You can train a small model on a lot of data, or a huge model on a little data, or something in between. The scaling law question is: which tradeoff is best?

Kaplan's answer, fit on models up to about 1.5B parameters, was that you should spend most of an increase in budget on $N$. If you double your compute, make the model roughly $1.7\times$ bigger and feed it only $1.2\times$ more data. The intuition was that parameters buy more loss-reduction-per-FLOP than tokens do.

Two years later, the DeepMind team behind the Chinchilla paper showed that Kaplan's allocation was wrong, in a way that had already shaped real decisions at frontier labs.

Chinchilla and the D/N ≈ 20 rule

Hoffmann et al. did a much larger, much more careful sweep. They trained more than 400 models across a wide range of $(N, D)$ pairs, and fit a joint loss surface:

Two separate power-law terms, one for parameters, one for data, over a shared irreducible floor. The fit they got had $\alpha \approx 0.34$ and $\beta \approx 0.28$, close enough to each other that the compute-optimal allocation works out to something almost embarrassingly clean: parameters and tokens should grow at roughly the same rate. Specifically, at the compute-optimal point, $D \approx 20 \cdot N$. Twenty tokens of training data per model parameter.

That is very different from Kaplan. For the Chinchilla team's own test, they took the compute budget that had been used to train the 280-billion-parameter Gopher model and instead trained a 70-billion-parameter model on four times as much data. Same compute. Chinchilla (70B, 1.4T tokens) beat Gopher (280B, 300B tokens) on almost every benchmark they tested. The smaller, longer-trained model was better.

In 2020, GPT-3 was trained on about 300B tokens with 175B parameters. The Kaplan recipe said that was roughly right. The Chinchilla recipe says GPT-3 was badly undertrained: to hit its compute-optimal allocation it should have used about $3.5{\times}10^{12}$ tokens, more than ten times what it actually saw. The model was too big for the data it had.

At small scale the two prescriptions overlap, which is why Kaplan's sweep (done mostly below 1B parameters) didn't catch the issue. It's only as you scale up that the divergence becomes dramatic. By frontier scale, Kaplan's recipe picks a model five or ten times bigger than Chinchilla's for the same compute, and it performs worse.

So what went wrong in Kaplan? Two things, mostly. The sweep used a fixed learning-rate schedule across all scales instead of retuning, which happened to hurt the smaller-model / longer-run corner disproportionately. And the sweep just didn't go high enough in $D$ to see the tradeoff flip. The shape of Kaplan's picture is still right. The allocation within the shape was wrong.

A worked example

Let's ground this in numbers. Say I have a compute budget of $C = 5.76 \times 10^{23}$ FLOPs. That's the budget DeepMind used for the original Chinchilla run, so we know what answer to expect.

Chinchilla's rule of thumb gives:

About 69 billion parameters. Then $D^* = 20 \cdot N^* \approx 1.4 \times 10^{12}$, or 1.4 trillion tokens. Those are precisely the numbers they used: a 70B model on 1.4T tokens. The math above is a one-line derivation: $C = 6ND$ and $D = 20N$ implies $N = \sqrt{C/120}$.

Try it on a different budget. The interactive below does this for you, and includes preset chips that jump to historical runs so you can see how far off (or on) they were.

Click "GPT-3 (2020)". The calculator says GPT-3's compute budget ($3.1{\times}10^{23}$ FLOPs) was about right for a 51B-parameter model trained on ~1T tokens. GPT-3 was actually 175B trained on 300B tokens, too big for its data by the Chinchilla accounting.

Now click "Llama-3-70B". The calculator says $C = 6.3{\times}10^{24}$ FLOPs prescribes about 229B params on 4.6T tokens. Meta actually trained Llama 3 at 70B params on 15T tokens. The deviation runs the other direction: the model is smaller than Chinchilla would pick, and the token count is much larger. That's not a mistake, and it brings us to the next idea.

Over-training: why Chinchilla isn't the goal anymore

Chinchilla answers one question: given a fixed training compute budget, what $(N, D)$ minimizes training loss? That's the right question if training compute is the only cost you care about. But for a production model, it almost never is.

Once a model ships, every forward pass through it costs inference compute. A model that serves a trillion tokens over its lifetime is going to spend far more FLOPs on inference than it spent on training. And inference FLOPs scale with $N$, not with $D$. Training cost is $6ND$. Inference cost over $V$ served tokens is about $2NV$. Those two numbers have very different shapes.

If you care about total lifetime compute (training plus inference), the optimum moves. Concretely: take the Chinchilla-optimal point. Now shrink $N$ by some factor, and pump $D$ up just enough to keep the final loss the same. Training cost goes up (you're training for longer) but inference cost goes down (the smaller model is cheaper every time it runs). If your expected inference volume is large, you come out ahead.

This is exactly the Llama 3 story. Meta trained the 70B model on 15T tokens, which is about $10\times$ past the Chinchilla-optimal ratio of $D/N = 20$ (the D/N ratio comes out to ~214, vs Chinchilla's 20). They called it "over-training," and they were explicit about why in the technical report: the model is going to serve billions of users, and a smaller-cheaper-slightly-more-expensive-to-train model wins on total cost of ownership. The trend across 2024-2026 has been for open-weight releases to push this ratio further. Llama-3-8B was trained on 15T tokens too, which is a $D/N$ of 1875. Chinchilla would have picked something closer to $D/N = 20$ for that compute, which is a factor-of-90 difference.

So "Chinchilla-optimal" is a benchmark, not a goal. It minimizes one specific cost. The cost you actually care about depends on what you're going to do with the model.

The minimal math, in one block

A lot of the rest of the literature is about fitting this joint loss surface more carefully or extending it to new regimes. The shape to carry around is:

Two power-law error terms stacked on an irreducible floor. The first falls as you make the model bigger. The second falls as you feed it more data. They don't interact; they just add. And because they add, there's a well-defined optimum for any compute budget: you want both error terms to be small, and for fixed $C = 6ND$ the sum is minimized at a particular $(N, D)$ ratio. For Hoffmann's fit that ratio came out to $D/N \approx 20$.

You can derive the rule of thumb yourself by plugging $D = C/(6N)$ into the loss and solving $\partial L / \partial N = 0$. The algebra is ugly; the conclusion is clean. If $\alpha \approx \beta$, the optimum sits at roughly equal contributions from the two error terms, which (with Hoffmann's coefficients) puts $D/N \approx 20$.

Implementation sketch

The math is simple enough that the calculator above is basically three lines. Here's the same thing in Python, with the Llama-3-70B comparison made explicit. The numbers line up with the worked example:

import math
 
def chinchilla_optimal(C: float, tokens_per_param: float = 20.0):
    """Given training compute C in FLOPs, return compute-optimal (N, D).
 
    Uses C = 6 * N * D and D / N = tokens_per_param.
    """
    N = math.sqrt(C / (6.0 * tokens_per_param))
    D = tokens_per_param * N
    return N, D
 
# --- The original Chinchilla run ---
C_chinchilla = 5.76e23
N, D = chinchilla_optimal(C_chinchilla)
print(f"chinchilla budget: C={C_chinchilla:.2e} FLOPs")
print(f"  N* = {N/1e9:.1f}B params, D* = {D/1e12:.2f}T tokens")
# chinchilla budget: C=5.76e+23 FLOPs
#   N* = 69.3B params, D* = 1.39T tokens
 
# --- GPT-3, under the Chinchilla recipe ---
C_gpt3 = 3.14e23
N, D = chinchilla_optimal(C_gpt3)
print(f"gpt-3 budget:      C={C_gpt3:.2e} FLOPs")
print(f"  N* = {N/1e9:.1f}B params, D* = {D/1e12:.2f}T tokens")
# gpt-3 budget:      C=3.14e+23 FLOPs
#   N* = 51.2B params, D* = 1.02T tokens
# (actual gpt-3: 175B params, 300B tokens — way above the curve on N,
#  way below on D)
 
# --- Llama-3-70B, "over-trained" ---
C_llama3 = 6.3e24
N_star, D_star = chinchilla_optimal(C_llama3)
N_actual, D_actual = 70e9, 15e12
print(f"llama-3-70b budget: C={C_llama3:.2e} FLOPs")
print(f"  chinchilla says:  N*={N_star/1e9:.0f}B, D*={D_star/1e12:.1f}T")
print(f"  meta actually did: N={N_actual/1e9:.0f}B, D={D_actual/1e12:.0f}T")
print(f"  D/N ratio: {D_actual/N_actual:.0f}  (chinchilla target: 20)")
# llama-3-70b budget: C=6.30e+24 FLOPs
#   chinchilla says:  N*=229B, D*=4.6T
#   meta actually did: N=70B, D=15T
#   D/N ratio: 214  (chinchilla target: 20)

There isn't much calculation here, which is sort of the point. A one-line formula is doing the work that the literature spent years pinning down.

Where scaling laws bend

The straight line isn't straight forever. There are several ways it deviates, and each tells you something different about the model you're building.

The irreducible floor. There's some $E$ in the equation, and loss does not go below it. At some point you're modeling genuine entropy, not systematic error, and more compute doesn't help. In practice the straight line on log-log starts to curve gently downward as you approach the floor; the hero animation at the top of this post shows that bend. Modern frontier models are starting to feel this on their pre-training objective, which is one reason test loss on web text has become a less interesting metric than capability benchmarks.

Broken neural scaling laws. Caballero et al. 2022 fit a generalized form that allows the power law to have a "break": one slope in the low-compute regime and a different (usually gentler) slope in the high-compute regime. They found, across many modalities, that the single-slope law systematically underestimates loss at scale, because scaling laws themselves bend. The intuition is that easy gains run out. The first few orders of magnitude of compute buy a lot of loss reduction per FLOP. Later orders buy less. Smooth extrapolation from a small sweep is too optimistic.

Benchmark saturation. Test loss can keep falling while accuracy on a specific benchmark plateaus at 100%, because you've exhausted what the benchmark can distinguish. A lot of the "emergent capability" and "capability plateau" debate is really about benchmark ceilings rather than model ceilings. The 2023-2025 wave of harder benchmarks (MMLU-Pro, GPQA, SWE-Bench, HLE) was an attempt to keep scaling laws legible at the top end by giving them somewhere to land.

Domain-specific ceilings. In narrow domains with limited data (medical imaging, certain programming languages, minority-language text), the $D$ term of the law bottoms out because you physically run out of tokens. You can keep making $N$ bigger, and loss will keep falling, but only to a point.

The mental picture I try to carry: scaling laws are lawful in the middle of their range and unreliable at the edges. The middle is where you should trust the extrapolation. The edges are where you should expect surprises.

Test-time compute, a second axis

One of the biggest shifts between 2023 and 2026 is that "scaling" stopped meaning "scale pre-training" and started meaning "scale both pre-training and inference." You can spend more compute while generating a single answer (chain-of-thought reasoning, search, self-consistency, best-of-N sampling) and get better answers out of the same base model.

Snell et al. fit scaling laws for this second axis in 2024. The headline result: on reasoning-heavy tasks, a given increase in inference-time compute can substitute for a much larger increase in pre-training compute. Past a crossover point, you get more capability per FLOP by spending your marginal compute on longer reasoning chains and wider search than by training a bigger base model.

That's roughly the math behind o1, o3, and their open-weight imitators. The base model is a substrate, and a lot of the observed capability shows up only once you spend reasoning-time compute on top of it. The pre-training scaling law still holds; it's just no longer the only one, and for some workloads it's no longer the most cost-effective one.

A way to hold this in your head: pre-training scaling buys you a better prior, and test-time scaling buys you better inference from that prior. Both are lawful, they stack, and the frontier is increasingly about figuring out how much to spend on each.

Misconceptions

"Chinchilla obsoleted Kaplan." Not really. Kaplan's shape-of-the-law result was correct: loss falls as a power law in compute, across an enormous range, cleanly enough to extrapolate. What Kaplan got wrong was the tilt inside that shape, specifically how much of a budget increase should go to $N$ versus $D$. Chinchilla fixed that one number. Most of the conceptual vocabulary ("power law in compute," "irreducible loss," "compute-efficient frontier") is still Kaplan's, and reading Kaplan is still a useful exercise. If I had to point someone at one scaling-laws paper, I'd still pick Kaplan for the exposition and then immediately hand them Hoffmann as the correction.

"Compute-optimal equals best-to-deploy." Chinchilla-optimal minimizes training loss for a fixed training budget. It does not minimize the total cost of serving a model to users, which includes inference. Llama 3 (and most open-weight releases since) trains well past the Chinchilla optimum on purpose, because smaller models at the same quality are cheaper to serve. A lot of 2024-2026 releases make more sense once you separate "optimal training allocation" from "optimal deployment allocation." The correct frame is that Chinchilla gives you one specific answer to a specific question, and deployment asks a different question.

"Scaling laws predict emergent capabilities." They don't. Loss falls smoothly on a log-log plot. A lot of capabilities, by contrast, show up as sharp phase transitions on accuracy-vs-scale plots: near-zero performance for models below some size, then a rapid rise to near-saturation for models above it. Schaeffer et al. have argued that a lot of this "emergence" is an artifact of using discontinuous metrics (exact-match accuracy) on top of smooth loss curves. Whether it's real emergence or a measurement artifact, the scaling law for loss does not, by itself, tell you when a specific capability will appear. If you want to know when a model will reliably do arithmetic, or chain three tools, or stop inventing citations, scaling laws are not the instrument.

"Scaling laws guarantee future progress." They don't guarantee anything. They describe a regime that has held, empirically, across many orders of magnitude, and they give you a principled way to extrapolate inside that regime. They do not guarantee that the regime continues. The history of the field includes several regimes that held beautifully until they didn't (the recurrent-sequence-model era, for one). The law is a summary of what has happened, not a promise about what will.

What's next

This post covered the original Kaplan power laws, Chinchilla's correction, the over-training move that replaced Chinchilla-optimal with inference-aware optimal, where the straight line bends, and the second scaling axis at inference time. The takeaway is that a small number of fitted curves shape almost every compute-allocation decision the field makes.

The next post goes into the other side of the deployment equation: Inference Economics: Cost per Token, Batching, and Caching, how those inference FLOPs actually turn into dollars per million tokens, why batch size and KV caching dominate the math, and what the unit economics of serving a modern model look like from the inside.

Additional reading (and watching)

• Hestness, J. et al. (2017). Deep Learning Scaling is Predictable, Empirically. arXiv:1712.00409. The pre-transformer observation that loss scales as a power law in data and model size, across domains.

• Kaplan, J. et al. (2020). Scaling Laws for Neural Language Models. arXiv:2001.08361. The original careful fit across modern transformers; responsible for the "scale parameters hard" recipe that shaped GPT-3.

• The $C \approx 6ND$ approximation is argued carefully in the appendices of Kaplan 2020 and Hoffmann 2022. It counts forward and backward FLOPs per parameter per token for dense transformers.

• Hoffmann, J. et al. (2022). Training Compute-Optimal Large Language Models. arXiv:2203.15556. The Chinchilla paper: ~400 models, the $L(N, D)$ joint fit, and the $D/N \approx 20$ rule of thumb.

• Meta AI. (2024). The Llama 3 Herd of Models. arXiv:2407.21783. Section on inference-aware scaling describes the rationale for training past the Chinchilla-optimal token count.

• Caballero, E. et al. (2022). Broken Neural Scaling Laws. arXiv:2210.14891. Proposes a generalized scaling form that can fit the bends the single-power-law form misses at the edges.

• Snell, C. et al. (2024). Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters. arXiv:2408.03314. Fits scaling laws for inference-time compute and derives the crossover with pre-training compute.

• Schaeffer, R. et al. (2023). Are Emergent Abilities of Large Language Models a Mirage? NeurIPS 2023. Argues that "emergence" in capabilities is often an artifact of discontinuous metrics on top of smooth loss scaling.

• OpenAI. (2024). Learning to Reason with LLMs. The o1 system card; documents that accuracy scales lawfully with both train-time and test-time compute on reasoning benchmarks.

• Sutton, R. (2019). The Bitter Lesson. Short essay arguing that methods which scale with compute consistently outperform methods that encode human insight; the philosophical backdrop of every scaling-law paper.

• Henighan, T. et al. (2020). Scaling Laws for Autoregressive Generative Modeling. arXiv:2010.14701. Extends the Kaplan-style analysis to images, video, multimodal image-text, and math, showing the law is not language-specific.

• Villalobos, P. et al. (2022). Will we run out of data? arXiv:2211.04325. Projects when high-quality text data will constrain scaling; relevant to the "domain-specific ceiling" bend in the scaling law at the frontier.