Pretraining Data: Mixtures, Curation, and What the Model Sees

Here's something that took me a long time to really believe. When you train two language models of the same size, with the same architecture and compute budget, and you change only the pretraining data, you end up with two very different models. Different benchmark scores, different failure modes, different personalities.

I used to think of the data as the boring part of the pretraining story. Scaling laws, attention variants, optimizer tweaks, those felt like the interesting levers. The data was just, you know, "the Internet."

That picture is wrong. The corpus shapes the model more than the architecture does. If you give me two teams, the same chip budget, and the same number of parameters, and you let one team pick the architecture and the other team pick the data, I'd bet on the data team.

So this post is about what's actually in the pile of text a model trains on. Where it came from, what gets thrown away, what ratios matter, and why.

Prerequisites

This post assumes you've read Training View vs. Inference View for why "how many tokens" is the resource you're spending, and BPE from Scratch for how raw text turns into the tokens we're counting. The math here is lighter than in most posts in this arc. Most of the lift is conceptual.

What the model actually sees

Let's start with the simple version. A frontier-scale pretraining run in 2026 consumes something between 10 and 20 trillion tokens. The process for getting those tokens looks roughly like this:

1. Download a giant snapshot of the public web, usually Common Crawl, which has been crawling the web for over a decade and publishes ~hundreds of terabytes of HTML per snapshot.
2. Strip the HTML boilerplate, throw away pages that aren't the right language, throw away pages that look low-quality.
3. Detect and remove near-duplicate documents. The web is astonishingly redundant. A single popular article gets republished across dozens of domains.
4. Scrub personally identifiable information and the worst of the toxic content.
5. Mix the result with carefully chosen smaller corpora: code from GitHub, books, scientific papers, high-quality Q&A threads, wikis, whatever you decide should round out the diet.
6. Tokenize the whole thing. Shuffle. Train.

Each step is load-bearing, and each is a decision with downstream capability consequences. Get any one of them wrong and you get a model that speaks fluent English but can't code, or one that scores great on benchmarks but memorizes parts of the test set, or one that regurgitates phone numbers from a forum someone scraped in 2017.

The volume of what gets thrown away is the thing that surprised me when I first looked at open pretraining pipelines. FineWeb, the best-documented open web dataset as of this writing, starts from ~30 petabytes of Common Crawl HTML and finishes with ~15 trillion quality-filtered tokens. That's a reduction of roughly three orders of magnitude before you even start designing the mixture.

Mixtures: what ratio of what

Once you've done the cleanup, you don't train on just "cleaned web." You train on a mixture. Different slices of text teach the model different things, and the ratio between them matters a lot.

A reasonable taxonomy of domains:

• Web text. The long tail of the Internet. Forums, blogs, news, Q&A, Wikipedia, everything. The broadest possible distribution of how humans use natural language.
• Code. GitHub, package registries, Stack Overflow answers. Teaches the model programming, but also (and this is the part that turned out to matter a lot) structured reasoning.
• Books. Long-form prose. Fiction, non-fiction, textbooks. Teaches the model what sustained arguments and narrative structure look like. Books also have far fewer grammatical errors per token than web text, which helps.
• Scientific papers. arXiv, PubMed. Teaches formal writing, math, citations, the vocabulary of technical fields.
• Multilingual text. Non-English web and book text. How much of this you put in determines how good the model is at every language it sees during training.

Each frontier model has its own mix, and the mixes have shifted a lot over time. GPT-3's 2020 pretraining mixture was basically all web text and books. LLaMA 1 in 2023 still leaned heavily on web text but added dedicated slices for GitHub, arXiv, and a handful of European languages. LLaMA 3 in 2024 broke dramatically from all of that: 17% code, 15% non-English, and a much more aggressive quality filter on the web part.

The knob that surprised me most was code. Meta's Llama 3 technical report calls this out directly: adding code to pretraining helps coding tasks (obvious), but it also helps reasoning and math benchmarks by a measurable amount. DeepSeek reported the same finding, and it's been reproduced by enough different teams now that it's a load-bearing assumption in how frontier mixtures are designed. Nobody fully understands the mechanism. The leading hypothesis is that code's forced structure (consistent indentation, explicit scopes, clean control flow, executable semantics) gives the model a cleaner signal to learn structured reasoning from than free-form natural language does.

I think of it like this. Natural language is full of implicit structure that you learn only by accumulating huge amounts of evidence. Code makes that structure explicit. A function definition, a loop, a conditional are tiny, perfectly formatted examples of "here's a premise, here's what follows from it." The model picks up on that, and the effect spills over into places we don't expect.

Curation: the pipeline inside the pipeline

Now let's open up step 2 of the pipeline. The cleanup step is where an enormous amount of the actual craft lives, and it's also the part that varies most between labs. The usual stages, in roughly the order a typical pipeline runs them:

Language filtering. A small classifier (often fastText) labels each page's language. For an English-primary run, you keep pages scoring above a threshold, typically 0.65 or 0.7 for en. This alone drops a huge chunk of Common Crawl. It also disproportionately drops African languages, code-switched text, and technical content the classifier misreads. Thoughtful teams get careful here.

Deduplication. Remove near-duplicates. This matters more than it looks like it should. A single viral news story can appear in hundreds of Common Crawl documents, and every duplicate you leave in biases the model toward memorizing that story rather than learning the general language pattern. We'll get into the math of how dedup actually works in a minute.

Quality scoring. The fuzziest stage. Some teams train a classifier on "is this page like Wikipedia or not" and keep pages that score high. FineWeb-Edu trained a classifier on whether a page is "educational" and found that quality-filtering aggressively on that signal beat much bigger unfiltered corpora on downstream benchmarks. Other teams use per-page perplexity from a small reference model as the quality signal, keeping the low-perplexity pages. Both approaches are lossy in their own ways.

PII and toxicity scrubbing. Strip phone numbers, email addresses, social security numbers (yes, really). Reject pages that score above some threshold on a toxicity classifier. This is genuinely hard, because the line between "useful toxicity discussion" and "harmful content" is not a line a classifier can draw cleanly, and over-aggressive scrubbing kills content you wanted.

Every one of these stages has knobs, and every knob has been the subject of at least one paper. The shared instinct across labs: be aggressive. Throw away more than feels comfortable. Frontier pretraining is a quality-limited problem, not a data-limited one. Adding more garbage is worse than adding less clean text.

Dedup, with a little bit of math

Deduplication is the one stage where it's worth working through the math, because the math is both beautiful and directly useful.

The naive version of dedup is: for every pair of documents, compute some similarity score, and drop one of each pair that's above a threshold. That doesn't work. If you have $N$ documents, comparing all pairs is $O(N^2)$, and $N$ is tens of billions. You'd die before finishing.

The clever version is MinHash with locality-sensitive hashing (LSH). The idea: represent each document as a set of overlapping character n-grams (called "shingles"), hash that set down to a tiny signature, and cluster documents whose signatures collide on enough sub-bands. The magic is that the probability of two documents' signatures agreeing, on a randomly chosen hash, equals the Jaccard similarity of their shingle sets.

Quick reminder of Jaccard similarity for two sets $A$ and $B$:

If two documents share most of their shingles, $J$ is close to 1. If they share almost none, $J$ is close to 0. The MinHash trick is: pick a random hash function $h$, compute $\min_{x \in A} h(x)$ and $\min_{x \in B} h(x)$, and then:

One hash, one number per document, and the probability of a collision is exactly the thing we want to estimate. Do this with $K$ independent hash functions and you get a $K$-element signature whose agreement rate is an unbiased estimator of Jaccard similarity.

Now you wrap this in LSH: split the $K$-element signature into bands, and declare two documents a candidate pair if any band matches exactly. Tuning the band size and count gives you a tunable similarity threshold. In practice, 128 hashes split into 14 bands of about 9 hashes each picks out pairs with Jaccard $\gtrsim 0.85$, which catches near-duplicates while leaving genuinely distinct documents alone.

The practical effect of running this on Common Crawl is dramatic. The Dolma paper reports that their URL- and paragraph-level dedup dropped the raw CC subset by roughly 60% before other filters ran. FineWeb reports similar numbers. A bit more than half of the web, by token count, is near-duplicate of something else.

The worked example: code share, and what it buys you

Time for the concrete example I promised. Take a hypothetical 7B-parameter model, trained for 2 trillion tokens, and vary exactly one thing: the fraction of code in the pretraining mixture. One run at 10% code, another at 50% code, everything else identical.

What happens to downstream benchmarks?

First, the coding benchmarks (HumanEval, MBPP) roughly double from 10% to 50% code. That's the direct effect, and it's huge.

Second, GSM8K, a grade-school math word-problem benchmark with nothing to do with code syntax, gains about 14 points. This is the surprising one. You didn't add math data; you added code. But structured-reasoning performance went up anyway. This is the "code helps reasoning" finding in a nutshell.

Third, HellaSwag (commonsense natural-language inference) slips by a couple points. This is the cost. Every token of code you add is a token of web text you didn't add. You moved capability around, you didn't summon it from nowhere.

Every single design decision in a frontier mixture has this shape. Adding multilingual helps the target languages and usually costs the English benchmarks a little. Adding scientific papers helps technical QA and hurts chit-chat. Overweighting books helps long-form prose and hurts the conversational feel. There's no free lunch at the mixture layer; there's only which tradeoffs you want.

Chinchilla, Kaplan, and the scaling law underneath all this

We'll get into scaling laws in depth much later in the series. For the mixture story, here's the short version you need right now.

For a fixed compute budget $C$ (FLOPs), how should you split your effort between making the model bigger (more parameters $P$) and training it for longer (more tokens $D$)?

Kaplan et al. in 2020 fit scaling laws to a bunch of models and concluded that you should spend most of your compute on model size, not data. In practice, this is how models like GPT-3 ended up trained at a ratio of about 1.7 tokens per parameter (175B parameters, 300B tokens) — well below what later work would call compute-optimal.

Hoffmann et al. in 2022 rerun the analysis more carefully and came to a different answer: models were being dramatically undertrained. The compute-optimal ratio is more like 20 tokens per parameter. They demonstrated this by training a 70B-parameter model (Chinchilla) on 1.4T tokens and beating Gopher, which was 280B parameters trained on 300B tokens, on every metric they tried.

Chinchilla's ratio replaced Kaplan's in every serious pretraining stack within about a year, and the open-weight model world (LLaMA, Mistral, Qwen, DeepSeek) was built on top of it.

Then something interesting happened around 2024. Meta trained Llama 3 70B on 15 trillion tokens. At D/P = 214, that's roughly 10× the Chinchilla ratio. The reason is boring and correct: Chinchilla optimizes loss per unit of training compute. But you're going to serve that model many times. If you can make training more expensive in exchange for a smaller, faster-to-serve model that hits the same quality, that's a good trade at any reasonable scale of inference traffic.

The deeper lesson: "compute-optimal" is a function of what you're optimizing. Chinchilla is compute-optimal for training loss. Llama 3 is compute-optimal for total-cost-of-ownership across training plus serving. The scaling ratio moves around depending on which cost you care about. When you see "20 tokens per parameter" cited as a universal rule, that context is usually missing.

The data wall

All of this raises a question. If frontier models now consume 10-20 trillion tokens per training run, and if deliberate overtraining pushes that number higher every year, are we going to run out of text?

Villalobos et al. estimated in 2022 that the total stock of high-quality English language text on the public web is somewhere between 9 and 17 trillion tokens, depending on how aggressively you filter. Frontier training is inside that band and pushing into it.

Three things are happening in response.

Multilingual data does not fall off a cliff. There's still vastly more non-English text on the web than frontier models have used. The ratio of information-per-token is worse in some languages than others, but there's room.

Synthetic data. Frontier labs now routinely train on outputs from other models, often from the previous generation of the same lab's own model. Done carefully, this works; done carelessly, it accumulates model artifacts (Shumailov et al. call this "model collapse"). The state of the art as of 2026 is carefully curated chains of "strong model produces completion → human reviews or filters → train on the filtered output."

Training with constraints. If you only have 5 trillion real tokens but want to train a model that a naive Chinchilla fit says wants 15 trillion, you can either repeat data or accept a worse loss. Muennighoff et al. study this rigorously and find that up to ~4 epochs of repetition is essentially free, and up to ~16 epochs still helps, after which you're better off just adding parameters.

So the data wall is real, but it's not a cliff. It's a slow-moving constraint that the field is adapting to with a mix of multilingual expansion, synthetic data, and controlled repetition. A lesson we are, uh, still learning.

Implementation sketch: a toy MinHash filter

Here's the smallest runnable thing that gets the MinHash intuition across. Eighty lines of Python, no dependencies outside the standard library, computing a signature for each document and clustering near-duplicates.

import hashlib
from collections import defaultdict
 
# --- Shingling: break a document into overlapping character n-grams ---
def shingles(text: str, k: int = 5) -> set[str]:
    text = text.lower()
    return {text[i:i + k] for i in range(len(text) - k + 1)}
 
# --- One MinHash value: the minimum hash over the shingle set ---
def minhash(sh: set[str], seed: int) -> int:
    # Each seed gives us a different hash function by prefixing the string.
    return min(
        int(hashlib.sha1(f"{seed}:{s}".encode()).hexdigest(), 16)
        for s in sh
    )
 
# --- A full signature: K independent MinHash values ---
def signature(text: str, num_hashes: int = 128) -> tuple[int, ...]:
    sh = shingles(text)
    return tuple(minhash(sh, seed) for seed in range(num_hashes))
 
# --- LSH: bucket signatures by bands and find collision pairs ---
def dedup(docs: list[str], num_hashes: int = 128, bands: int = 16):
    rows = num_hashes // bands
    sigs = [signature(d, num_hashes) for d in docs]
 
    # Each band is a slice of the signature. Docs collide in a band if
    # they have identical values in that slice.
    buckets: dict[tuple, list[int]] = defaultdict(list)
    for i, sig in enumerate(sigs):
        for b in range(bands):
            key = (b,) + sig[b * rows:(b + 1) * rows]
            buckets[key].append(i)
 
    # Any pair of docs that share a bucket is a candidate near-duplicate.
    keep = [True] * len(docs)
    for members in buckets.values():
        if len(members) > 1:
            # Keep the earliest, drop the rest.
            for idx in members[1:]:
                keep[idx] = False
    return [d for d, k in zip(docs, keep) if k]
 
# --- Sanity check ---
if __name__ == "__main__":
    docs = [
        "The quick brown fox jumps over the lazy dog.",
        "The quick brown fox jumped over a lazy dog.",   # near-dup of #0
        "Pretraining data shapes capabilities more than architecture.",
        "Bananas are yellow. Dogs are loyal. The sky is blue.",
    ]
    survivors = dedup(docs, num_hashes=32, bands=8)
    for s in survivors:
        print(s)
    # Expected: docs 0 (or 1), 2, and 3 survive.

Two things to notice. First, the algorithm is trivially parallel across documents: compute signatures independently, then bucket. This is why it scales to terabytes. Second, the quality of dedup depends on num_hashes and bands — the product sets the cost per document, the ratio sets the similarity threshold. Real pipelines (Dolma's, FineWeb's, the commercial pipelines inside frontier labs) all use some variant of exactly this, usually with 128-256 hashes and 14-20 bands.

Misconceptions

"The pretraining data is just the Internet." Technically true in that most of the tokens come from the public web. Practically misleading, because the pipeline between "the Internet" and "what the model sees" throws away more than 99% of what it starts with, reweights what remains, blends in carefully curated specialized corpora, and then shuffles the result. "The Internet, aggressively curated and blended with code and books" is more accurate.

"You can always just add more data." Not anymore, not at the frontier. The total stock of high-quality English web text is finite, frontier models are training inside that stock's size, and the marginal return on adding low-quality data is negative. This is the scarcity the field is optimizing against in 2026, and it's driving the shift toward multilingual expansion and synthetic data.

"More parameters is what makes a model smarter." Only half the story. You also need enough tokens to train those parameters, and the tokens have to come from a mixture that teaches what you want taught. A 7B model trained on 15T high-quality tokens, with code and multilingual and high-edu-score web, beats a 70B model trained on 300B unfiltered tokens on almost every benchmark. The architecture is a multiplier; the data sets the ceiling.

What's next

The next post covers Distributed Training: FSDP, DeepSpeed, and Megatron, where we pick up the mixture we just designed and spread the training run across hundreds of GPUs, looking at the three frameworks that dominate modern pretraining and the communication costs each one pays.

Additional reading (and watching)

• Common Crawl Foundation. commoncrawl.org. The decade-running open crawl of the public web. Every frontier model's pretraining pipeline starts with one or more CC snapshots.

• Penedo, G., et al. (2024). The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale. arXiv:2406.17557. Documents the full FineWeb pipeline, including dedup thresholds, language filters, and the FineWeb-Edu quality classifier.

• Meta AI (2024). The Llama 3 Herd of Models. arXiv:2407.21783. The Llama 3 technical report. Best public account of mixture design at frontier scale, including the code-helps-reasoning result.

• Broder, A. Z. (1997). On the resemblance and containment of documents. Compression and Complexity of Sequences. The original MinHash paper. Introduces the probabilistic-signatures-equals-Jaccard trick that underpins every modern dedup pipeline.

• Soldaini, L., et al. (2024). Dolma: An Open Corpus of Three Trillion Tokens for Language Model Pretraining Research. arXiv:2402.00159. AI2's fully open pretraining pipeline, with every stage ablated. The best single reference for what a curation pipeline actually looks like.

• Kaplan, J., et al. (2020). Scaling Laws for Neural Language Models. arXiv:2001.08361. The first serious scaling-law paper; concluded (incorrectly, as Chinchilla later showed) that models should be larger relative to their token budgets.

• Hoffmann, J., et al. (2022). Training Compute-Optimal Large Language Models. arXiv:2203.15556. The Chinchilla paper. Re-derived compute-optimal scaling and demonstrated that models had been massively undertrained.

• Villalobos, P., et al. (2022). Will We Run Out of Data? Limits of LLM Scaling Based on Human-Generated Data. arXiv:2211.04325. The earliest careful estimate of the total stock of high-quality English web text. Frontier runs are now inside its middle range.

• Shumailov, I., et al. (2023). The Curse of Recursion: Training on Generated Data Makes Models Forget. arXiv:2305.17493. Coined "model collapse" for the degradation that happens when you train repeatedly on model-generated data without careful filtering.

• Muennighoff, N., et al. (2023). Scaling Data-Constrained Language Models. arXiv:2305.16264. Extends Chinchilla into the data-constrained regime; finds that modest data repetition is nearly free and large repetition still helps.

• Together Computer (2024). RedPajama: an Open Dataset for Training Large Language Models. arXiv:2411.12372. The V2 paper. Thirty trillion tokens with quality-signal metadata, designed for custom-mixture experimentation.

• Touvron, H., et al. (2023). LLaMA: Open and Efficient Foundation Language Models. arXiv:2302.13971. The original LLaMA paper, whose Table 1 is the canonical small-lab-scale documented pretraining mixture everyone in the open-weights world started from.