Pretraining Data: Mixtures, Curation, and What the Model Sees

Two models with the same architecture and training budget can learn different capabilities from different corpora. A coding assistant needs enough useful code; a multilingual assistant needs coverage of its languages. Repeated or contaminated examples can make evaluation look better without making either model more useful.

The question is how to turn a collection of documents into a training distribution. That means choosing what to collect, what to remove, and how often the remaining material should be sampled.

Prerequisites

This post assumes you've read Training View vs. Inference View for why "how many tokens" is the resource you're spending, and BPE from Scratch for how raw text turns into the tokens we're counting.

What the model actually sees

For a concrete scale reference, the 2024 FineWeb release contained roughly 15 trillion tokens from 96 Common Crawl snapshots. A web-based pretraining pipeline might look like this:

1. Download a giant snapshot of the public web, for example Common Crawl.
2. Strip the HTML boilerplate, throw away pages that aren't the right language, throw away pages that look low-quality.
3. Detect and remove near-duplicate documents. An article may appear on several domains, giving the same text extra weight in training.
4. Scrub personally identifiable information and the worst of the toxic content.
5. Mix the result with carefully chosen smaller corpora: code from GitHub, books, scientific papers, high-quality Q&A threads, wikis, whatever you decide should round out the diet.
6. Tokenize and shuffle the selected text for training.

These choices affect coverage, memorization, and privacy. Evaluation contamination needs its own check: general deduplication does not automatically remove benchmark material.

FineWeb documents filtering and deduplication decisions through controlled training experiments. The original 15T-token corpus and 1.3T-token FineWeb-Edu subset are useful examples of curation at scale. Comparing raw HTML bytes with final tokens does not tell us a retention percentage: those are different units.

Mixtures: what ratio of what

After cleanup, choose how often to sample each source. A corpus can contain code, books, and web pages in one ratio while the training run samples them in another.

A reasonable taxonomy of domains:

• Web text. The long tail of the Internet. Forums, blogs, news, Q&A, Wikipedia, everything. A varied but selected sample of how people use written language.
• Code. GitHub, package registries, Stack Overflow answers. Provides programming examples.
• Books. Long-form prose. Fiction, non-fiction, textbooks. Teaches the model what sustained arguments and narrative structure look like. Quality varies within this category too.
• Scientific papers. arXiv, PubMed. Teaches formal writing, math, citations, the vocabulary of technical fields.
• Multilingual text. Non-English web and book text. Coverage, quality, tokenization, and sampling frequency all matter for language performance.

Meta reports an approximate Llama 3 mixture of 50% general knowledge, 25% math and reasoning, 17% code, and 8% multilingual tokens. These are the report’s categories. They are not interchangeable with source categories such as books or web pages, which can contain several of those kinds of content.

Curation: the pipeline inside the pipeline

Cleanup combines several filters, each with its own criteria and failure modes:

Language filtering. A small classifier (often fastText) labels each page’s language. An English-primary run might retain pages above a chosen confidence threshold. Audit false rejections, including code-switched text and underrepresented languages or dialects; the source corpus and classifier determine which material is missed.

Deduplication. Repeated pages and passages give duplicated material extra weight in training. Deduplication can reduce repetition and memorization risk, but the effect depends on what is removed and what replaces it.

Quality scoring. The scoring rule determines what counts as quality. Some teams train a classifier on "is this page like Wikipedia or not" and keep pages that score high. FineWeb-Edu trained a classifier on whether a page is "educational" and found that its selected corpus improved performance in the reported controlled benchmark comparisons. That result depends on the filtering threshold, training budget, and evaluation tasks. Other teams use per-page perplexity from a small reference model as the quality signal, keeping the low-perplexity pages. Both approaches are lossy in their own ways.

PII and toxicity scrubbing. Strip phone numbers, email addresses, social security numbers. Reject pages that score above some threshold on a toxicity classifier. This is difficult, because the line between "useful toxicity discussion" and "harmful content" is not a line a classifier can draw cleanly, and over-aggressive scrubbing kills content you wanted.

Filtering trades quantity and diversity against the properties a filter rewards. Aggressive filtering can remove valuable dialects, specialist writing, or difficult examples. Compare candidate filters with controlled training runs and inspect what they exclude.

Dedup, with a little bit of math

Deduplication needs a way to find likely matches without comparing every pair of documents.

The naive version of dedup is: for every pair of documents, compute some similarity score, and drop one of each pair that's above a threshold. Comparing all pairs is $O(N^2)$, which becomes prohibitively expensive for tens of billions of documents.

MinHash and locality-sensitive hashing (LSH) make candidate search cheaper. Represent each document by a set of overlapping character n-grams, or shingles. Under a uniformly random ordering, two sets have the same minimum element with probability equal to their Jaccard similarity. Repeating that construction gives a compact similarity estimate.

Quick reminder of Jaccard similarity for two sets $A$ and $B$:

If two documents share most of their shingles, $J$ is close to 1. If they share almost none, $J$ is close to 0. The MinHash trick is: pick a uniformly random ordering of the shingle universe, represented by $h$, compute $\min_{x \in A} h(x)$ and $\min_{x \in B} h(x)$, and then:

For nonempty sets, the equality holds for a uniformly random permutation (or an ideal minwise-independent hash). Practical seeded hashes approximate this condition. Repeating the construction estimates Jaccard from signature agreement.

Split a $K=br$ signature into $b$ bands of $r$ rows. Two documents become a candidate if any band matches. Under independent hashes, a pair with similarity $s$ becomes a candidate with probability $1-(1-s^r)^b$. For example, 128 hashes can form 16 bands of 8 rows. This is a soft probability curve, not a hard similarity threshold; verify candidates before dropping documents.

Worked example: spending a fixed token budget

Suppose two training runs each process 2T tokens. At 10% code, a run sees 0.2T code tokens and 1.8T other tokens. At 50%, it sees 1T of each. Increasing code exposure by 0.8T means removing 0.8T from another part of the mixture.

To measure the effect, keep architecture, tokenizer, total tokens, and optimization settings fixed. Evaluate coding, reasoning, language coverage, and the tasks served by the displaced data. Repeat runs where practical.

Chinchilla, Kaplan, and the scaling law underneath all this

For a fixed compute budget $C$ (FLOPs), how should you split your effort between making the model bigger (more parameters $P$) and training it for longer (more tokens $D$)?

Kaplan et al. (2020) and Hoffmann et al. (2022) reached different fitted allocations of training compute between parameter count and token count. GPT-3’s observed 300B/175B ratio is about 1.7 tokens per parameter; that number is not itself a Kaplan scaling law.

Hoffmann et al. trained Chinchilla, a 70B-parameter model on 1.4T tokens, and reported stronger performance than the larger Gopher under a similar training-compute budget. The resulting 20-token-per-parameter heuristic is useful context, but its best value depends on the fitted regime and objective.

A training-compute optimum does not include the cost of serving the model. Spending more training compute on a smaller model can be worthwhile if it reduces inference cost at the required quality.

Llama 3’s smaller models were deliberately trained beyond the training-compute-optimal allocation. Meta describes this as improving quality within an inference budget. That objective includes serving cost, unlike a fixed pretraining-compute budget.

The data wall

If a run consumes more tokens than a corpus contains, it must repeat data, expand the corpus, or change its training plan. The number of training tokens is therefore not the same as the number of distinct tokens available.

Estimates of available text depend on language, access, filtering, deduplication, and tokenizer. Treat them as estimates under a specified definition, not a fixed line every model will cross.

In the data-constrained regime studied by Muennighoff et al., repeating data for up to four passes incurred little loss penalty, with diminishing returns after that. The useful number of passes depends on the corpus and training setup. Synthetic data and broader language coverage are other options, each requiring its own quality checks.

Implementation sketch: a toy MinHash filter

This standard-library example finds candidate matches with MinHash and verifies their shingle similarity.

import hashlib
from collections import defaultdict
 
def shingles(text: str, k: int = 5) -> set[str]:
    if k < 1:
        raise ValueError("k must be positive")
    text = text.lower()
    # A short or empty document still has a nonempty representation.
    return {text[i:i+k] for i in range(len(text)-k+1)} or {text}
 
def signature(sh: set[str], count: int) -> tuple[int, ...]:
    # Seeded SHA-1 approximates independent orderings for this toy example.
    return tuple(min(int(hashlib.sha1(f"{seed}:{s}".encode()).hexdigest(), 16)
                     for s in sh) for seed in range(count))
 
def dedup(docs, num_hashes=128, bands=16, threshold=0.8):
    if bands < 1 or num_hashes < 1 or num_hashes % bands:
        raise ValueError("positive hash count must be divisible by bands")
    if not 0 <= threshold <= 1:
        raise ValueError("threshold must lie in [0, 1]")
    rows = num_hashes // bands
    buckets = defaultdict(list)
    sets = [shingles(d) for d in docs]
    kept = []
    for i, sh in enumerate(sets):
        sig = signature(sh, num_hashes)
        keys = [(b,) + sig[b*rows:(b+1)*rows] for b in range(bands)]
        candidates = {j for key in keys for j in buckets[key]}
        duplicate = any(len(sh & sets[j]) / len(sh | sets[j]) >= threshold
                        for j in candidates)
        if not duplicate:
            kept.append(i)
            for key in keys:
                buckets[key].append(i)
    return [docs[i] for i in kept]
 
if __name__ == "__main__":
    docs = ["The quick brown fox.", "The quick brown fox.",
            "Training data shapes a model.", "", ""]
    assert dedup(docs) == [docs[0], docs[2], ""]
    print(dedup(docs))
 
 

Signature construction is independent across documents and can be parallelized. The number of hashes sets signature cost; bands and rows trade candidate recall against verification work. This toy keeps the earliest verified representative. Approximate candidate search can still miss near-duplicates, so a production pipeline needs a measured recall/precision tradeoff.

What the counts leave out

A token count alone does not describe source quality, repetition, domain coverage, or benchmark contamination. A parameter count alone does not say whether a model received enough useful training. Compare both, and keep evaluation separate from curation decisions.

What's next

The next post covers Distributed Training: FSDP, DeepSpeed, and Megatron: how to distribute model state and computation across GPUs, and what communication each approach requires.

Additional reading (and watching)

• Common Crawl Foundation. commoncrawl.org. The decade-running open crawl of the public web. A widely used source for open web corpora.

• Penedo, G., et al. (2024). The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale. arXiv:2406.17557. Documents the full FineWeb pipeline, including dedup thresholds, language filters, and the FineWeb-Edu quality classifier.

• Meta AI (2024). The Llama 3 Herd of Models. arXiv:2407.21783. The Llama 3 technical report. Includes mixture design and data-mixture experiments.

• Broder, A. Z. (1997). On the resemblance and containment of documents. Compression and Complexity of Sequences. The original MinHash paper. Introduces the probabilistic-signatures-equals-Jaccard trick that supports many near-duplicate detection pipelines.

• Soldaini, L., et al. (2024). Dolma: An Open Corpus of Three Trillion Tokens for Language Model Pretraining Research. arXiv:2402.00159. Documents AI2’s open corpus and its curation pipeline.

• Kaplan, J., et al. (2020). Scaling Laws for Neural Language Models. arXiv:2001.08361. An influential empirical scaling analysis; its compute allocation differs from the later Chinchilla results.

• Hoffmann, J., et al. (2022). Training Compute-Optimal Large Language Models. arXiv:2203.15556. The Chinchilla paper. Compared parameter and token allocations under a fixed compute budget.

• Villalobos, P., et al. (2022). Will We Run Out of Data? Limits of LLM Scaling Based on Human-Generated Data. arXiv:2211.04325. Estimates human-generated data stocks under explicit assumptions; these estimates are not a fixed corpus-size limit.

• Shumailov, I., et al. (2023). The Curse of Recursion: Training on Generated Data Makes Models Forget. arXiv:2305.17493. Studies distribution loss in recursive training settings.

• Muennighoff, N., et al. (2023). Scaling Data-Constrained Language Models. arXiv:2305.16264. Studies the loss and compute tradeoffs of repeated data in a data-constrained regime.

• Together Computer (2024). RedPajama: an Open Dataset for Training Large Language Models. arXiv:2411.12372. The V2 paper. Thirty trillion tokens with quality-signal metadata, designed for custom-mixture experimentation.

• Touvron, H., et al. (2023). LLaMA: Open and Efficient Foundation Language Models. arXiv:2302.13971. Documents LLaMA’s pretraining data mixture in Table 1.