Skip to content
Arc 7: Training & Post-Training

Synthetic Data and Distillation

16 min readbuilds on Pretraining Data: Mixtures, Curation, and What the Model Sees, Supervised Fine-Tuning and Cross-Entropy, KL Divergence, and What Loss Functions Measure

Distillation copies a frozen teacher's behavior into a smaller student — either by matching its output distribution (logit KD) or by training on the strings it samples (synthetic-data SFT).

A frontier model costs a ridiculous amount of money to train. Tens of millions of dollars in GPUs, months of engineering, terabytes of carefully filtered text. Once it's finished, you have this giant artifact that can do extraordinary things, and also a very natural question: can I get most of that behavior out of a model 10× smaller?

That question has two families of answers, and they get mixed up all the time because the pipelines look similar from the outside. One is distillation, which means training a small model to match a big model's output distribution. The other is synthetic data, which means using a strong model to manufacture training examples for a weaker one. They share a lot of machinery and often get combined, but the mechanism by which the student gets better is different in each case. This post separates them, builds the intuition for each, and then looks at what happens when the thing eating its own outputs is a whole ecosystem.

Prerequisites

This post assumes you've read:

The problem

Say you have a 70B-parameter model that behaves the way you want — follows instructions, writes clean code, gives reasonable answers about biology. You'd love to deploy it everywhere, but inference on a 70B model is expensive: big GPU, lots of memory bandwidth, slow tokens per second. Every prompt you serve is a tax on your infrastructure. Meanwhile a 7B-parameter model runs on a single consumer GPU, generates tokens fast, and if it could imitate the 70B's behavior closely enough, it would be good enough for most of what you care about.

The question is how to get the 7B to act like the 70B. You can't just copy weights, because the architectures have different shapes. Training the 7B from scratch on the same data won't work either; that's the whole reason the 70B is better in the first place. What you need is a way to let the 70B teach the 7B directly.

Here's the setup:

  • The big model. Call it the teacher. It's frozen. Nobody's updating its weights.
  • The small model. Call it the student. It has its own weights, its own loss, and we're going to train it.
  • Some way of copying "what the teacher knows" into the student's weights.

The rest of the post is about the different ways people have tried to close that loop.

Soft labels: what the teacher actually knows

The first insight, the one that made Hinton et al.'s 2015 paper a classic, is that a trained model's output distribution contains way more information than its top prediction alone. If you ask the teacher "what comes after the cat sat on the?" and it says mat with probability 0.42, rug at 0.24, floor at 0.14, couch at 0.09, and a long tail after that, those probabilities are a quiet confession about what the model has learned. The teacher thinks rug is almost as likely as mat, but much more likely than couch. That relative ordering encodes the teacher's sense of what makes sense in context.

Hinton called this "dark knowledge." The one-hot "correct" label says mat and nothing else. The full distribution says mat, and here's the entire shadow of things that could have been said, and here's how they rank. If you train the student to match the full distribution, you transmit all of that, not just the argmax.

context
The cat sat on the ___ (predict next)
hard label
one-hot · 1 bit of info
mat
1.00
rug
0.00
floor
0.00
couch
0.00
bed
0.00
chair
0.00
teacher's soft label
full distribution
mat
0.62
rug
0.19
floor
0.09
couch
0.06
bed
0.03
chair
0.02
τ = 1.00
sharp (argmax-like)flat (more dark knowledge)
Same prompt, two targets. On the left, the hard label points at one right answer. On the right, the teacher's soft distribution reveals its whole sense of the space. Nudge the temperature slider up and the soft distribution flattens, surfacing even more of the teacher's relative-preference structure.

So.

The distillation loss works on that right-hand distribution. Given the teacher's probabilities pteacherp_{\text{teacher}} over the vocabulary and the student's pstudentp_{\text{student}}, you minimize the KL divergence between them:

LKD=τ2⋅KL ⁣(pteacherτ ∥ pstudentτ)L_{\text{KD}} = \tau^2 \cdot \text{KL}\!\left( p_{\text{teacher}}^{\tau} \,\Vert\, p_{\text{student}}^{\tau} \right)

The τ\tau is a temperature applied to the softmax before you compute the KL. Higher τ\tau flattens both distributions, which drags the student's attention away from the single mode and onto the relative ordering of secondary options. The τ2\tau^2 out front is the rescaling factor Hinton derives so that the gradient magnitudes don't fall off as you warm up.

In practice you usually train the student on a weighted combination of the distillation loss and the normal cross-entropy on the ground-truth label, if you have one. Something like:

Ltotal=α⋅LCE+(1−α)⋅LKDL_{\text{total}} = \alpha \cdot L_{\text{CE}} + (1 - \alpha) \cdot L_{\text{KD}}

But the critical piece is the KD term. That's what makes distillation different from just copying the teacher's top pick.

Two flavors of "use the teacher"

Once you understand soft labels, there are two recognizable paths people actually take. The vocabulary gets loose here, so let me pin it down.

Logit distillation. You run the teacher and the student on the same input, grab both of their output distributions, and optimize the KL between them. This is the closest thing to "classical" knowledge distillation. Gemma 2 notably did a version of this for its 2B and 9B models at pretraining scale. It requires the teacher's logits, which means you need access to the model — it's easiest inside one lab, harder to do against a closed API.

Synthetic-data SFT. You sample text from the teacher, keep the tokens as ground-truth targets, and fine-tune the student with ordinary cross-entropy on those tokens. No logits needed, just the sampled strings. Alpaca, Self-Instruct, Orca, and most of what people informally call "distilling GPT-4" are this flavor. The student isn't matching a distribution; it's matching the teacher's samples and hoping that the teacher's distribution shows up in the aggregate.

Both of these get called "distillation" in the wild. Both work. They have different properties: logit distillation is a stronger signal per example, while synthetic-data SFT is easier to set up and composes naturally with RLHF or DPO downstream. Most of what you read about in production uses some mix.

The Alpaca-style pipeline

The synthetic-data-as-SFT route is worth walking through because the scale involved is genuinely surprising. In 2023, the Stanford Alpaca team started with 175 human-written seed tasks, asked text-davinci-003 to generate variations, filtered the results, and ended up with 52,000 instruction-response pairs. They fine-tuned a Llama 7B on that set for under $600 of compute. The resulting model was, for casual instruction-following, within spitting distance of text-davinci-003's actual behavior. A lot of the early open-weights instruct models are descendants of that exact recipe.

SEEDTEACHER · GENERATEFILTERSTUDENT SFT175human-writtentasksGPT-3.5 / 40generated (instruct, response)dedup · lengthROUGE-L > 0.7kept 07B base0SFT rows×470 expand×297 finalAlpaca-style pipeline: 175 seeds → teacher-generated → filtered → 52k SFT rows
Alpaca-style pipeline. 175 seeds become tens of thousands of (instruction, response) rows via a teacher. The filter stage is the unglamorous part: ROUGE-L dedup, length cutoffs, NSFW rejection, format sanity. The surviving rows are what the student actually sees.

The pipeline is roughly:

  1. Start with a small, hand-written set of diverse tasks (the "seed" pool).
  2. Prompt the teacher to produce new tasks in the same style, either from the seeds directly or via templated few-shot prompts.
  3. For each generated instruction, prompt the teacher again to produce a response.
  4. Filter. Deduplication (e.g., ROUGE-L similarity cutoff), length constraints, cheap safety filters, sometimes LLM-as-judge scoring.
  5. Feed the surviving rows into an SFT loop against the student.

The filter stage tends to get glossed over. In the Self-Instruct paper, less than half of the raw generations survive filtering. The pipeline's real output is not "many tokens from GPT-4"; it's "many tokens from GPT-4 that look like the kind of task you wanted to teach the student." The quality bar is what moves the needle, not the raw volume.

Now.

Why does it work at all?

Here's the intuition I carry around. The student is a function with billions of parameters and a specific inductive bias (the transformer architecture, pretrained on text). The teacher is a much bigger function with the same bias, trained longer. The student has enough capacity to represent most of what the teacher knows about any narrow behavior like writing polite refusals, formatting markdown lists, or answering MMLU-style multiple choice. But the student's base model doesn't know which behaviors to put on top.

Distillation, in either flavor, shows the student the teacher's behavior on a huge number of inputs. The student's weights adjust to imitate that behavior. Where the teacher is strong, the student inherits the strength. Where the teacher is weak or just wrong, the student inherits the weakness. This is a pointed copy of the parts of the teacher's behavior you expose the student to. It is not a general uplift.

This has a useful consequence. The student can exceed the teacher on narrow axes, but very rarely on everything. Orca's 13B model beat GPT-3.5 on several reasoning benchmarks because the Orca team specifically distilled GPT-4's reasoning traces, not just its final answers. Phi-3 mini does unreasonably well on MMLU because the Phi team specifically curated synthetic "textbook-like" content. Neither of those small models is uniformly better than a frontier model. They're better on the slice they were distilled toward.

And the other side of that coin: the student can inherit the teacher's blind spots, too. If the teacher has a bad habit (confabulating citations, refusing benign prompts, preferring a certain rhetorical style), the student will come out with the same habit baked in. There's no filter for "only learn the good parts" short of the explicit one you build during data curation.

student scores vs. teacher, per task
training progress · 0%
instruction followingpipeline explicitly targeted this
0.88
0.18
markdown formattingeasy surface behavior to copy
0.95
0.32
multi-step reasoningtraces helped, but capacity bounds it
0.72
0.21
rare language QAno Swahili in the SFT set
0.54
0.14
citation accuracyteacher confabulates, student learns the same
0.38
0.28
teacher (ceiling)
student after distillation
student baseline
The student chases the teacher only on the tasks its training data covered. Where the pipeline didn't include rare-language QA, the student never closes the gap. Where the teacher itself is wrong (confabulating citations), the student inherits the same wrongness and hits the same ceiling.

Model collapse: the thing nobody wants

There's a failure mode that took a few years to articulate cleanly. If you train a model on synthetic data sampled from another model, and then later train a third model on data sampled from the second, and so on, the distribution of the outputs gets narrower and narrower. The tails fall off. Rare modes disappear. Eventually the whole thing collapses onto a handful of bland generations.

iterative training on synthetic data
G0
G1
G2
G3
G4
G5
-4-2024latent feature (diversity / topic / style)p(x)real datageneration 0
tail coverage (mass outside the central mode)
62.7% · real: 62.7%
Each generation is trained on samples drawn from the previous one. By G1 the tails are visibly thinner. By G3 the smaller modes have mostly evaporated. By G5 the distribution is a spike. The reference green dashed line is what 'real' looks like — the gap between your solid line and that dashed one is what you've lost.

The Shumailov et al. "Curse of Recursion" paper made this formal. Their argument is basically: sampling from a distribution is a lossy estimate of that distribution, and if you train the next generation on a finite sample, you underweight everything that wasn't well represented. Run this loop a few times and the dropout compounds. The model's output support shrinks until it barely covers the real support at all.

Why should you care? Two reasons.

First, iterative self-distillation as a training recipe is tempting and dangerous. If you let a team keep training smaller and smaller checkpoints on the previous one's outputs without injecting fresh real data, you walk right into collapse. The pattern is: model v1 is good, v2 is almost as good but slightly bland, v3 is noticeably narrower, v4 is unusable, and no single step in that chain looks alarming. Budget a minimum fraction of real, non-synthetic data at every generation and resist the efficiency argument that "we can just keep looping the teacher."

Second, even if you personally never do that loop, the open web is doing it for you. A significant fraction of text on the internet after 2023 is model-generated. If you scrape aggressively and train a new base model on post-2023 crawl data, you are implicitly doing one round of "train on the previous generation's outputs." The collapse math says this gets worse over time unless the field develops durable ways to mark and filter synthetic text. That's an open problem as of early 2026.

Worked example

Let me put some numbers on it. Say you want to distill a 70B-parameter chat model into a 7B base model, the pattern people have actually done for Llama, Mistral, Qwen, and so on.

You have a few knobs:

  • Data budget. 50k rows is the Alpaca scale and it produces a coherent instruction-follower. 500k to 5M rows is the Orca / Nectar scale and it produces noticeably better reasoning. Beyond that you run into diminishing returns and filtering becomes the bottleneck.
  • Teacher diversity. One teacher gives you a consistent style but also consistent blind spots. Several teachers combined (GPT-4, Claude, Gemini) widen the coverage at the cost of some stylistic incoherence.
  • KD weight α\alpha. If you have the teacher's logits, you can blend real-label cross-entropy with KL against the teacher at α∈[0.3,0.7]\alpha \in [0.3, 0.7]. No single number wins across tasks; you tune this per domain.
  • Temperature τ\tau. Standard Hinton starts at 2–4 and anneals down. Above 6 you dilute the signal so much it hurts; below 1.5 you lose most of the dark knowledge.

Compared to direct SFT with human-written data, the synthetic route gets you to a "usable instruction-follower" about 10× cheaper, because the teacher's time is free once you pay for the API tokens. Compared to training the 7B from scratch for equivalent behavior, it's something like 1000× cheaper, because you're not burning pretraining FLOPs.

The catch, again: this buys you imitation of the teacher on the slice the data covers. It does not raise the student's ceiling above the teacher's. Distillation is a fast-follow lever, not a frontier lever. If you want to push the frontier with a small model, you need something else.

Implementation sketch

Here's the minimum viable version of both flavors. One PyTorch loss function for logit distillation, one tiny wrapper for generating synthetic data. Real pipelines add batching, caching, retry logic, safety filters, and a bunch of other scaffolding, but the core loop is this small.

import torch
import torch.nn.functional as F
 
# --- Logit distillation loss (Hinton et al.) ---
 
def kd_loss(
    student_logits: torch.Tensor,   # (batch, seq, vocab)
    teacher_logits: torch.Tensor,   # (batch, seq, vocab)
    labels: torch.Tensor,           # (batch, seq)  — ground-truth ids
    tau: float = 2.0,
    alpha: float = 0.5,
    ignore_index: int = -100,
) -> torch.Tensor:
    """
    Combined cross-entropy (hard label) + KD (soft label) loss.
 
    When labels == ignore_index at a position, skip that position in the
    hard loss. The KD term uses every valid position.
    """
    # Hard-label CE on real tokens
    ce = F.cross_entropy(
        student_logits.reshape(-1, student_logits.size(-1)),
        labels.reshape(-1),
        ignore_index=ignore_index,
    )
 
    # Soft-label KL at temperature tau
    s = F.log_softmax(student_logits / tau, dim=-1)
    t = F.softmax(teacher_logits / tau, dim=-1)
    kd = F.kl_div(s, t, reduction="batchmean") * (tau ** 2)
 
    return alpha * ce + (1.0 - alpha) * kd
 
# --- Synthetic data generation (Alpaca / Self-Instruct flavor) ---
 
def generate_synthetic_row(teacher, seed_tasks: list[str]) -> dict | None:
    """
    One round of "ask the teacher for a new task similar to these seeds,
    then ask the teacher to answer it."
    Returns None when the row fails filters (dedup, length, safety).
    """
    few_shot = "\n\n".join(
        f"Task: {t}" for t in random.sample(seed_tasks, k=3)
    )
    instruction = teacher(
        f"Generate one new instruction in the same style:\n{few_shot}\nTask:"
    ).strip()
 
    if len(instruction) < 8 or len(instruction) > 400:
        return None
    if rouge_l_against_seeds(instruction, seed_tasks) > 0.7:
        return None   # too similar to an existing seed
 
    response = teacher(
        f"Follow this instruction exactly:\n{instruction}\n\nResponse:"
    ).strip()
 
    if len(response) < 4:
        return None
 
    return {"instruction": instruction, "response": response}

The kd_loss is about ten lines of real logic, which is about how big the conceptual content is. And the generator function has three filters for every one generation call, which is the right ratio. Synthetic data pipelines that skip the filters produce data that hurts more than it helps.

Misconceptions

"Distillation makes the small model as good as the big model." No. Distillation makes the small model a better imitator of the big model on the domains where you trained it. If the teacher is 80% accurate on a task and you distill cleanly, the student will land somewhere between "the small model's baseline" and 80%. The student's ceiling is bounded by the teacher. Carefully chosen data can make the gap surprisingly small; nothing makes it negative in general. When you see a headline claiming a small model "beats" a bigger one, it's almost always on a narrow benchmark that was explicitly targeted in the training mix.

"Synthetic data is free." Sure, the marginal cost per token is low, but every synthetic dataset was produced by running a teacher model at some cost. More importantly, the quality of the data is bounded by the quality of the teacher plus your filter. If your teacher is wrong about a topic, your synthetic data is wrong about that topic, and your student will be confidently wrong after training. The cost shows up as quality debt, not as a dollar line item.

"If the teacher is aligned, the student will be too." Alignment transfers weakly, at best. A teacher that politely refuses harmful prompts can produce training data that looks aligned on the surface. But if you fine-tune a base model on that data you haven't installed the underlying preferences. You've installed a surface imitation of refusals for the specific triggers that showed up in the training set. Safety work on the student model is a separate effort, usually involving RLHF or DPO on top of the distilled SFT stage. Don't skip it.

What's next

The next post covers LoRA and parameter-efficient fine-tuning, where instead of distilling a whole new model we bolt small trainable modules onto a frozen base. The cheapest way to teach an existing model a new trick without retraining it.

Additional reading (and watching)