Synthetic Data and Distillation

Distillation copies a frozen teacher's behavior into a smaller student — either by matching its output distribution (logit KD) or by training on the strings it samples (synthetic-data SFT).

A smaller model can be cheaper to serve but less capable on the task you need. Distillation asks whether examples or probability distributions from a stronger model can improve that smaller model enough to make it useful.

Two common approaches are logit distillation, which matches teacher distributions, and synthetic-data SFT, which trains on generated strings. Both can be called distillation.

Prerequisites

This post assumes you've read:

• Pretraining the Base Model for where the "base model" comes from.
• Supervised Fine-Tuning for what SFT is and why the (prompt, response) format matters.
• Cross-Entropy and Loss for what KL divergence is doing underneath.

The problem

Suppose a 70B teacher performs well on a task, but a 7B student would be easier to deploy. The student’s serving cost still depends on precision, context length, batching, and hardware. The question is whether teacher supervision can bring it to the required quality.

Let the teacher and student produce distributions over the same classes:

• The big model. Call it the teacher. It's frozen. Nobody's updating its weights.
• The small model. Call it the student. It has its own weights, its own loss, and we're going to train it.
• Some way of copying "what the teacher knows" into the student's weights.

Soft labels: what the teacher actually knows

A trained model’s output distribution contains more information than its top prediction alone. If you ask the teacher "what comes after the cat sat on the?" and it says mat with probability 0.42, rug at 0.24, floor at 0.14, couch at 0.09, and a long tail after that, the probabilities specify relative likelihoods for several possible continuations. The teacher thinks rug is almost as likely as mat, but much more likely than couch. That relative ordering encodes the teacher's sense of what makes sense in context.

Hinton called this "dark knowledge." The one-hot "correct" label says mat and nothing else. The full distribution also assigns probabilities to the alternatives. Matching it gives the student a training signal for those alternatives as well.

The distillation loss works on that right-hand distribution. Given the teacher's probabilities $p_{\text{teacher}}$ over the vocabulary and the student's $p_{\text{student}}$, you minimize the KL divergence between them:

The $\tau$ is a temperature applied to the softmax before you compute the KL. Higher $\tau$ flattens both distributions, which drags the student's attention away from the single mode and onto the relative ordering of secondary options. The $\tau^2$ out front is the rescaling factor Hinton derives so that the gradient magnitudes don't fall off as you warm up.

In practice you usually train the student on a weighted combination of the distillation loss and the normal cross-entropy on the ground-truth label, if you have one. Something like:

Two flavors of "use the teacher"

Logit distillation. You run the teacher and the student on the same input, grab both of their output distributions, and optimize the KL between them. This is the closest thing to "classical" knowledge distillation. Gemma 2 notably did a version of this for its 2B and 9B models at pretraining scale. A direct vocabulary-wise KL also requires aligned token vocabularies and positions. It requires the teacher’s logits, which means you need access to the model — it's easiest inside one lab, harder to do against a closed API.

Synthetic-data SFT. You sample text from the teacher, keep the tokens as ground-truth targets, and fine-tune the student with ordinary cross-entropy on those tokens. No logits needed, just the sampled strings. Alpaca, Self-Instruct, Orca, and most of what people informally call "distilling GPT-4" are this flavor. Expected cross-entropy on teacher-sampled tokens also matches a distribution: it differs from forward KL by the teacher’s fixed entropy. Finite samples, decoding choices, filtering, and changed prompts affect which distribution the student actually sees.

The Alpaca-style pipeline

Stanford Alpaca used 175 seed tasks and 52,000 instruction examples generated with text-davinci-003. The team reported under $500 for generation and under$100 for its 7B fine-tune. Its preliminary evaluation found similar behavior to the teacher on a small instruction-following test; that is not evidence of broad equivalence.

The pipeline is roughly:

1. Start with a small, hand-written set of diverse tasks (the "seed" pool).
2. Prompt the teacher to produce new tasks in the same style, either from the seeds directly or via templated few-shot prompts.
3. For each generated instruction, prompt the teacher again to produce a response.
4. Filter. Deduplication (e.g., ROUGE-L similarity cutoff), length constraints, cheap safety filters, sometimes LLM-as-judge scoring.
5. Feed the surviving rows into an SFT loop against the student.

Inspect the accepted examples as well as the rejection count. Deduplication and formatting checks do not establish factual correctness or task coverage. Reserve independently checked examples for evaluation.

Why does it work at all?

A pretrained student already has useful representations. Teacher supervision supplies targets that can help adapt those representations to the selected tasks. Capacity, prior training, coverage, and optimization affect how much transfers.

A student can exceed its teacher on a particular task; teacher accuracy is not a mathematical upper bound. The Born Again Networks experiments demonstrate this in classification. Orca’s comparison with GPT-3.5 is a different claim: Orca learned from GPT-4 explanations, so beating GPT-3.5 does not mean beating that teacher.

Unverified citations, false refusals, or stylistic habits in teacher outputs can become training targets. Independent checks and data curation give the pipeline information beyond the teacher’s own judgment.

Model collapse and rare examples

Repeatedly replacing a dataset with finite samples from the previous model can lose rare parts of the original distribution. Shumailov and colleagues study this failure under recursive training assumptions. It is not a claim that every synthetic-data pipeline inevitably collapses.

Data retention changes the problem. Gerstgrasser et al. found that accumulating original and generated data avoided collapse in their experiments, unlike replacing the old dataset. There is no universal safe percentage of real data established by these results.

Track performance on rare cases and preserve access to independently sourced data.

Worked example

Suppose a 70B teacher supplies data for a 7B student. Keep a fixed evaluation set and vary one part of the recipe at a time:

• Data budget: compare increasing numbers of validated examples while checking domain coverage and duplicates.
• Teacher choice: evaluate candidate teachers on the actual task; multiple teachers may improve coverage or introduce conflicting labels.
• KD weight: tune the blend of hard-label cross-entropy and teacher KL when both are available.
• Temperature: sweep positive values on validation data.

Include teacher inference, filtering, human review, and student training in the cost comparison. A cheap generation step does not by itself establish a cheaper route to the required quality.

Implementation sketch

The code contains a PyTorch loss for logit distillation and a generator for synthetic examples. The generator accepts caller-provided teacher, deduplication, and validation functions.

import random
import torch
import torch.nn.functional as F
 
# --- Logit distillation loss (Hinton et al.) ---
 
def kd_loss(
    student_logits: torch.Tensor,   # (batch, seq, vocab)
    teacher_logits: torch.Tensor,   # (batch, seq, vocab)
    labels: torch.Tensor,           # (batch, seq)  — ground-truth ids
    tau: float = 2.0,
    alpha: float = 0.5,
    ignore_index: int = -100,
) -> torch.Tensor:
    """Next-token CE and KD, averaged over the same scored targets.
 
    Teacher and student must use the same vocabulary and token positions.
    labels are unshifted token IDs with padding/unscored targets set to -100.
    """
    if tau <= 0 or not 0 <= alpha <= 1:
        raise ValueError("Require tau > 0 and alpha in [0, 1]")
    if teacher_logits.shape != student_logits.shape:
        raise ValueError("Teacher/student logit shapes must match")
    targets = labels[:, 1:]
    valid = targets != ignore_index
    if not valid.any():
        raise ValueError("At least one scored next-token target is required")
    student = student_logits[:, :-1, :][valid].float()
    teacher = teacher_logits[:, :-1, :][valid].detach().float()
    ce = F.cross_entropy(student, targets[valid])
    log_p = F.log_softmax(student / tau, dim=-1)
    q = F.softmax(teacher / tau, dim=-1)
    kd = F.kl_div(log_p, q, reduction="none").sum(-1).mean() * tau**2
    return alpha * ce + (1 - alpha) * kd
 
 
 
# --- Synthetic data generation (Alpaca / Self-Instruct flavor) ---
 
def generate_synthetic_row(teacher, seed_tasks: list[str],
                           is_duplicate, accept_row) -> dict | None:
    """
    One round of "ask the teacher for a new task similar to these seeds,
    then ask the teacher to answer it."
    Returns None when the row fails filters (dedup, length, safety).
    """
    if not seed_tasks:
        raise ValueError("Provide at least one seed task")
    few_shot = "\n\n".join(
        f"Task: {t}" for t in random.sample(seed_tasks, k=min(3, len(seed_tasks)))
    )
    instruction = teacher(
        f"Generate one new instruction in the same style:\n{few_shot}\nTask:"
    ).strip()
 
    if len(instruction) < 8 or len(instruction) > 400:
        return None
    if is_duplicate(instruction):
        return None   # too similar to an existing seed
 
    response = teacher(
        f"Follow this instruction exactly:\n{instruction}\n\nResponse:"
    ).strip()
 
    if len(response) < 4:
        return None
 
    row = {"instruction": instruction, "response": response}
    return row if accept_row(row) else None

The caller supplies teacher(prompt), a duplicate checker against all accepted data, and an accept_row validator appropriate to the task. The loss shifts next-token targets, detaches teacher logits, and masks both terms. It does not silently include padding in KL or sum longer sequences into a larger per-example loss.

Misconceptions

"Teacher accuracy is a hard ceiling." It is not. Distillation optimizes a training objective, while accuracy is measured on a separate task distribution. A student can improve through its existing pretraining, additional labels, regularization, or a different inductive bias. That possibility does not guarantee it will match the teacher broadly.

"Synthetic data is free." Its budget includes teacher inference, filtering, review, and student training. Cheap generation can still leave expensive errors to find. Measure cost per usable example and the quality of the trained student.

"If the teacher behaves safely, the student will too." Safety behavior needs evaluation on the student, including cases absent from the synthetic dataset. SFT can teach such behavior; additional preference training is one option, not a requirement implied by distillation itself.

What's next

The next post covers LoRA and parameter-efficient fine-tuning, which add small trainable modules to a frozen base to reduce the parameters and optimizer state updated during a fine-tune.

Additional reading (and watching)

• Hinton, G., Vinyals, O., & Dean, J. (2015). Distilling the Knowledge in a Neural Network. arXiv:1503.02531. The original knowledge-distillation paper. Introduces the temperature-scaled KL loss and the "dark knowledge" framing.

• Gemma Team. (2024). Gemma 2: Improving Open Language Models at a Practical Size. arXiv:2408.00118. Describes logit-level distillation from a larger teacher during pretraining for the 2B and 9B models.

• Taori, R., et al. (2023). Alpaca: A Strong, Replicable Instruction-Following Model. Stanford CRFM. The seed project that popularized GPT-3.5 → 7B synthetic SFT.

• Wang, Y., et al. (2023). Self-Instruct: Aligning Language Models with Self-Generated Instructions. ACL 2023. The canonical pipeline for bootstrapping instruction data from a small seed pool.

• Mukherjee, S., et al. (2023). Orca: Progressive Learning from Complex Explanation Traces of GPT-4. arXiv:2306.02707. Distilling GPT-4 reasoning traces into a 13B student.

• Abdin, M., et al. (2024). Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone. arXiv:2404.14219. The Phi-3 report describing the synthetic-textbook data recipe.

• Shumailov, I., et al. (2023). The Curse of Recursion: Training on Generated Data Makes Models Forget. arXiv:2305.17493. The formal treatment of model collapse under iterative self-training.

• Dohmatob, E., et al. (2024). A Tale of Tails: Model Collapse as a Change of Scaling Laws. ICML 2024. Shows how model collapse interacts with pretraining scaling laws when synthetic data contaminates the mix.

• Gunasekar, S., et al. (2023). Textbooks Are All You Need. arXiv:2306.11644. The original Phi-1 paper introducing the synthetic-textbook pretraining idea.

• Zhang, H., Da, J., Lee, D., et al. (2024). A Careful Examination of Large Language Model Performance on Grade School Arithmetic. arXiv:2405.00332. Discussion of benchmark contamination issues, including the Phi family's reported scores.

• DeepSeek-AI. (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv:2501.12948. Includes six smaller models trained on roughly 800k curated examples, including reasoning and non-reasoning data.

• Long, L., et al. (2024). On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey. arXiv:2406.15126. Helpful survey of the current synthetic-data landscape across SFT, distillation, and evaluation uses.

• Furlanello, T., et al. (2018). Born Again Neural Networks. Demonstrates students exceeding their teachers on evaluated classification tasks.

• Gerstgrasser, M., et al. (2024). Is Model Collapse Inevitable?. Compares data replacement with accumulation across generations.