Supervised Fine-Tuning

Same prompt, same architecture. SFT only scores loss on the response tokens, and that mask is what flips a base completer into an instruction follower.

Here's a thing about base models that surprised me when I first read about it. You take a pretrained base model, the kind that just completes text, and it's a genuinely strange object. Give it the prompt "Write a haiku about the sea." and it won't write you a haiku. It'll echo the prompt back, or start listing other prompts from whatever blog post it remembers being trained on, or go off and write a little essay about marine biology. It's not broken. It's doing exactly what it was trained to do, which is guess the next token in a document. A question floating in the middle of a document doesn't usually lead to an answer in that document. It leads to more of the document.

Then you do a modest amount of supervised fine-tuning on a few thousand (instruction, response) pairs. Nothing exotic, same loss function, same architecture, same optimizer, lower learning rate. And the thing snaps into a completely different mode. It starts behaving like an assistant. It answers questions. It follows instructions. It even knows to stop.

So what happened? Why does such a small amount of data change the behavior so drastically? And what exactly is the gradient updating to cause that flip?

Prerequisites

This post assumes you've read:

• Training View vs. Inference View for the difference between parallel training on a dataset and sequential generation at inference time.
• Cross-Entropy, KL Divergence, and What Loss Functions Measure for the per-token loss we'll be masking.
• Pretraining Data from earlier in this arc for what the base model was originally trained on.

The problem SFT solves

A pretrained base model is a next-token predictor, and that's nearly all it is. It was trained on trillions of tokens of scraped text, and its entire job was to put high probability on whatever came next.

The issue is that "whatever came next" is rarely an answer. The training distribution is mostly documents: articles, code, forum threads, recipe sites, chunks of books. When you paste a question into the middle of a document, the most plausible continuation is often more document, not a response. Base models will write plausible-looking user bios, invent follow-up questions, ramble off into irrelevant context. They have the raw capability to answer you, but the decoder doesn't know that "answering" is what you want.

SFT is the phase where we teach the model what shape of continuation we want. We show it a few thousand examples of "this is what an instruction looks like, this is what a good response looks like," and we let gradient descent do its usual thing. The model doesn't really learn new facts in SFT. A few thousand examples is almost nothing compared to the pretraining corpus. What it learns is which slice of its existing distribution to serve.

I think this framing is the one that made SFT click for me. Pretraining builds the capability. SFT picks the behavior.

The data format

Every SFT example is a pair: an instruction (what the user asks) and a response (what the model should produce). In the rawest possible form, that looks like two strings in a JSON record:

{
  "instruction": "Explain photosynthesis to a 10-year-old in one sentence.",
  "response": "Plants eat sunlight and drink water, then breathe out oxygen for us."
}

But this isn't what the model sees during training. The model sees one flat sequence of tokens. So we need to wrap the pair in some kind of template that marks where the user's turn ends and the assistant's turn begins. These wrappers go by names like "chat templates" or "instruction formats," and every model family has its own.

A few things to notice as you toggle between templates. The Alpaca format from the original Stanford release uses human-readable section headers. ChatML uses <|im_start|> and <|im_end|> as explicit boundary tokens. Llama 3 uses more structured markers with a <|eot_id|> at the end of each turn. The underlying pair is identical in all three. What differs is the marker vocabulary.

This matters for one reason that's easy to underestimate. At inference time the exact same template has to wrap the user's query or the model sees an out-of-distribution input and its behavior gets weird. If you trained on ChatML and serve the model with an Alpaca-style wrapper, you haven't broken the model, but you've put it in a slightly wrong mood. Response quality drops in subtle ways. The special tokens and chat templates post back in arc 3 goes into this more, but the basic point for SFT is: pick a format, stick to it.

Loss masking: the one thing that makes SFT work

Here's the piece that's easy to miss on a first read of the code.

During pretraining, loss is computed at every position in the sequence. Every token gets a prediction, every prediction gets compared to the next token, every position contributes to the gradient. That's the right thing when your whole corpus is documents.

During SFT, we do something different. We do a full forward pass over the entire wrapped string, exactly like training, but we only compute the loss on the assistant-response tokens. The instruction tokens get masked out.

Formally, if the full sequence is $x_1, x_2, \ldots, x_T$ and the response tokens live at positions in the set $R$, the SFT loss is:

Compare this to the standard language modeling loss where the sum is over all positions. The only difference is the set we sum over.

Why mask? Because we don't want the model to learn to generate instructions. The prompt side is input, not output. If we let the loss fire on those tokens, the model spends capacity learning the distribution of user questions, and the very first thing you see at inference is that it starts its reply by re-emitting your question. Forgetting the mask is a known way to induce this bug, and the fix is the same place the bug came from: put the mask back.

The mask also makes the training signal denser per example in a specific way. A 512-token training example might have 50 tokens of prompt and 462 tokens of response. The 50 prompt tokens still need to go through the forward pass (the response tokens need to attend to them to make sensible predictions), but they don't contribute to the loss. So one training pair gives you 462 prediction-and-loss computations, not 512. The prompt tokens are there as context, not as targets.

Worked example

Let me trace through a single SFT batch so the pieces click together.

Say we have one example: the photosynthesis pair from earlier, wrapped in ChatML. After tokenizing we get a sequence of, let's say, 47 tokens total. The first 38 are instruction + markers. The last 9 are the response.

We build a batch of size 1. We need two tensors:

• input_ids: shape (1, 47) — the full wrapped token sequence.
• labels: shape (1, 47) — usually a copy of input_ids, but with the instruction positions set to -100 so PyTorch's cross-entropy function ignores them.

We push input_ids through the model. We get logits of shape (1, 47, vocab_size). The usual shift-by-one trick applies (position $i$ predicts token $i+1$), so we compare logits at positions 0–45 against the labels at positions 1–46.

Now here's the masked part. We only get gradient from the 9 assistant-response positions. Everywhere else the label is -100, cross-entropy returns zero, the gradient is zero. The backward pass still propagates through the full model (gradients have to flow through the attention to the prompt tokens, because the response tokens read the prompt), but the gradient with respect to "predict the instruction correctly" is exactly nothing.

One loss.backward(), one optimizer.step(). The whole 7B parameter model just got nudged to be about 1% better at producing that particular response. Repeat this 1,000 to 50,000 times with varied examples and the "respond helpfully when given an instruction" behavior emerges across the board.

The surprising thing is how few gradient steps you need.

Quality over quantity, the LIMA result

Before 2023, the conventional wisdom was that more SFT data was better, and teams were building massive instruction-tuning corpora. FLAN had 1,800+ tasks and millions of examples. Alpaca had 52,000. The assumption was that SFT was like a scaled-down version of pretraining: keep piling on examples and quality goes up monotonically.

Then Meta published LIMA. Their claim was that a carefully hand-curated 1,000-example SFT set could produce a model that humans rated comparably to the same base model fine-tuned on hundreds of thousands of mixed-quality examples. Their explanation, which has aged well: pretraining is where the knowledge lives. SFT mostly teaches the model what format of response is desired, and 1,000 high-quality examples is plenty of signal for that.

A few things LIMA got right that have since become standard practice:

They aggressively deduplicated. They filtered out examples where the response was stylistically inconsistent with the rest of the set. They preferred longer, more complete responses to short ones. And they removed examples where the response disagreed with or refused the instruction.

That last one surprised me when I first read the paper. You'd think showing the model a few examples of "refuse this kind of request" would be a feature. But during SFT specifically, it muddies the signal. If every other example teaches "instruction in, thoughtful response out," a handful of "instruction in, polite refusal out" examples just adds noise to the formatting pattern. That kind of behavior shaping belongs downstream in RLHF or DPO, not SFT.

The modern consensus after LIMA, Zephyr, and Tulu is roughly: a well-filtered 5k–50k example set usually matches or beats a sloppy 500k-example set, and quality of responses dominates over quantity of prompts.

Where the data comes from

Three broad sources, and most modern recipes mix all three:

Human-written. A trained annotator reads an instruction and writes a response from scratch. Slow, expensive, gold-standard. The original InstructGPT pipeline used this. Databricks' Dolly-15k is a famous open example.

Distilled from a stronger model. You prompt GPT-4 (or Claude, or whatever the current frontier is) with a set of instructions and take its responses as training data for a smaller model. This is the main engine behind Vicuna, Zephyr, and most of the Qwen-derivative ecosystem. It's cheap and the responses are often surprisingly good. The downside is that you inherit your teacher's biases and failure modes.

Self-instruct, where the model generates its own data. The Self-Instruct recipe (Wang et al., 2022) bootstraps an instruction corpus from a tiny seed set: you prompt an existing model to generate new instructions and new responses, filter the results, and add them to the training set. Surprisingly effective. Modern variants layer this with rejection sampling and judge-model filtering.

In practice you don't pick one. Modern SFT datasets like Tulu 3 blend human-written examples for reliability, distilled data for breadth, and self-generated data for weird edge cases the other two miss.

Hyperparameters that matter

A few things consistently matter when running SFT, in rough order of how much damage you can do by getting them wrong:

Learning rate. Much lower than pretraining. Pretraining runs at something like $3 \times 10^{-4}$; SFT typically lives at $1 \times 10^{-5}$ to $5 \times 10^{-5}$. The base model already knows things. The gradient is mostly nudging it to speak in a particular voice. A too-large LR will wreck the pretrained weights and the model will start producing garbage.

Epochs. Usually 1 to 3. SFT overfits fast because the dataset is small. Past epoch 3 you're almost always hurting generalization. The non-monotonic case (epoch 2 beating both epoch 1 and epoch 3) shows up often enough that tracking held-out eval at every epoch is the standard advice.

Batch composition. Don't sort by length, don't pack documents across example boundaries without a block-diagonal attention mask, and pad with the correct pad token. The Packing and Masking post covers the packing tricks. For SFT specifically, block-diagonal masking is critical because concatenating unrelated instruction examples and letting them attend to each other is nonsense.

Prompt formatting consistency. Whatever template you use, use it everywhere. Every training example. Every eval example. Every inference call. Off-by-one-whitespace bugs in a chat template have silently degraded a lot of SFT runs.

Implementation sketch

Here's the minimal SFT training step, stripped of everything except the loss-masking logic. If you understand this snippet you understand 90% of what matters.

import torch
import torch.nn.functional as F
 
def sft_step(model, batch, optimizer):
    """
    batch["input_ids"]:    (B, T) — full wrapped sequence
    batch["labels"]:       (B, T) — same as input_ids, but
                                    prompt positions set to -100
    batch["attention_mask"]: (B, T)
    """
    input_ids      = batch["input_ids"]
    labels         = batch["labels"]
    attention_mask = batch["attention_mask"]
 
    # Full forward pass over the whole sequence.
    # Response tokens need to attend to the prompt, so we can't
    # just drop the prompt tokens.
    logits = model(input_ids, attention_mask=attention_mask).logits
    # logits: (B, T, vocab)
 
    # Standard next-token shift.
    # Position i predicts token i+1.
    shift_logits = logits[..., :-1, :].contiguous()
    shift_labels = labels[..., 1:].contiguous()
 
    # Cross-entropy ignores positions where label == -100.
    # That's how the prompt gets masked out of the loss.
    loss = F.cross_entropy(
        shift_logits.view(-1, shift_logits.size(-1)),
        shift_labels.view(-1),
        ignore_index=-100,
    )
 
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()
    return loss.item()
 
def mask_prompt_labels(input_ids, prompt_len):
    """
    Build a labels tensor where the first `prompt_len` tokens
    are set to -100 (ignored by cross-entropy).
    """
    labels = input_ids.clone()
    labels[:, :prompt_len] = -100
    return labels

The whole thing, really, is the ignore_index=-100 on cross-entropy. That one argument is what differentiates SFT from continued pretraining. Everything else (the model, the optimizer, the tokenizer, the batching) is identical.

Most real codebases compute the mask during data preprocessing (so each training record already carries its own prompt length) rather than on-the-fly. HuggingFace TRL's SFTTrainer and Axolotl both handle this for you when you pass a chat-template-formatted dataset.

Catastrophic forgetting, the thing SFT actually costs you

SFT is not free. When you fine-tune a base model to be an assistant, you also partially degrade its base capabilities. Raw knowledge benchmarks dip. Coding benchmarks dip. Multilingual recall dips. Whatever narrow distribution you tuned toward pulls the model's overall distribution with it.

The dips are usually small. A few points of MMLU, maybe a bit more on multilingual tasks if your SFT data is English-heavy. For instruction-following benchmarks like IFEval or MT-Bench, the SFT gains are enormous. So the overall trade is a clear win, but it's worth seeing it as a trade and not a pure upgrade.

Mitigations that help:

Mix in pretraining data. During SFT, blend a small fraction of your original pretraining corpus into each batch. This keeps the base distribution anchored and meaningfully reduces forgetting on general benchmarks. Llama 3's SFT recipe does something in this neighborhood.

Keep the learning rate low. As noted above, a conservative LR hurts forgetting less than an aggressive one.

Use LoRA or a PEFT method. Freeze most of the weights and train only a small adapter. The base model stays bit-for-bit unchanged; only the adapter shifts. Forgetting is structurally bounded. We'll cover this in depth in LoRA and PEFT.

Don't run too many epochs. As noted above, running past 2–3 epochs overfits the SFT distribution and accelerates forgetting on everything else.

None of these fully eliminate the effect. They shrink it.

Misconceptions

"SFT teaches the model new facts." Not really. A few thousand examples is barely a blip against the pretraining corpus. The facts you see in SFT outputs were almost always present in pretraining. The SFT part just taught the model to surface them in response to instructions rather than bury them in plausible-looking documents. If you need the model to know new facts, you either continue-pretrain on a corpus that contains them or you fetch them at inference time via RAG. SFT is the wrong tool.

"Loss masking is just an optimization." It really isn't. If you train with the loss unmasked (computing loss on both the instruction and the response), you're actively training the model to generate instruction-shaped text. At inference, it'll start its reply by re-emitting your question, or stop early, or produce other weird artifacts. The mask is what makes SFT mean what it's supposed to mean. Without it you've just done a tiny round of continued pretraining on a weird corpus.

"You need tens of thousands of examples for SFT to work." LIMA and the follow-on work strongly suggest otherwise, at least for stylistic instruction-following. A curated 1k–5k set is often enough to hit diminishing returns. Where you do need more data is for coverage of specific capabilities (code, math, tool use), because those are narrower distributions that need more examples to learn well. But a base SFT for "behave like a helpful assistant" doesn't need the 500k-example firehose.

What's next

SFT gets you a model that follows instructions. It does not get you a model that's calibrated to human preferences, knows when to refuse, or handles tone and safety well. For that you need a preference signal on top of SFT.

The next post covers the RLHF pipeline, how human preferences get compressed into a reward model, then used to steer generation via reinforcement learning, step by step.

Additional reading (and watching)

• Ouyang, L., Wu, J., Jiang, X., et al. (2022). Training language models to follow instructions with human feedback. NeurIPS 2022. The InstructGPT paper, which formalized the three-stage pipeline (SFT → reward model → PPO) that most modern post-training still follows.

• Meta AI. (2024). Llama 3 Model Card. Model card covering architecture, training data, benchmarks, and safety considerations. For the instruct template (<|start_header_id|> / <|eot_id|> conventions), see the linked llama-recipes repository.

• Chung, H. W., Hou, L., Longpre, S., et al. (2022). Scaling Instruction-Finetuned Language Models. arXiv:2210.11416. The Flan-T5 paper, which pushed the large-scale multi-task SFT hypothesis about as far as it would go before LIMA-style work pushed back.

• Zhou, C., Liu, P., Xu, P., et al. (2023). LIMA: Less Is More for Alignment. NeurIPS 2023. The "1,000 high-quality examples" paper. The specific numbers are debated, but the qualitative story (data quality dominates quantity) has held up.

• Bai, Y., Jones, A., Ndousse, K., et al. (2022). Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback. arXiv:2204.05862. Good context on why refusal and safety behaviors are typically shaped post-SFT rather than during SFT.

• Tunstall, L., Beeching, E., Lambert, N., et al. (2023). Zephyr: Direct Distillation of LM Alignment. arXiv:2310.16944. The Hugging Face open-recipe SFT + DPO paper. Good reference for a complete, reproducible distillation pipeline.

• Lambert, N., Morrison, J., Pyatkin, V., et al. (2024). Tulu 3: Pushing Frontiers in Open Language Model Post-Training. arXiv:2411.15124. AI2's fully-open post-training recipe, including detailed SFT data composition and ablations.

• Conover, M., Hayes, M., Mathur, A., et al. (2023). Free Dolly: Introducing the World's First Truly Open Instruction-Tuned LLM. Databricks blog post introducing Dolly-15k, one of the first fully human-written open SFT datasets.

• Wang, Y., Kordi, Y., Mishra, S., et al. (2022). Self-Instruct: Aligning Language Models with Self-Generated Instructions. ACL 2023. The self-bootstrapping recipe that underlies Alpaca and a lot of downstream work.

• von Werra, L., Belkada, Y., Tunstall, L., et al. (2020–2025). TRL: Transformer Reinforcement Learning. Hugging Face's canonical library for SFT, DPO, and PPO. The SFTTrainer class handles prompt masking and chat-template application for you.

• Dubey, A., Jauhri, A., Pandey, A., et al. (2024). The Llama 3 Herd of Models. arXiv:2407.21783. Meta's Llama 3 technical report. Section 4 (Post-Training) has the clearest public description of a frontier-scale SFT pipeline I'm aware of.