Supervised Fine-Tuning

A base language model can answer a question, continue a list of questions, or imitate a document containing the question. Supervised fine-tuning makes demonstrated responses more likely for the inputs we care about.

The architecture and next-token objective can stay the same. What changes is the training distribution: examples now show the desired task, format, and response. A common recipe also concentrates loss on assistant responses. Understanding that mask requires separating the tokens used as context from the tokens used as prediction targets.

Prerequisites

This post assumes you've read:

• Training View vs. Inference View for the difference between parallel training on a dataset and sequential generation at inference time.
• Cross-Entropy, KL Divergence, and What Loss Functions Measure for the per-token loss we'll be masking.
• Pretraining Data from earlier in this arc for what the base model was originally trained on.

The problem SFT solves

Pretraining supplies broad patterns and knowledge; SFT adapts the model using labeled demonstrations. Those demonstrations can teach formatting and conversational behavior, but they can also add task-specific information or capabilities.

For example, a dataset might teach an assistant to answer a question, ask for a missing detail, decline a particular request, or emit a structured tool call. All of these can be supervised targets. The useful question is whether the examples cover the behavior you want on unseen inputs.

The data format

A simple SFT example is a pair: an instruction (what the user asks) and a response (what the model should produce). In the rawest possible form, that looks like two strings in a JSON record:

{
  "instruction": "Explain photosynthesis to a 10-year-old in one sentence.",
  "response": "Plants use sunlight to turn water and carbon dioxide into sugar, releasing oxygen."
}

But this isn't what the model sees during training. The model sees one flat sequence of tokens. So we need to wrap the pair in some kind of template that marks where the user's turn ends and the assistant's turn begins. These wrappers go by names like "chat templates" or "instruction formats," and every model family has its own.

The Alpaca format from the original Stanford release uses human-readable section headers. ChatML uses <|im_start|> and <|im_end|> as explicit boundary tokens. Llama 3 uses more structured markers with a <|eot_id|> at the end of each turn. The underlying pair is identical in all three. What differs is the marker vocabulary.

Use the model’s intended chat template during training and serving. Role boundaries, end-of-turn markers, and the assistant generation prefix affect the input distribution. Mismatches can degrade behavior. See special tokens and chat templates.

Choosing which targets contribute to loss

Ordinary causal language-model training scores valid next-token targets, excluding padding and any deliberately masked boundaries. SFT uses the same prediction mechanism.

In response-only SFT, the full conversation is input context, while assistant-response tokens supply the scored targets. Full-sequence SFT, which also scores prompt tokens, is another valid training choice; masking is not the definition of SFT.

Formally, if the full sequence is $x_1, x_2, \ldots, x_T$ and the response tokens live at positions in the set $R$, the SFT loss is:

Compare this to the standard language modeling loss where the sum is over all positions. The only difference is the set we sum over.

Response-only loss allocates the objective to the behavior the assistant is meant to generate. Including prompt loss also trains the input distribution, which changes the allocation of learning signal.

Worked example

Say we have one example: a hypothetical tokenized record with 47 tokens total; these counts are not the tokenizer output of the photosynthesis sentence. The first 38 are instruction + markers. The last 9 are response targets.

We build a batch of size 1. We need two tensors:

• input_ids: shape (1, 47) — the full wrapped token sequence.
• labels: shape (1, 47) — usually a copy of input_ids, but with the instruction positions set to -100 so PyTorch's cross-entropy function ignores them.

We push input_ids through the model. We get logits of shape (1, 47, vocab_size). The usual shift-by-one trick applies (position $i$ predicts token $i+1$), so we compare logits at positions 0–45 against the labels at positions 1–46.

The nine response targets contribute local cross-entropy terms. Ignored targets contribute no direct term, but the response loss can backpropagate through prompt representations used by attention. The mask selects prediction targets, not a subset of input activations that are allowed to learn.

One loss.backward() and optimizer.step() update the trainable parameters. The resulting probability changes depend on the gradient, optimizer, learning rate, and interactions with other examples.

The LIMA experiment

LIMA fine-tuned a 65B LLaMA model on 1,000 curated prompt–response examples and reported strong human evaluations within its test setting. It showed that a small, carefully selected dataset can produce useful instruction following from a capable base.

Where the data comes from

Three broad sources, and most modern recipes mix all three:

Human-written. A trained annotator reads an instruction and writes a response from scratch. Quality depends on annotator expertise and review. The original InstructGPT pipeline used this. Databricks' Dolly-15k is a famous open example.

Distilled from a stronger model. Generate candidate responses with a teacher, validate them, and use accepted examples to train the student. This can expand coverage, but generation and review cost money, and unchecked teacher errors can become training targets.

Self-instruct, where the model generates its own data. The Self-Instruct recipe starts from seed tasks, generates instructions and responses, filters them, and adds accepted rows to the training set. Evaluate the resulting task coverage and quality separately.

Recipes can combine sources. Tulu 3 documents an SFT mixture assembled for different capabilities; its data composition and ablations provide a concrete example.

Training settings

Check these settings against held-out examples:

Learning rate. Treat full fine-tuning and adapter tuning as different regimes. Start from a relevant recipe, then compare held-out task performance and retained capabilities; no single learning-rate range fits all model sizes and methods.

Epochs. Save checkpoints and use held-out evaluations to select the number of passes. Small datasets can overfit, and the useful stopping point varies by task and model.

Batch composition. Padding targets must be ignored. If packed examples should remain independent, use the corresponding attention boundaries and position handling. Length grouping can improve efficiency; check that it does not create an unintended curriculum. See Packing and Masking.

Prompt formatting consistency. Keep the training, evaluation, and inference templates compatible. Inspect role boundaries, whitespace, end-of-turn markers, and the resulting token IDs. A rendered string alone can hide a target-mask mismatch.

Implementation sketch

This training step applies a response-only loss mask to a causal language model. Preprocessing must identify the response boundaries in labels.

import torch
import torch.nn.functional as F
 
def sft_step(model, batch, optimizer):
    """
    batch["input_ids"]:    (B, T) — full wrapped sequence
    batch["labels"]:       (B, T) — same as input_ids, but
                                    prompt and padding positions set to -100
    batch["attention_mask"]: (B, T)
    """
    input_ids      = batch["input_ids"]
    labels         = batch["labels"]
    attention_mask = batch["attention_mask"]
 
    # Full forward pass over the whole sequence.
    # Response tokens need to attend to the prompt, so we can't
    # just drop the prompt tokens.
    logits = model(input_ids, attention_mask=attention_mask).logits
    # logits: (B, T, vocab)
 
    # Standard next-token shift.
    # Position i predicts token i+1.
    shift_logits = logits[..., :-1, :].contiguous()
    shift_labels = labels[..., 1:].contiguous()
 
    if not (shift_labels != -100).any():
        raise ValueError("batch has no scored next-token targets")
 
    # Cross-entropy ignores positions where label == -100.
    # That's how the prompt gets masked out of the loss.
    loss = F.cross_entropy(
        shift_logits.view(-1, shift_logits.size(-1)),
        shift_labels.view(-1),
        ignore_index=-100,
    )
 
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()
    return loss.item()
 
def mask_prompt_labels(input_ids, prompt_lengths, attention_mask):
    # prompt_lengths counts non-padding prompt tokens, for left or right padding.
    positions = attention_mask.bool().long().cumsum(-1)
    prompt_lengths = torch.as_tensor(prompt_lengths, device=input_ids.device)
    labels = input_ids.clone()
    labels[positions <= prompt_lengths[:, None]] = -100
    labels[attention_mask == 0] = -100
    return labels
 
 

ignore_index=-100 implements the selected target mask.

Create masks during preprocessing and inspect decoded examples with their scored spans. In TRL, conversational assistant-only training uses SFTConfig(assistant_only_loss=True) and a compatible generation-aware chat template. Merely supplying a formatted conversation does not turn that setting on. Prompt-completion datasets have separate completion-loss defaults.

Checking for forgetting

Fine-tuning can reduce performance on tasks absent from the new data, but regression is not inevitable on every benchmark. The only way to know is to compare held-out evaluations before and after training.

Possible responses to a measured regression include changing the mixture, reducing update size, stopping at an earlier checkpoint, or using an adapter. A frozen base makes an adapter easy to remove; it does not bound how much the active adapter can change behavior. Replay data is another option when the original distribution and data are available.

What SFT can and cannot establish

SFT can teach facts and procedures present in its demonstrations, as well as style, safety behavior, and response structure. Learning those examples does not guarantee reliable factual recall or generalization. Retrieval can be a better fit when facts change frequently or answers need traceable evidence.

A small dataset can work when the base already has relevant capabilities and the examples cover the intended task. Broader languages, tools, and domains may require more coverage.

What's next

Demonstrations can express preferences through the answers selected for training. Pairwise comparisons provide a more explicit signal about which of two responses is preferred. The next post follows that signal through the RLHF pipeline: fitting a reward model and using it to guide policy updates.

Additional reading (and watching)

• Ouyang, L., Wu, J., Jiang, X., et al. (2022). Training language models to follow instructions with human feedback. NeurIPS 2022. The InstructGPT paper, which formalized the three-stage pipeline (SFT → reward model → PPO) used in that training recipe.

• Meta AI. (2024). Llama 3 Model Card. Model card covering architecture, training data, benchmarks, and safety considerations. For the instruct template (<|start_header_id|> / <|eot_id|> conventions), see the linked llama-recipes repository.

• Chung, H. W., Hou, L., Longpre, S., et al. (2022). Scaling Instruction-Finetuned Language Models. arXiv:2210.11416. The Flan-T5 paper, studying large-scale instruction tuning across many tasks.

• Zhou, C., Liu, P., Xu, P., et al. (2023). LIMA: Less Is More for Alignment. NeurIPS 2023. The "1,000 high-quality examples" paper. Its result concerns that model, curated dataset, and evaluation; it does not prove that quality always dominates coverage or quantity.

• Bai, Y., Jones, A., Ndousse, K., et al. (2022). Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback. arXiv:2204.05862. An account of training and evaluating helpfulness and harmlessness.

• Tunstall, L., Beeching, E., Lambert, N., et al. (2023). Zephyr: Direct Distillation of LM Alignment. arXiv:2310.16944. The Hugging Face open-recipe SFT + DPO paper. Good reference for a complete, reproducible distillation pipeline.

• Lambert, N., Morrison, J., Pyatkin, V., et al. (2024). Tulu 3: Pushing Frontiers in Open Language Model Post-Training. arXiv:2411.15124. AI2's fully-open post-training recipe, including detailed SFT data composition and ablations.

• Conover, M., Hayes, M., Mathur, A., et al. (2023). Free Dolly: Introducing the World's First Truly Open Instruction-Tuned LLM. Databricks blog post introducing Dolly-15k, one of the first fully human-written open SFT datasets.

• Wang, Y., Kordi, Y., Mishra, S., et al. (2022). Self-Instruct: Aligning Language Models with Self-Generated Instructions. ACL 2023. The self-bootstrapping recipe that underlies Alpaca and a lot of downstream work.

• von Werra, L., Belkada, Y., Tunstall, L., et al. (2020–2025). TRL: Transformer Reinforcement Learning. Hugging Face's canonical library for SFT, DPO, and PPO. Consult SFTTrainer documentation for loss-mask configuration and template requirements.

• Dubey, A., Jauhri, A., Pandey, A., et al. (2024). The Llama 3 Herd of Models. arXiv:2407.21783. Meta's Llama 3 technical report. Section 4 documents its post-training data and training stages.