DPO and Its Variants: Skipping the Reward Model

RLHF works. It has also accumulated a lot of scar tissue. You train a reward model on human preferences, then fine-tune a policy against that reward using PPO, while clipping against a reference with a KL penalty so the policy doesn't run off into nonsense. Three models in memory. Three training loops. Two sets of hyperparameters that interact in unintuitive ways. And a reward model that drifts off whenever the policy drifts too far from the distribution it was trained on.

It is a lot of machinery for "take some pairs of responses that humans ranked and shape the model toward the preferred ones." Does it have to be this complicated?

Rafailov et al.'s Direct Preference Optimization from 2023 says no. The move is algebraically clean. They show that the optimal RLHF policy, the one that maximizes expected reward while staying close to the reference in KL, can be written as a closed-form function of the reference policy and the reward. That relationship is invertible: you can express the reward as a function of the policy. Plug that into the preference likelihood and the reward model disappears. What's left is a supervised loss over log-probabilities. One model, one loop, no PPO.

In practice it turned out to be trickier than the paper makes it sound, and a bunch of "almost-DPO" variants have grown up around it. I want to build the intuition for the original loss from first principles, then walk through the main variants (IPO, KTO, ORPO), and then be honest about when DPO-style methods match RLHF and when they don't.

Prerequisites

This post assumes you've read:

• The RLHF pipeline for what we're simplifying away from.
• Supervised fine-tuning for the SFT model that becomes π_ref.
• Cross-entropy and loss for why log-probability ratios have the shape they do.
• Gradients and how machines learn for the one-loop picture.

Very little math in this post is actually new. What's new is the way a few familiar pieces snap together.

The magic trick, in five lines

The whole derivation is short enough to fit on one page if you don't dwell on the motivation. Let me lay it out flat, then spend the rest of the section pointing at the parts that matter.

max  E[ r(x, y) ]  −  β · KL(π_θ || π_ref)

π*(y | x)  =  (1 / Z(x)) · π_ref(y | x) · exp( r(x, y) / β )

r(x, y)  =  β · log ( π*(y | x) / π_ref(y | x) )  +  β · log Z(x)

P(y_w ≻ y_l | x)  =  σ( r(x, y_w) − r(x, y_l) )

L_DPO  =  − log σ(  β · log π_θ(y_w|x)/π_ref(y_w|x)  −  β · log π_θ(y_l|x)/π_ref(y_l|x)  )

Here's the one sentence version. RLHF's optimal policy is the reference distribution tilted by exp(reward/β). That tilt can be inverted: the reward is β times the log-ratio between policy and reference, plus a term that only depends on the prompt. Bradley–Terry turns a reward difference into a preference probability; the prompt-only term cancels in the subtraction; and you're left with a sigmoid of a difference of log-ratios. Training that sigmoid as a binary classifier with cross-entropy is the DPO loss.

Concretely, for a single preference pair $(x, y_w, y_l)$ where $y_w$ is the chosen response and $y_l$ is the rejected one:

Read that slowly. Four log-probabilities: the policy on the chosen response, the reference on the chosen response, the policy on the rejected response, the reference on the rejected response. Two log-ratios. One subtraction times β. Sigmoid, negative log.

The things that vanished are the reward model and the PPO loop. What replaced them is a clever reparameterization. The reward isn't gone; it's sitting inside the log-ratio $\beta \log \pi_\theta(y | x) / \pi_{\text{ref}}(y | x)$, which is the implicit reward DPO is optimizing. If you want to know what reward your DPO-trained model has learned, you compute exactly that quantity and treat it like a scalar reward on any (x, y) pair.

There's a subtle thing I want to flag. The step where the partition function cancels is only valid if you can write the reward as a function of (x, y) that doesn't depend on any other response in the batch. That's true in Bradley–Terry because you're comparing two responses at a time. It wouldn't be true for a Plackett–Luce ranking over five responses where the normalization mixes all of them. That's one of the reasons DPO variants for richer feedback modalities look different from the plain binary case.

What a DPO training step actually does

Let me switch from algebra to what happens on the GPU.

You have a batch of preference pairs. For each one, you run four forward passes: the policy on $y_w$, the policy on $y_l$, the reference on $y_w$, and the reference on $y_l$. The reference passes are with no_grad and often cached. The reference doesn't change during training, so once you've computed those log-probs for a given dataset, you can store them and skip those forward passes entirely in subsequent epochs.

You get four log-probabilities per pair. You compute two ratios, subtract, scale by β, pass through log-sigmoid, take the negative mean over the batch, and backprop. Exactly the computational shape of a supervised fine-tuning step, just with a different loss function.

First, β in the loss controls how much weight the loss puts on divergence from the reference. Small β (like 0.01–0.1) means small moves per update and strong anchoring to π_ref. Large β (like 1.0) means the loss becomes more willing to move the policy aggressively. In practice DPO papers almost always use β around 0.1.

Second, notice what happens to a response that the reference already likes and the policy now likes even more. The log-ratio is large and positive. For the chosen side that's great; the loss is happy. But look at the loss landscape for that case: the log-sigmoid has a vanishingly small gradient. Far out along the margin, the loss stops pushing. That sounds like a feature (stop overfitting), but the derivative is still nonzero and SGD will happily keep nudging. On long generations with many pairs clearly ranked, this nudging accumulates and the policy drifts.

Third: the reference is frozen. It sits in memory, but it never updates. This is a structural thing that matters for a later misconception. DPO's "KL regularization" is not the same KL regularization as PPO's. It's implicit, baked into the loss, and it only exists as a pressure toward the reference's distribution on responses in your preference dataset. The policy is completely free to do whatever it wants on prompts and responses that never show up in the pairs.

Worked example

Let me put numbers on this with one concrete pair.

Say the prompt is "Explain transformers in one sentence." The chosen response $y_w$ is a reasonable one-sentence explanation. The rejected response $y_l$ is a joke about robots from Michael Bay movies. We pass each through both the current policy and the frozen reference, and we get four log-probabilities.

The numbers won't look impressive in isolation. A loss of 0.57 and a preference probability of 57% don't sound like a strong signal. But remember, DPO runs this over millions of pairs. Each gradient step moves the four relevant log-probs by a tiny amount. The magic is that those tiny moves are targeted: they push up log π_θ(y_w) and push down log π_θ(y_l) proportional to how wrong the current ordering is. Across a whole dataset, those targeted nudges add up to a policy that consistently prefers the kinds of responses humans ranked higher.

The minimal implementation

If you've ever written an SFT training loop, the DPO loop is surprisingly close. The loss function is the one part that needs thought. Here's the whole thing, stripped of batching tricks and reference-caching machinery.

import torch
import torch.nn.functional as F
 
def dpo_loss(
    policy_chosen_logps: torch.Tensor,   # log π_θ(y_w | x)   shape: (batch,)
    policy_rejected_logps: torch.Tensor, # log π_θ(y_l | x)   shape: (batch,)
    ref_chosen_logps: torch.Tensor,      # log π_ref(y_w | x) shape: (batch,)
    ref_rejected_logps: torch.Tensor,    # log π_ref(y_l | x) shape: (batch,)
    beta: float = 0.1,
) -> torch.Tensor:
    """
    DPO loss for a batch of preference pairs.
 
    Each input is a 1-D tensor of per-sequence log-probabilities. You get those
    by summing per-token log-probs over the response tokens (not the prompt).
    """
    # Log-ratio of policy vs reference on the chosen response
    chosen_logratio = policy_chosen_logps - ref_chosen_logps
    # Log-ratio of policy vs reference on the rejected response
    rejected_logratio = policy_rejected_logps - ref_rejected_logps
 
    # The implicit-reward margin
    margin = beta * (chosen_logratio - rejected_logratio)
 
    # DPO loss: -log σ(margin) = softplus(-margin)
    loss = -F.logsigmoid(margin).mean()
 
    # Handy to log during training: what fraction of pairs the policy ranks correctly
    with torch.no_grad():
        accuracy = (margin > 0).float().mean()
 
    return loss, accuracy
 
# --- minimal training step ---
def dpo_step(policy, ref, batch, optimizer, beta=0.1):
    """
    batch is a dict with:
      - prompt_ids, chosen_ids, rejected_ids (all token tensors)
      - chosen_logp_mask, rejected_logp_mask (0 for prompt, 1 for response)
    """
    # Forward pass — policy gets gradients, reference does not.
    p_chosen = sequence_logp(policy, batch["chosen_ids"], batch["chosen_mask"])
    p_rejected = sequence_logp(policy, batch["rejected_ids"], batch["rejected_mask"])
 
    with torch.no_grad():
        r_chosen = sequence_logp(ref, batch["chosen_ids"], batch["chosen_mask"])
        r_rejected = sequence_logp(ref, batch["rejected_ids"], batch["rejected_mask"])
 
    loss, acc = dpo_loss(p_chosen, p_rejected, r_chosen, r_rejected, beta=beta)
 
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()
    return loss.item(), acc.item()

The whole DPO loss function is seven lines. The training loop is three more. Compare that to a working PPO-RLHF setup with a reward model, an old-policy buffer, value estimation, advantage computation, clipping, and KL tracking. The appeal of the simpler loop is obvious.

Where DPO falls short, and the variants that patch it

DPO works well, but it doesn't work perfectly. Two clear failure modes have driven most of the variant literature.

The first is overfitting to preference margins. The log-sigmoid in DPO's loss never saturates. On pairs where the policy already prefers $y_w$ strongly, the loss is small but nonzero, and its gradient keeps pushing the margin wider. In practice this looks like the policy becoming over-confident on the training pairs (margins going to 5, 6, 8+) while generalization degrades. Azar et al. diagnose this explicitly in their IPO paper and propose replacing the log-sigmoid with a squared distance from a target margin τ:

When the margin reaches τ, the loss bottoms out and stops pushing. The policy is allowed to learn "prefer y_w over y_l by some amount," not "push the margin to infinity."

The second is data format. DPO needs preference pairs. A lot of real feedback is thumbs-up / thumbs-down on single responses, not A-vs-B comparisons. Ethayarajh et al.'s KTO adapts the loss for this setting by drawing on prospect theory from behavioral economics: gains from chosen samples and losses from rejected samples are weighted asymmetrically (usually with losses weighing more, matching the empirical finding that humans are loss-averse). You can run KTO on a dataset that's just (response, thumbs-up-or-down), no pairing required. For a lot of production setups where you're collecting user feedback naturally, this format-friendliness matters more than the exact loss shape.

A third direction, taken by Hong et al.'s ORPO, is to ask whether you even need a reference model at all. ORPO combines the SFT loss and a preference signal into a single objective that uses an odds-ratio penalty. No separate reference forward pass. No two-stage pipeline. You do SFT and preference alignment in the same training run. The trade-off is that ORPO gives up the principled RLHF derivation; it's more of a pragmatic regularizer than a closed-form solution to an objective. But on small models with modest compute, it frequently matches or beats staged DPO with half the moving parts.

There's also SimPO from Meng et al., which drops the reference entirely and uses a length-normalized log-prob margin. It trades some of the theoretical grounding for lower memory footprint (no reference model in memory) and often comparable empirical results. If I had to pick one "useful for practitioners" variant, it would be either SimPO or ORPO, depending on whether you want a reference-free DPO-style loss or a reference-free combined SFT+alignment loss.

These are tools with different trade-offs, not competitive benchmarks waiting to resolve. IPO is useful for overfitting-prone setups, KTO for unpaired feedback, and ORPO or SimPO for smaller models where you're paying attention to wall-clock and memory.

When DPO matches RLHF, and when it doesn't

The cleanest statement I've seen about this comes from Tajwar et al.: preference fine-tuning benefits most from on-policy data, especially in the regime where the policy has enough capacity to be making mistakes the reference model wouldn't. DPO uses offline preference data. The pairs were collected before training started, maybe by ranking outputs from a different model entirely. PPO-RLHF samples from the current policy, ranks those samples with the reward model, and uses the rewards to update. On any prompt where the current policy generates something the offline pairs don't cover, DPO has no gradient signal. PPO does, via the reward model extrapolating from the pairs it was trained on.

This matters most on long generations. The longer the generation, the larger the space of possible outputs, and the more quickly the policy can drift into regions not represented in the offline preference set.

This doesn't mean DPO is broken. For short responses, standard helpfulness/harmlessness fine-tuning, and preference datasets that were collected on something close to the model you're training, DPO matches PPO-RLHF closely on most benchmarks at a fraction of the engineering cost. That's why the open community, starting with Zephyr-7B in late 2023 and continuing through Tulu 2, OpenHermes, and a lot of the Nous/Argilla recipes, has standardized on DPO as the default alignment method.

For frontier labs with lots of compute and reasoning-heavy tasks where the policy can wander far from the reference during generation, the picture is more nuanced. OpenAI, Anthropic, and Google have all kept RLHF-style online components in their post-training stacks, often mixed with DPO-style offline passes. The frontier recipe as of 2026 is typically: SFT → one or more rounds of DPO/IPO for broad alignment → one or more rounds of PPO-style online RL for the domains where online matters most (code, math, agentic tasks). The industry didn't "switch to DPO"; it absorbed DPO into a larger toolkit.

Misconceptions

"DPO doesn't need a reward model, so it's reward-model-free." Not really. DPO trains an implicit reward. The quantity $\beta \log \pi_\theta(y | x) / \pi_{\text{ref}}(y | x)$ is the reward your model has learned; that's not metaphor, it's what the derivation gives you. Rafailov's paper shows this explicitly: you can take a DPO-trained model and extract a reward function from it that ranks responses coherently. The reward model didn't disappear. It became the policy.

"DPO is just supervised fine-tuning on preferences." The forward pass looks like SFT, but the loss function is not an SFT loss. SFT maximizes log-probability of the target. DPO maximizes the log-odds ratio of the chosen vs rejected response's policy-to-reference log-prob ratios. Those are very different objectives. An SFT loss on $y_w$ alone would happily push the probability of all reasonable completions up, with no push-down on rejected responses. DPO's combined push-up-and-push-down is what makes it a preference method.

"You can run DPO without an SFT stage." Usually a worse outcome. The reference model needs to be a reasonable starting point. Without that anchor, the "push up on chosen, push down on rejected" dynamic can send the policy into degeneracy. Almost every DPO recipe in production is SFT → DPO, not DPO-from-base-model. ORPO is the one that deliberately collapses these two stages, and it does so by adding a term that behaves like the SFT loss anyway.

"DPO's β is the same as PPO's KL penalty coefficient." They play analogous roles, in that both control how far the policy can drift from the reference. But they do different things. PPO's KL term penalizes divergence from the reference on the current policy's samples at each step. DPO's β scales the implicit reward margin and only regularizes on the preference pairs in the dataset. A DPO policy is free to be wildly different from π_ref on prompts no one labeled. This is closely related to why DPO struggles with distribution shift on long generations.

What's next

The next post covers Constitutional AI and RLAIF, where we replace human preference labelers with model-generated critiques guided by explicit principles, and see what it buys us at scale.

Additional reading (and watching)

• Rafailov, R., et al. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. NeurIPS 2023. The original DPO paper. The derivation in section 4 is the one I walked through.

• Azar, M. G., et al. (2024). A General Theoretical Paradigm to Understand Learning from Human Feedback. AISTATS 2024. Introduces IPO and identifies DPO's preference-margin overfitting as a structural problem.

• Ethayarajh, K., et al. (2024). KTO: Model Alignment as Prospect Theoretic Optimization. ICML 2024. Makes the case for prospect-theoretic (asymmetric, bounded) loss functions and shows they work with unpaired thumbs-up/thumbs-down data.

• Hong, J., et al. (2024). ORPO: Monolithic Preference Optimization without Reference Model. EMNLP 2024. Combines SFT and preference alignment into a single objective using an odds-ratio penalty.

• Meng, Y., et al. (2024). SimPO: Simple Preference Optimization with a Reference-Free Reward. NeurIPS 2024. A length-normalized, reference-free variant that matches or beats DPO on most open-weight benchmarks with a smaller memory footprint.

• Tajwar, F., et al. (2024). Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data. ICML 2024. The cleanest empirical study of when offline (DPO-style) and online (PPO-style) preference methods diverge, and how on-policy data closes the gap.

• Ivison, H., et al. (2024). Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback. NeurIPS 2024. Careful head-to-head experiments that separate the effects of algorithm choice, data quality, and reward modeling.

• Tunstall, L., et al. (2023). Zephyr: Direct Distillation of LM Alignment. arXiv:2310.16944. The paper behind Zephyr-7B, the proof that DPO on open-weight base models could match larger RLHF-trained chat models.

• Lambert, N., et al. (2024). Tulu 3: Pushing Frontiers in Open Language Model Post-Training. arXiv:2411.15124. Allen AI's multi-stage post-training recipe: SFT → DPO on on-policy-ish data → a GRPO-style RL stage. A good case study in mixing preference methods.

• Guo, S., et al. (2024). Direct Language Model Alignment from Online AI Feedback. arXiv:2402.04792. Bridges DPO and online RLHF by sampling pairs from the current policy and having a model-as-judge label them on the fly.