DPO and Its Variants: Skipping the Reward Model

Reward-model-based RLHF requires preference-model training, policy rollouts, and policy optimization. PPO compares the updated policy with the rollout policy; a separate KL penalty can compare it with a frozen reference. A critic may add another trainable model. Can we use preference pairs without that online optimization loop?

Rafailov et al.'s Direct Preference Optimization (2023) shows that the optimal RLHF policy, the one that maximizes expected reward while staying close to the reference in KL, can be written as a closed-form function of the reference policy and the reward. That relationship is invertible: you can express the reward as a function of the policy. Plug that into the preference likelihood and the reward model disappears. What's left is a supervised loss over log-probabilities. One trainable policy and a frozen reference (or cached reference scores), without PPO.

Prerequisites

This post assumes you've read:

• The RLHF pipeline for what we're simplifying away from.
• Supervised fine-tuning for the SFT model that becomes π_ref.
• Cross-entropy and loss for why log-probability ratios have the shape they do.
• Gradients and how machines learn for the one-loop picture.

From a regularized reward objective to preference loss

max  E[ r(x, y) ]  −  β · KL(π_θ || π_ref)

π*(y | x)  =  (1 / Z(x)) · π_ref(y | x) · exp( r(x, y) / β )

r(x, y)  =  β · log ( π*(y | x) / π_ref(y | x) )  +  β · log Z(x)

P(y_w ≻ y_l | x)  =  σ( r(x, y_w) − r(x, y_l) )

L_DPO  =  − log σ(  β · log π_θ(y_w|x)/π_ref(y_w|x)  −  β · log π_θ(y_l|x)/π_ref(y_l|x)  )

RLHF's optimal policy is the reference distribution tilted by exp(reward/β). That tilt can be inverted: the reward is β times the log-ratio between policy and reference, plus a term that only depends on the prompt. Bradley–Terry turns a reward difference into a preference probability; the prompt-only term cancels in the subtraction; and you're left with a sigmoid of a difference of log-ratios. Training that sigmoid as a binary classifier with cross-entropy is the DPO loss.

Concretely, for a single preference pair $(x, y_w, y_l)$ where $y_w$ is the chosen response and $y_l$ is the rejected one:

The loss uses four log-probabilities: policy and reference scores for each response. Subtract the two log-ratios, scale by β, then apply the negative log-sigmoid.

DPO replaces separate reward-model training and PPO with a loss that uses the log-ratio $\beta \log \frac{\pi_\theta(y | x)}{\pi_{\text{ref}}(y | x)}$, which is the implicit reward DPO is optimizing. If you want to know what reward your DPO-trained model has learned, you compute exactly that quantity and treat it like a scalar reward on any (x, y) pair.

The prompt-only normalization cancels because all compared responses share the same prompt. This also works under the Plackett–Luce ranking model: the original DPO paper derives that extension in Appendix A.3. Pairwise Bradley–Terry is the simpler case, not the only one where cancellation is possible.

What a DPO training step actually does

You have a batch of preference pairs. For each one, you run four response scores, often computed in two batched forward passes: the policy on $y_w$, the policy on $y_l$, the reference on $y_w$, and the reference on $y_l$. The reference passes are with no_grad and often cached. The reference doesn't change during training, so once you've computed those log-probs for a given dataset, you can store them and skip those forward passes entirely in subsequent epochs.

You get four log-probabilities per pair. You compute two ratios, subtract, scale by β, pass through log-sigmoid, take the negative mean over the batch, and backprop. Exactly the computational shape of a supervised fine-tuning step, just with a different loss function.

In the regularized reward objective used to derive DPO, larger $\beta$ means a stronger KL penalty and an optimum closer to the reference for fixed rewards. In the implemented loss, $\beta$ also scales the margin and its gradients, so changing it is not simply changing the learning rate.

For a separable pair, logistic loss approaches zero as the margin grows, and its gradient approaches zero too. There is no finite preferred margin, though finite training, conflicting pairs, and parameter sharing affect the result. A larger training margin does not by itself establish better held-out behavior.

The reference stays fixed, but DPO does not explicitly measure a global KL on fresh policy samples at each update. Its observed training signal comes from dataset pairs; behavior outside their coverage depends on generalization and parameter sharing.

Worked example

Say the prompt is "Explain transformers in one sentence." The chosen response $y_w$ is a reasonable one-sentence explanation. The rejected response $y_l$ is a joke about robots from Michael Bay movies. We pass each through both the current policy and the frozen reference, and we get four log-probabilities.

The four log probabilities are assumed values. Only the two policy scores have gradients; the reference scores are fixed. For this pair, lowering the loss increases the relative margin, though the realized changes in either response probability depend on all shared model parameters and other examples.

The minimal implementation

The code below computes response scores and the pairwise loss, then updates the policy while keeping the reference fixed.

import torch
import torch.nn.functional as F
 
def dpo_loss(
    policy_chosen_logps: torch.Tensor,   # log π_θ(y_w | x)   shape: (batch,)
    policy_rejected_logps: torch.Tensor, # log π_θ(y_l | x)   shape: (batch,)
    ref_chosen_logps: torch.Tensor,      # log π_ref(y_w | x) shape: (batch,)
    ref_rejected_logps: torch.Tensor,    # log π_ref(y_l | x) shape: (batch,)
    beta: float = 0.1,
) -> tuple[torch.Tensor, torch.Tensor]:
    """
    DPO loss for a batch of preference pairs.
 
    Each input is a 1-D tensor of per-sequence log-probabilities. You get those
    by summing per-token log-probs over the response tokens (not the prompt).
    """
    # Log-ratio of policy vs reference on the chosen response
    chosen_logratio = policy_chosen_logps - ref_chosen_logps.detach()
    # Log-ratio of policy vs reference on the rejected response
    rejected_logratio = policy_rejected_logps - ref_rejected_logps.detach()
 
    # The implicit-reward margin
    margin = beta * (chosen_logratio - rejected_logratio)
 
    # DPO loss: -log σ(margin) = softplus(-margin)
    loss = -F.logsigmoid(margin).mean()
 
    # Fraction of pairs with positive implicit-reward margin, relative to reference
    with torch.no_grad():
        accuracy = (margin > 0).float().mean()
 
    return loss, accuracy
 
def sequence_logp(model, ids, response_mask, attention_mask):
    logits = model(ids, attention_mask=attention_mask).logits[:, :-1].float()
    token_logps = logits.log_softmax(-1).gather(-1, ids[:, 1:, None]).squeeze(-1)
    mask = response_mask[:, 1:].bool() & attention_mask[:, 1:].bool()
    if not mask.any(-1).all():
        raise ValueError("every response needs a scored next-token target")
    return (token_logps * mask).sum(-1)
 
# Each chosen/rejected IDs tensor contains prompt + response.
# The corresponding response masks select targets; attention masks exclude padding.
def dpo_step(policy, ref, batch, optimizer, beta=0.1):
    policy.eval()  # disables dropout; eval mode still allows gradients
    ref.eval()
    def score(model, side):
        return sequence_logp(model, batch[f"{side}_ids"],
                             batch[f"{side}_mask"], batch[f"{side}_attention_mask"])
    p_chosen, p_rejected = score(policy, "chosen"), score(policy, "rejected")
    with torch.no_grad():
        r_chosen, r_rejected = score(ref, "chosen"), score(ref, "rejected")
    loss, acc = dpo_loss(p_chosen, p_rejected, r_chosen, r_rejected, beta)
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()
    return loss.item(), acc.item()
 
 

The helper sums response-token log probabilities and disables dropout while retaining policy gradients.

Where DPO falls short, and the variants that patch it

Variants of DPO change the margin objective or the feedback format.

The first is overfitting to preference margins. Logistic loss has no finite minimizer on a perfectly separable pair, even though its gradient tends to zero for a large positive margin. IPO uses a squared objective with a finite target. In the unscaled log-ratio convention below, that target is $\tau=1/(2\beta)$; practical implementations can also change sequence-length normalization:

When the margin reaches τ, the loss bottoms out and stops pushing. The policy is allowed to learn "prefer y_w over y_l by some amount," not "push the margin to infinity."

The second is data format. KTO supports unpaired desirable/undesirable feedback. It uses reference-relative rewards, a KL-based reference point, and weights for the two label types. Those weights are configuration choices, not a requirement that every rejected example always count more than every chosen one.

ORPO combines chosen-response likelihood with an odds-ratio preference term, avoiding a separate reference policy.

SimPO uses length-normalized response log probabilities and a target margin without a separate reference policy.

Compare these methods’ memory use and held-out performance with model, data, and training budget controlled.

When DPO matches RLHF, and when it doesn't

Offline DPO repeatedly trains on a fixed set of labeled pairs. If current outputs differ substantially from those pairs, evaluation on the training set may say little about current behavior. Online sampling can expose those outputs for new labels or reward estimates, but the feedback may still be wrong outside the judge’s training distribution.

The distinction is not permanently attached to the optimizer: DPO can be run iteratively on fresh pairs, and PPO can optimize an unreliable reward model. Published comparisons need to separate algorithm, sampling distribution, data quality, and reward-model quality. Tulu 3 is a concrete example combining SFT, DPO, and reinforcement learning with verifiable rewards.

Misconceptions

"DPO has no reward interpretation." It does not train a separate reward network. The derivation gives an implicit reward, $\beta \log[\pi_\theta(y|x)/\pi_{\text{ref}}(y|x)]$, up to a prompt-dependent constant. That score can rank responses, but the quality of those rankings outside the training pairs still needs evaluation.

"DPO is just SFT on chosen answers." SFT maximizes target log probability. DPO fits a preference margin involving chosen and rejected responses relative to a reference. Neither objective guarantees that all reasonable completions increase in probability; normalized distributions and shared parameters couple them.

"DPO requires a separate SFT stage mathematically." It needs a policy and reference with suitable support, not a theorem-mandated SFT stage. Starting from an instruction-tuned model is common and can help. Whether DPO from a different starting point works is an empirical question.

"DPO's β is the same as PPO's KL penalty coefficient." They play analogous roles, in that both control how far the policy can drift from the reference. But they do different things. PPO's KL term penalizes divergence from the reference on the current policy's samples at each step. DPO's β scales the implicit reward margin and only regularizes on the preference pairs in the dataset. A DPO policy is free to be wildly different from π_ref on prompts no one labeled. This is closely related to why DPO struggles with distribution shift on long generations.

What's next

The next post covers Constitutional AI and RLAIF, where written principles guide model critiques and some preference labels, alongside human feedback.

Additional reading (and watching)

• Rafailov, R., et al. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. NeurIPS 2023. The original DPO paper. Section 4 derives the implicit reward and preference objective used above.

• Azar, M. G., et al. (2024). A General Theoretical Paradigm to Understand Learning from Human Preferences. AISTATS 2024. Introduces IPO and identifies DPO's preference-margin overfitting as a structural problem.

• Ethayarajh, K., et al. (2024). KTO: Model Alignment as Prospect Theoretic Optimization. ICML 2024. Makes the case for prospect-theoretic (asymmetric, bounded) loss functions and shows they work with unpaired thumbs-up/thumbs-down data.

• Hong, J., et al. (2024). ORPO: Monolithic Preference Optimization without Reference Model. EMNLP 2024. Combines SFT and preference alignment into a single objective using an odds-ratio penalty.

• Meng, Y., et al. (2024). SimPO: Simple Preference Optimization with a Reference-Free Reward. NeurIPS 2024. A length-normalized, reference-free variant that evaluated against DPO in the authors’ experiments.

• Tajwar, F., et al. (2024). Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data. ICML 2024. An empirical comparison of offline and online preference methods, including the effect of on-policy data.

• Ivison, H., et al. (2024). Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback. NeurIPS 2024. Careful head-to-head experiments that separate the effects of algorithm choice, data quality, and reward modeling.

• Tunstall, L., et al. (2023). Zephyr: Direct Distillation of LM Alignment. arXiv:2310.16944. The paper behind Zephyr-7B, with comparisons to larger chat models in the authors’ evaluations.

• Lambert, N., et al. (2024). Tulu 3: Pushing Frontiers in Open Language Model Post-Training. arXiv:2411.15124. Allen AI's multi-stage post-training recipe: SFT → DPO using model-generated candidates → reinforcement learning with verifiable rewards. A good case study in mixing preference methods.

• Guo, S., et al. (2024). Direct Language Model Alignment from Online AI Feedback. arXiv:2402.04792. Bridges DPO and online RLHF by sampling pairs from the current policy and having a model-as-judge label them on the fly.