The RLHF Pipeline: Reward Models and PPO

So we have a supervised fine-tuned model from the previous post. It answers questions in the right format. It follows instructions, mostly. It knows what a chat turn looks like. What it does not know, and really cannot know from SFT alone, is which of its own valid-looking responses humans prefer.

I think this is the key unlock. SFT teaches the model to produce outputs that look like the training distribution. "Look like" is not the same as "is better than the alternative." If a prompt has ten reasonable responses and the model learned one demonstration of the format, the model has no signal about why that one was chosen and not the other nine.

RLHF is the field's answer to that gap. The pipeline is doing one specific thing: turning pairwise human judgments ("this response was better than that one") into a gradient signal that pushes the language model toward the preferred distribution. Everything about reward models and PPO and KL penalties is plumbing to make that work without blowing up the model you started with.

Prerequisites

This post assumes you've read:

• Next-Token Prediction for the base-model setup and next-token loss.
• Supervised Fine-Tuning for the SFT stage we build on top of.
• Attention From Scratch for what "policy" and "logits" mean in transformer terms.
• A cross-entropy / softmax comfort level. Policy gradients will make more sense if you've seen $\nabla \log \pi$ before, but I'll motivate it as we go.

The shape of the pipeline

Here's the pipeline in one sentence: take an SFT model, sample responses from it, ask humans which is better, fit a reward model to those preferences, then use the reward model to steer the language model with reinforcement learning.

Four moving pieces:

1. SFT model ($\pi_{\text{SFT}}$). Already trained. The anchor. It's frozen during RLHF as the reference point and never touched again.
2. Preference dataset. Pairs $(x, y_w, y_l)$ where $y_w$ is the chosen response and $y_l$ is the rejected one, given the same prompt $x$.
3. Reward model ($r_\phi$). Usually the same architecture as the LM, with the language-modeling head swapped out for a single scalar. Trained to rank the chosen response above the rejected one.
4. Policy ($\pi_\theta$). The thing we want. It starts as a copy of $\pi_{\text{SFT}}$, and PPO gradually updates it to produce responses the reward model likes. The KL penalty tethers it back to $\pi_{\text{SFT}}$ so it does not drift off a cliff.

The hero loop above cycles through all four. Watch the PPO step: reward flows in from the right, the policy gets an update, and then a KL tether pulls it back toward the SFT box. That tether is doing more work than it looks like.

Reward model: turning "A > B" into a scalar

Humans are terrible at assigning absolute scores to language model outputs. Ask ten people to rate "how good is this answer on a 1-10 scale" and you'll get ten different calibrations. Ask them "which of these two answers is better?" and agreement jumps.

So we don't ask for absolute scores. We ask for comparisons. Given a prompt $x$ and two candidate responses $y_a$ and $y_b$, a labeler picks one. That's the entire data format: $(x, y_w, y_l)$, where $w$ means "won" and $l$ means "lost."

The reward model learns to assign a scalar $r_\phi(x, y)$ to any (prompt, response) pair, such that the chosen response scores higher than the rejected one. The objective comes from the Bradley–Terry preference model, which has been the workhorse for pairwise ranking for over seventy years. The model says: the probability that $y_w$ is preferred over $y_l$ is

where $\sigma$ is the logistic sigmoid. That should feel familiar: it's a binary-classification head on the reward gap.

The training loss is the negative log-likelihood of the observed preferences:

No elaborate multi-headed ranking objective, no listwise anything. Just a logistic loss on the score gap.

First, the reward model's architecture is usually initialized from the SFT model itself. Same transformer backbone, replace the next-token head with a scalar readout on the last-token hidden state. This is cheap (you reuse all the pretraining), and it turns out to matter a lot. A reward model built on top of a weak base ranks answers worse, which quietly caps the quality of everything downstream.

Second, the reward model scores a full response, not each token. It reads the whole thing, then outputs one number. In PPO we'll assign that terminal reward to the last token of the sequence and treat every token before it as having reward zero. More on that in the PPO section.

Third, the reward model is pretty bad at assigning absolute quality. It's calibrated for ranking, not for scoring. A reward of $+1.3$ doesn't mean anything by itself; only the gap to another response is meaningful. If you find yourself reading an RM output and thinking "oh that's a 1.3, that's pretty good," you're doing it wrong.

PPO for language models

Now we have a reward model. We have an SFT model. We need to use the first to update the second. This is where reinforcement learning shows up, and specifically where we use Proximal Policy Optimization.

Let me try to motivate PPO as narrowly as possible, just enough to show what it's protecting against. The naive move would be: sample a response, score it with the RM, and do a policy-gradient update $\nabla_\theta \log \pi_\theta(y \mid x) \cdot r$. That works in toy environments. For a language model with billions of parameters, it explodes. A single lucky high-reward sample can push the policy into a region where it assigns near-zero probability to coherent English. Once that happens, every subsequent sample is gibberish, the RM score collapses, and training never recovers.

PPO's fix is a trust region. At each update, we compute the probability ratio

between the current policy and the policy we used to sample the data. If the ratio is one, nothing has changed. If it's 1.5, we've made the action 50% more likely. PPO says: don't let the update move that ratio too far from one. Specifically, the clipped surrogate objective is

where $\hat{A}_t$ is the advantage (how much better this action was than the baseline) and $\varepsilon$ is typically $0.2$.

The min-of-two-things is what makes this work. If the advantage is positive (the action was good) and the ratio is already at $1 + \varepsilon$, increasing it further gives you no additional reward, because the clipped term caps the objective. If the advantage is negative (the action was bad) and the ratio has already dropped to $1 - \varepsilon$, pushing it lower gives you no additional penalty. Either way, the gradient through that sample is zero once you leave the trust region.

Watch the sweep. The marker traces $L^{\text{CLIP}}$ as a function of $r$. The dashed line is the unclipped $r \cdot A$, which is what you'd get with vanilla policy gradient. The solid line bends at the clip boundaries and goes flat. That flat region is the "nothing more to gain" zone, and it's the reason PPO stays stable where naive policy gradient would diverge.

Reward, minus a baseline, minus KL

Before the PPO loss can fire, we need $\hat{A}_t$. In a classic actor-critic setup, this would be "reward minus value function." For LLM RLHF, three specific things happen:

1. Reward at the end. $r_\phi(x, y)$ scores the full response and gets credited to the last token. Every other token sees zero reward directly. This sounds wrong, but PPO (via GAE or equivalent) spreads the final-token signal backward through time.
2. Baseline subtraction. We typically subtract the mean reward across the rollout batch (or a learned value function's prediction) so the advantages are mean-zero. This is just variance reduction; the trick you do in any REINFORCE-style setup.
3. KL penalty rolled into the reward. This is the one that matters most for stability. At every token, we subtract $\beta \cdot \log(\pi_\theta(y_t \mid x, y_{\lt t}) / \pi_{\text{SFT}}(y_t \mid x, y_{\lt t}))$ from the per-token reward. If the current policy is wildly more confident than SFT on this token, we pay for it. If it's less confident, we earn back a little. In expectation, this is exactly $\beta \cdot \mathrm{KL}(\pi_\theta \| \pi_{\text{SFT}})$ per sequence.

So the effective per-token reward that PPO actually uses is

where $r^{\text{RM}}_t$ is zero everywhere except the last token.

The KL term is not a regularizer in the "encourage parsimony" sense. It's an anchor. Without it, PPO will cheerfully drive the policy toward whatever collapsed distribution maximizes the RM score, because the RM is an imperfect model and it can be fooled.

Watch what happens when the drift is small: reward goes up, KL is small, net objective rises. At moderate drift, KL has grown but reward is still rising faster. Then the policy starts to concentrate too much mass on one token, and the story inverts. KL explodes while reward barely moves. The net objective (the thing PPO is actually maximizing) falls off a cliff. This is what reward hacking looks like when it's caught by the KL tether. When it's not caught by the KL tether, you get the next viz.

Why RLHF is hard: the reward model is the bottleneck

Most of the engineering problems come down to one fact: the reward model is a lossy approximation to human preference, and PPO will find every imperfection.

If the RM rewards longer responses slightly more than shorter ones at equivalent quality, PPO will produce longer responses. If the RM slightly prefers confident hedging to honest uncertainty, PPO will produce confident hedges. If the RM has a blind spot where obviously-incorrect-but-plausible answers score fine, PPO will exploit that blind spot. These have all shown up in production RLHF runs.

The canonical picture is this: plot the RM score against human-judged quality over the course of training. For a while, they track each other. Then they diverge. The RM score keeps climbing, and the human-judged quality plateaus, then drops.

The usual mitigations are all imperfect: regularize harder with KL (limits drift but also improvement), re-train the RM periodically on fresh policy samples (expensive), use multiple RMs and take a conservative aggregate, or cap training steps with humans watching for drift. The last one is the default in practice.

Anthropic's "Helpful and Harmless" paper spends a lot of pages on this tension, and Casper et al.'s 2023 review catalogs the full taxonomy. None of this makes RLHF useless. The pipeline is less "solve alignment" and more "compress human preferences into a gradient signal, knowing it's going to be imperfect, and stop before the imperfections eat the model."

Worked example

Let me trace one step of the PPO update for a single prompt. Made-up numbers, but the shape matches a real rollout.

Prompt: "Explain gradient descent briefly."

Step 1: rollout. We sample four completions from the current policy $\pi_{\theta_{\text{old}}}$, each sampled with the standard temperature/top-p setup.

Step 2: score. We run each $(x, y_i)$ through the reward model.

Step 3: compute advantage. We subtract the batch mean reward $\bar{r} = 0.4$ as a baseline, and subtract the per-token KL penalty with $\beta = 0.02$ (a common default). In this toy, the KL term is small, and most of the advantage comes from $r_i - \bar{r}$.

Step 4: compute ratios. The update runs for a couple of inner epochs on this same rollout, so by the time we compute gradients, $\pi_\theta$ has drifted from $\pi_{\theta_{\text{old}}}$. We compute the per-token ratio $r_t = \pi_\theta(y_t \mid \cdot) / \pi_{\theta_{\text{old}}}(y_t \mid \cdot)$. The per-sequence product of ratios is what goes into the clipped objective.

Step 5: clip. For any token where $r_t$ has moved outside $[1 - \varepsilon, 1 + \varepsilon]$ in the wrong direction, the gradient is killed (clipped contribution of zero). For tokens inside the trust region, we get the normal $r_t \cdot \hat{A}_t$ gradient.

A couple of things to notice in the table:

• Completion 1 has advantage $+1.85$ (positive). Its per-token ratios hover just above 1 (the policy now mildly prefers it). All ratios stay inside $[0.80, 1.20]$, so the full gradient flows and the policy gets nudged further toward completion 1.
• Completion 4 has advantage $-2.15$ (strongly negative). Its ratios have already dropped to $[0.55, 0.71]$, meaning the policy has already moved substantially away. Every one of those ratios is outside $1 - \varepsilon = 0.80$. PPO clips the gradient contribution to zero. The policy already moved; PPO stops further churn on this particular sample.
• Completion 3 also has negative advantage, but its ratios are only at $[0.79, 0.91]$, mostly inside the trust region. PPO still pushes here, just not hard.

Without the clip, completion 4's gigantic negative advantage would dominate the update and rip the policy apart on any token it touched. With the clip, the policy continues to move, but a single spicy sample cannot destroy everything.

Implementation sketch

The mechanics in twenty lines of PyTorch-ish pseudocode. Not runnable verbatim. You'd need the surrounding RL plumbing (GAE, value function, optimizer schedules, reference-model handling), and in practice you'd use a library like trl or trlX. But this captures the two losses.

import torch
import torch.nn.functional as F
 
# ------------------------------------------------------------------
# 1) Reward-model loss: Bradley–Terry on preference pairs.
#    rm(x, y) returns a scalar. We want rm(x, y_w) > rm(x, y_l).
# ------------------------------------------------------------------
def rm_loss(rm, x, y_w, y_l):
    r_w = rm(x, y_w)              # shape (batch,)
    r_l = rm(x, y_l)              # shape (batch,)
    # -log σ(r_w − r_l) == softplus(-(r_w − r_l))
    return F.softplus(-(r_w - r_l)).mean()
 
# ------------------------------------------------------------------
# 2) One PPO step on a rollout.
#    We have: prompts, sampled responses, per-token log-probs under
#    the *old* policy (logp_old), and the RM reward for each response.
# ------------------------------------------------------------------
def ppo_step(policy, ref_model, prompts, responses,
             logp_old, rm_rewards, beta=0.02, eps=0.2):
    # (a) forward pass — get new log-probs and the reference log-probs.
    logp_new = policy.logprobs(prompts, responses)   # (B, T)
    with torch.no_grad():
        logp_ref = ref_model.logprobs(prompts, responses)  # (B, T)
 
    # (b) per-token KL penalty — this is the anchor.
    kl_per_token = logp_new - logp_ref               # (B, T)
 
    # (c) effective reward per token: RM reward lives on the LAST token,
    #     minus beta * KL at every token.
    token_rewards = -beta * kl_per_token
    token_rewards[:, -1] += rm_rewards               # credit terminal reward
 
    # (d) advantage: here, a cheap batch-mean baseline. In practice you'd
    #     compute returns and use a learned value function (GAE).
    advantages = token_rewards - token_rewards.mean()
 
    # (e) clipped surrogate objective.
    ratio = torch.exp(logp_new - logp_old)           # (B, T)
    unclipped = ratio * advantages
    clipped = torch.clamp(ratio, 1 - eps, 1 + eps) * advantages
    loss = -torch.min(unclipped, clipped).mean()     # minus because we maximize
 
    return loss

A few comments on what's happening here.

The reward-model loss is one line of actual work. softplus(-(r_w − r_l)) is the numerically-stable form of $-\log \sigma(r_w - r_l)$. No magic.

The PPO step has five parts. Part (c) is the one I find confusing every time I reread it: the RM reward is a single scalar per sequence, but the loss is summed over tokens, so we fold the RM reward onto the terminal token and let the advantage propagate backward through time. Part (d) is a toy baseline; real systems use a learned value head. Part (e) is the clipped objective, exactly as written in Schulman's PPO paper.

The with torch.no_grad() on the reference model matters. ref_model is frozen $\pi_{\text{SFT}}$. We need its log-probs, but we absolutely do not want to update it. The KL anchor only makes sense if the anchor stays still.

Misconceptions

"The reward model learns what humans want." The reward model learns what humans prefer between pairs of samples from a specific distribution. That is not the same as "what humans want." If the RM only ever saw completions from a 13B SFT model in English on helpful-assistant prompts, it has no idea how to score completions from a 70B model doing creative writing in Spanish. Reward models generalize poorly outside the distribution they were trained on, and this is the root cause of most of RLHF's stability problems.

"PPO is the RL part." PPO is a stability trick layered on top of the actual RL part. The RL part (sample trajectories, get reward, compute policy gradient) would work with vanilla REINFORCE in principle. In practice, vanilla REINFORCE on a billion-parameter language model diverges within a handful of updates. PPO is the specific machinery (clipped ratio, multiple epochs on the same rollout) that keeps the updates inside a trust region so the policy does not collapse. There's a real sense in which PPO is less about RL and more about "how to do policy-gradient updates without catastrophically damaging a large pretrained network."

"The KL penalty is optional regularization." Skipping the KL penalty does not give you slightly-less-regularized RLHF. It gives you a policy that converges to a degenerate distribution maximizing the RM's flaws. This is the first thing that happens, not some subtle long-tail failure. The KL penalty is load-bearing, and the choice of $\beta$ is one of the two or three most important hyperparameters in the entire pipeline. A common framing is "steering between reward and KL," and that framing is closer to what's going on than calling KL a regularizer.

"RLHF makes the model honest / aligned / safe." RLHF makes the model's output distribution look more like the distribution humans prefer when ranking pairs. That is a narrower claim than "aligned." If the preference data systematically rewards confident-sounding answers over uncertain-but-correct ones, RLHF will make the model confidently wrong. The technique is agnostic about what humans actually value; it only reflects what labelers chose in the specific moment they chose it. Alignment properties come from the data curation, not from the optimizer.

What's next

The next post covers DPO and its variants, the supervised shortcut that replaces the entire RM-plus-PPO pipeline with a single closed-form loss on preference pairs, and why it has taken over most of the open-source post-training stack.

Additional reading (and watching)

• Ouyang, L., et al. (2022). Training language models to follow instructions with human feedback. arXiv:2203.02155 (InstructGPT). The paper that made the modern RLHF pipeline legible — SFT model, preference data, reward model, PPO with KL.

• Bradley, R. A., & Terry, M. E. (1952). Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons. Biometrika, 39(3/4), 324-345. The original formulation of the Bradley–Terry preference model, which the RM loss is a direct application of.

• Schulman, J., et al. (2017). Proximal Policy Optimization Algorithms. arXiv:1707.06347. The clipped-surrogate objective itself. Dense, worth reading once for the clip equation and the experiments showing it outperforms TRPO and vanilla PG.

• Ziegler, D. M., et al. (2019). Fine-Tuning Language Models from Human Preferences. arXiv:1909.08593. The earlier OpenAI paper that laid out the per-token KL penalty against the SFT reference. Most of the operational details of modern RLHF come from here.

• Christiano, P., et al. (2017). Deep Reinforcement Learning from Human Preferences. arXiv:1706.03741. The foundational paper showing that a reward model trained from pairwise preferences can steer a policy. Predates RLHF-for-LLMs but the template is identical.

• Bai, Y., et al. (2022). Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback. arXiv:2204.05862. Anthropic's deep dive on what actually shows up when you run RLHF at production scale, including reward hacking and the helpfulness-harmlessness tradeoff.

• Casper, S., et al. (2023). Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback. arXiv:2307.15217. A thorough taxonomy of what's hard about RLHF. Essential reading for the known failure modes of an RLHF pipeline at production scale.

• Rafailov, R., et al. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. NeurIPS 2023. The paper that collapsed the RLHF pipeline into a single supervised loss. Enormous practical impact on open-model post-training; covered in the next post.

• Bai, Y., et al. (2022). Constitutional AI: Harmlessness from AI Feedback. arXiv:2212.08073. The RLAIF (RL from AI Feedback) approach that replaces human preference labelers with a second LM guided by written principles.

• Lightman, H., et al. (2023). Let's Verify Step by Step. arXiv:2305.20050. The process-reward-model paper — score every step of a reasoning chain rather than just the final answer. Influential for the 2024-2026 shift toward verifier-based RL for reasoning tasks.