A pretrained base model is like a very well-read person who has never been taught to answer questions, follow instructions, or be polite. It will complete your prompt, not respond to it. Getting from capable but unruly base model to a helpful, harmless assistant requires several additional training stages: supervised fine-tuning, reinforcement learning from human feedback, and increasingly, direct preference optimization. This module explains exactly what each stage does and why it changes the model's behavior.
By the end of this module you will be able to
Explain the difference between a pretrained base model and an instruction-following model
Describe the three stages of the RLHF pipeline from Ouyang et al. (arXiv:2203.02155)
Explain what the reward model does and how it is trained from human preference data
Describe DPO (Rafailov et al. arXiv:2305.18290) and why it eliminates the need for a reward model
Explain the alignment tax: what models gain and lose from RLHF and fine-tuning
The Gap Between Base Model and Assistant
After pretraining, a model is good at one thing: completing text in the style of its training data. Show it "The capital of France is" and it says "Paris." Show it "Write a poem about autumn" and it might continue with another poem prompt rather than writing a poem. It has learned to predict text, not to follow instructions.
The InstructGPT paper (Ouyang et al., arXiv:2203.02155) named this gap clearly: the model's training objective, next-token prediction over internet text, is different from what users actually want, which is helpful, honest, harmless responses to their specific requests. Bridging this gap requires a new training approach that incorporates human judgment about what good responses look like.
Stage 1: Supervised Fine-Tuning (SFT)
The first stage is supervised fine-tuning. Human contractors write ideal responses to a set of diverse prompts. This creates a dataset of (prompt, high-quality-response) pairs. The base model is then fine-tuned on this dataset using standard next-token prediction, but now the targets are ideal human responses rather than random internet text.
SFT teaches the model the format of helpful responses: answer the question, follow the instruction, be concise and clear. After SFT, the model is much more likely to respond as a helpful assistant would. But SFT has a limitation: it is expensive to produce enough high-quality labeled data to cover the full range of possible user requests. And human contractors writing ideal responses may not perfectly capture what users actually prefer when they see model outputs.
Stage 2: Reward Model Training
The second stage addresses SFT's limitation by training a separate reward model. For a given prompt, the base model or SFT model is used to generate multiple candidate responses. Human raters compare these responses and rank them from best to worst. This preference data trains the reward model: a model that takes a (prompt, response) pair as input and outputs a scalar reward score predicting how much a human would prefer that response.
Preference data is much cheaper to collect than demonstrations. Judging which of two responses is better is faster and more consistent than writing an ideal response from scratch. The reward model learns to approximate human preference across a much larger variety of prompts and response types than could be covered by SFT demonstrations alone.
Stage 3: Reinforcement Learning with PPO
With a reward model in hand, the third stage uses reinforcement learning to update the SFT model. The SFT model (now called the policy) generates responses to prompts. The reward model scores each response. The policy is updated to produce responses that earn higher reward model scores, using the Proximal Policy Optimization (PPO) algorithm.
PPO optimizes the policy while constraining how far it can move from the SFT starting point in each update. This constraint, measured by KL divergence from the reference SFT model, prevents the policy from finding degenerate strategies that maximize the reward model score without producing genuinely good responses. Without this constraint, the model quickly learns to produce responses that fool the reward model without being actually helpful, a form of reward hacking.
Fig 1 · The Three-Stage RLHF Pipeline (Ouyang et al. arXiv:2203.02155)
Direct Preference Optimization (DPO): Eliminating the Reward Model
RLHF with PPO works but is complex: it requires maintaining four models simultaneously (the policy, the reward model, a reference policy, and a value function for PPO). Rafailov et al. "Direct Preference Optimization: Your Language Model is Secretly a Reward Model" (arXiv:2305.18290) showed that this complexity is unnecessary.
DPO derives a closed-form solution that directly optimizes the policy on preference data without a separate reward model or RL. Given pairs of (chosen, rejected) responses to the same prompt, DPO increases the log-probability of the chosen response while decreasing the log-probability of the rejected response, with implicit regularization toward the reference policy built into the loss function.
Where y_w is the chosen (winning) response, y_l is the rejected (losing) response, pi is the policy being trained, pi_ref is the reference SFT model, and beta controls the regularization strength. This loss increases the margin between chosen and rejected responses while not moving too far from the reference model.
DPO is substantially simpler than PPO-based RLHF. It trains stably on preference data, does not require a separately trained reward model, and produces alignment quality comparable to RLHF in practice. Many recent aligned models use DPO or DPO variants rather than PPO.
The Alignment Tax: What Fine-Tuning Changes
Alignment training changes model behavior in ways beyond making it follow instructions. Researchers have documented what is sometimes called the alignment tax: the tradeoff between helpfulness and the model's raw capability or willingness to engage with certain topics.
RLHF-trained models become more helpful, more consistent, and safer, but they can also become more verbose (human raters tend to prefer longer responses, so reward models learn to score length positively even when brevity would be better), more cautious (overly broad refusals are a common failure mode), and sometimes worse at certain technical benchmarks where the pretraining base model outperformed.
Wei et al. (arXiv:2307.15043) documented how competing objectives and mismatched generalization create alignment failures. The model's RLHF training tries to make it helpful and safe simultaneously, and these objectives can conflict. A model trained to be maximally helpful might assist with harmful requests; a model trained to be maximally safe might refuse legitimate ones.
Key insight
Fine-tuning and RLHF do not add new knowledge to the model. They change the model's behavior in generating from the knowledge it already has. A fine-tuned model that refuses a harmful request is not doing so because it lacks the knowledge to fulfill it; it is doing so because the training process shaped its output distribution toward refusal for that class of request.
SFT
Supervised Fine-Tuning
Train on (prompt, ideal-response) pairs. Teaches the model the format of helpful responses but limited by how much human demonstration data you can collect.
RM
Reward Model
A model trained on human preference rankings. Scores any (prompt, response) pair. More scalable than demonstrations: ranking is faster than writing. Used in RLHF PPO.
DP
DPO
Direct Preference Optimization. Eliminates the reward model and PPO. Directly optimizes the policy on preference data via a closed-form loss. Simpler, stable, competitive with RLHF.
Interactive 4: Alignment Behavior ShifterTry it
Adjust the SFT training amount and the RLHF reward beta to see how each stage shifts the distribution of model outputs between pretraining distribution, helpfulness, and safety behaviors. This is a directional illustration.
50
0.3
Helpfulness: -- Safety: -- Raw capability retention: --
Python · DPO Loss (Rafailov et al. arXiv:2305.18290)
import torch
import torch.nn.functional as F
def dpo_loss(
pi_logps_chosen, # log probs of chosen response tokens under policy
pi_logps_rejected, # log probs of rejected response tokens under policy
ref_logps_chosen, # log probs of chosen response tokens under reference model
ref_logps_rejected,# log probs of rejected response tokens under reference model
beta=0.1 # KL regularization strength
):
"""
Compute the DPO loss for a batch of preference pairs.
Increases probability of chosen responses, decreases rejected,
while staying close to the reference model (controlled by beta).
"""
# Log-ratio for chosen: how much the policy prefers this vs reference
pi_logratios_chosen = pi_logps_chosen - ref_logps_chosen
# Log-ratio for rejected: how much the policy prefers this vs reference
pi_logratios_rejected = pi_logps_rejected - ref_logps_rejected
# DPO reward: beta * (log ratio for chosen - log ratio for rejected)
# Positive means policy shifted more toward chosen than rejected
logits = beta * (pi_logratios_chosen - pi_logratios_rejected)
# Loss: sigmoid cross-entropy encourages logits to be positive
losses = -F.logsigmoid(logits)
return losses.mean()
# Example batch of 4 preference pairs
# Positive log-ratio = policy moved toward this response vs reference
pi_chosen = torch.tensor([-2.1, -1.8, -2.5, -1.9])
pi_rejected = torch.tensor([-3.0, -3.2, -2.8, -3.1])
ref_chosen = torch.tensor([-2.5, -2.2, -2.8, -2.3])
ref_rejected= torch.tensor([-2.6, -2.4, -2.5, -2.5])
loss = dpo_loss(pi_chosen, pi_rejected, ref_chosen, ref_rejected, beta=0.1)
print(f"DPO loss: {loss.item():.4f}")
# Lower loss means the policy is correctly preferring chosen over rejected
Try this
Access any instruction-tuned model via API and try this test: give the same ambiguous prompt to a base model completion endpoint (if available) and an instruction-tuned endpoint. For example: "Tell me how to pick a lock." The base model will likely continue describing lock picking as a narrative. The instruction-tuned model will likely add safety caveats or ask for context. Neither response is "more accurate" as raw prediction; the difference reflects how RLHF reshaped the output distribution for sensitive prompts.
Why does RLHF sometimes make models more verbose even when shorter answers are better?▼
Human preference raters tend to rate longer, more detailed responses more favorably, even when the question called for a brief answer. The reward model learns this pattern from the rater data and scores longer responses higher. The RLHF policy then learns to generate longer responses to maximize reward model score. This is a classic case of reward hacking: the model is optimizing for what the reward model measures (which correlates with human preference but is not identical to it). This specific failure mode is well-documented and has driven research into better reward model training, calibration, and multi-dimensional reward modeling that separates helpfulness, quality, and length.
Knowledge check
What problem does DPO (Rafailov et al. arXiv:2305.18290) solve compared to RLHF with PPO?
Correct. DPO derives a closed-form loss that directly optimizes the policy on preference data. It eliminates the need to train a separate reward model and avoids the complexity and instability of PPO reinforcement learning, while achieving comparable alignment quality.
Not quite. DPO still requires human preference data (chosen vs rejected response pairs). Its advantage is that it eliminates the reward model training step and the PPO RL training loop, making alignment training significantly simpler.
What does the KL divergence constraint in PPO-RLHF prevent?
Correct. Without the KL constraint, the policy quickly learns degenerate strategies that score high on the reward model without being actually helpful. The KL constraint limits how far the policy can stray from the SFT reference point in each update, keeping exploration conservative.
Not quite. The KL divergence constraint measures how far the current policy has moved from the reference SFT model. It limits the policy from exploiting the reward model (finding responses that score high but are not actually good), which is the primary failure mode of unconstrained RL on a learned reward signal.
Before you go
RLHF has three stages: supervised fine-tuning on human demonstrations, reward model training on human preferences, and policy optimization with PPO constrained by KL divergence from the reference model.
DPO achieves comparable alignment without a separate reward model or RL loop, using a closed-form loss on preference pairs with implicit regularization toward the reference policy.
Alignment training does not add knowledge; it reshapes the output distribution. Models become more helpful and safe but may exhibit verbosity bias and occasional over-refusal as byproducts.
Reflection: A DPO-aligned model is trained by telling it which of two responses is preferred. What happens when the "preferred" response is actually wrong, but more confident-sounding?
Share this insight: "RLHF does not teach an AI to be helpful. It teaches it to produce outputs that humans rate as helpful. Those are the same thing most of the time. When they diverge, you get reward hacking." (LLMs from Scratch, arjunjaggi.com)