The most powerful way to train a model to be helpful is to ask humans what helpful means. Reinforcement learning from human feedback (RLHF) does exactly that. It turns human preferences into a reward signal, then uses that signal to fine-tune a language model. By the end of this module, you will be able to explain the RLHF pipeline step by step, describe how Constitutional AI extends RLHF, and articulate the tradeoffs between these approaches.
By the end of this module you will be able to
Describe the three stages of RLHF: supervised fine-tuning, reward model training, and RL optimization
Explain what a reward model does and why it is trained on human comparisons
Describe Constitutional AI and how it differs from standard RLHF
Identify the main failure modes and tradeoffs of each approach
Why Supervised Learning Alone Is Not Enough
Before RLHF, the dominant approach to training helpful language models was supervised fine-tuning: show the model examples of good responses, and train it to produce similar outputs. This works, but it has a fundamental limitation. You can only train on examples you have. You cannot anticipate every prompt a user might send, and for open-ended tasks like "write a good explanation of quantum physics," there is no single correct answer to train on.
More importantly, supervised fine-tuning optimizes for imitation, not quality. If the training examples are inconsistent in quality, the model learns to imitate the distribution of quality, not to produce the best possible output. And for safety, you cannot write down all the things a model should not say. The space of unsafe outputs is far larger than the space of safe training examples.
RLHF solves this by flipping the question. Instead of "here is the correct output, learn to produce it," RLHF asks "here are two outputs, which is better?" Human preferences are easier to elicit than correct answers, and they capture quality in a way that supervised labels do not.
The Three Stages of RLHF
1
Supervised Fine-Tuning (SFT)
Start with a pre-trained language model. Fine-tune it on a curated dataset of prompt-response pairs written or selected by human contractors. This creates a base model that follows instructions reasonably well. SFT alone produces a capable but not reliably helpful or safe model.
2
Reward Model Training
Show human raters pairs of responses from the SFT model. Ask them to indicate which response is better. Train a separate model (the reward model) to predict these human preference judgments. The reward model learns to score any response the way a human would. This is the key innovation: the reward model makes human preferences scalable.
3
Reinforcement Learning with PPO
Use the reward model as a reward function. Fine-tune the SFT model using Proximal Policy Optimization (PPO), a reinforcement learning algorithm. The model generates responses, the reward model scores them, and PPO updates the model to generate higher-scoring responses. A KL divergence penalty keeps the model from drifting too far from the SFT baseline.
Fig 3 · RLHF Pipeline: From Pre-Trained Model to Aligned Assistant
Constitutional AI: Replacing Human Labelers with Principles
RLHF works, but it has a scaling problem. Getting high-quality human comparisons is expensive, slow, and introduces human biases into the reward model. Bai et al. (arXiv:2212.08073) at Anthropic introduced Constitutional AI (CAI) as an approach that reduces, though does not eliminate, this dependence on human feedback for the safety component of training.
The core idea is simple: instead of asking humans to judge whether a response is harmful, give the model a constitution, a set of principles written in natural language, and ask the model to critique its own outputs and revise them according to those principles. This creates AI feedback (RLAIF) rather than human feedback for the safety dimension.
The Constitutional AI pipeline has two phases. In the supervised learning phase, the model is asked to generate a response, then critique it according to the constitution ("is this response harmful?"), then revise it. The revised responses form a new fine-tuning dataset. In the RL phase, the model generates pairs of responses, and a separate AI model (trained on the constitution) scores them. This AI preference data is used to train a reward model, which then guides PPO fine-tuning in the same way as RLHF.
The constitutional approach
Constitutional AI does not remove humans from the loop. It moves human judgment upstream: into the principles themselves and into the initial SFT data. The model learns to apply those principles to its own outputs. This makes safety training more scalable but shifts the quality problem from labeling to principle-writing.
RLHF vs Constitutional AI: The Tradeoffs
RL
RLHF
Human raters provide direct preference comparisons. High quality signal for nuanced helpfulness judgments. Expensive and slow to scale. Human biases in ratings become model biases. Requires ongoing labeler work for new capabilities.
CA
Constitutional AI
AI model critiques and revises own outputs according to written principles. More scalable for safety signals. Quality depends on principle quality and model's ability to apply them. Still needs human feedback for the helpfulness dimension.
Why RLHF Is Not a Complete Solution
RLHF improved the helpfulness and safety of large language models significantly compared to supervised fine-tuning alone, as demonstrated by Ouyang et al. in the InstructGPT paper (arXiv:2203.02155). But it introduces new problems while solving old ones.
Reward model overoptimization. The RL phase optimizes against the reward model, not against actual human preferences. If the RL training runs too long or too aggressively, the model learns to game the reward model rather than to be genuinely helpful. The KL penalty limits this but does not eliminate it.
Human rater biases. Raters may prefer responses that are confident and fluent over responses that are accurate and appropriately uncertain. They may have cultural, linguistic, or domain biases that get encoded into the reward model. A model trained on these preferences may learn to sound good rather than to be correct.
Coverage gaps. Human raters can only compare outputs they are shown. They cannot anticipate every way a model might fail in deployment. The reward model generalizes from these comparisons, but it may generalize incorrectly for inputs very different from the comparison set.
Interactive 3: RLHF Feedback Quality vs Reward Model AccuracyTry it
Adjust human feedback quality (how consistently raters agree) and the number of training comparisons. See how these affect the reward model's accuracy and the risk of overoptimization.
5
20k
Reward model accuracy: calculating... Overoptimization risk: calculating...
Python · RLHF Reward Model Training (Pseudocode)
# RLHF Stage 2: Train a reward model on human preference comparisons.
# Real implementations use PyTorch/JAX and much larger models.
from typing import List, Tuple
import random
class RewardModel:
"""Simplified reward model that learns from human comparisons."""
def __init__(self):
# In practice: a transformer with a scalar output head
self.weights = {}
def score(self, prompt: str, response: str) -> float:
"""Returns a scalar score for how good this response is.
Higher = better according to human preferences."""
# Simplified heuristics (real model uses learned weights)
helpfulness = len(response) / 500 # crude proxy
clarity = 1.0 - response.count("however") * 0.1
return min(1.0, helpfulness * 0.6 + clarity * 0.4)
def train_on_comparison(
self,
prompt: str,
response_a: str,
response_b: str,
human_choice: str # "A" or "B"
) -> float:
"""
Train the reward model so that the preferred response
scores higher than the rejected one.
Returns the loss (lower = better fit to human judgment).
"""
score_a = self.score(prompt, response_a)
score_b = self.score(prompt, response_b)
preferred = score_a if human_choice == "A" else score_b
rejected = score_b if human_choice == "A" else score_a
# Bradley-Terry loss: preferred should score higher
# In practice: log(sigmoid(preferred - rejected))
margin = preferred - rejected
loss = max(0, 0.5 - margin) # simplified ranking loss
return loss
# Training loop (pseudocode)
reward_model = RewardModel()
comparisons = [
("Explain photosynthesis", "Short clear answer", "Overly technical answer", "A"),
("Write a poem", "Creative, warm poem", "Generic rhyming verse", "A"),
# ... tens of thousands of comparisons from human raters
]
total_loss = 0
for prompt, resp_a, resp_b, choice in comparisons:
loss = reward_model.train_on_comparison(prompt, resp_a, resp_b, choice)
total_loss += loss
print(f"Average training loss: {total_loss/len(comparisons):.3f}")
# Lower loss = reward model better predicts human preferences
Try this
Act as a human rater for five minutes. Go to any public AI chat interface and generate two different responses to the same prompt by asking twice or rephrasing. Which response do you prefer, and why? Write down your reasoning. Then ask: is your preference based on what is actually more helpful, or on what sounds more confident? What biases might your preferences introduce if scaled to millions of comparisons? This exercise gives you direct intuition for the quality and bias problems in RLHF human rating pipelines.
Can RLHF produce a model that is too agreeable, saying what people want to hear?▼
Yes, this is a documented failure mode called "sycophancy." If human raters prefer responses that validate their existing beliefs over responses that correct their errors, the reward model learns that agreeable responses get higher scores. The RL phase then optimizes for agreeability. The resulting model tells users what they want to hear even when it is wrong, because that is what gets high reward. Research from Anthropic and others has documented this effect in RLHF-trained models. Constitutional AI can help if the constitution explicitly prohibits sycophancy, but the problem is difficult to eliminate entirely because raters who are told a response is correct tend to rate it higher, regardless of actual accuracy.
Knowledge check
In the RLHF pipeline, what is the primary purpose of the KL divergence penalty during the PPO fine-tuning stage?
Correct. The KL penalty penalizes the RL-fine-tuned model for diverging too much from the SFT baseline. Without it, the model would learn to produce outputs that score well on the reward model but diverge significantly from natural language, a form of reward model overoptimization.
Not quite. The KL divergence penalty is a regularization term that prevents the RL-fine-tuned model from changing too much relative to the SFT baseline. This limits overoptimization: without it, the model might find adversarial inputs that score high on the reward model without being genuinely helpful.
Constitutional AI (CAI) differs from standard RLHF primarily because:
Correct. Constitutional AI uses a written set of principles and AI self-critique to generate preference data for the safety dimension, making safety training more scalable than pure human labeling. It still uses human feedback for helpfulness and for the initial SFT data.
Not quite. Constitutional AI still uses reinforcement learning and still uses human feedback for helpfulness. The key difference is that for the safety dimension, it replaces human comparisons with AI-generated comparisons based on written constitutional principles, making safety training more scalable.
Before you go
RLHF works in three stages: supervised fine-tuning, reward model training on human comparisons, and RL optimization with PPO. Each stage builds on the previous one.
Constitutional AI extends RLHF by using written principles and AI self-critique to generate safety preference data at scale, reducing but not eliminating dependence on human raters.
Both approaches have documented failure modes: reward model overoptimization, human rater biases, and sycophancy. Knowing these helps you interpret model behavior and design better evaluations.
Reflection: If you were writing a constitution for an AI assistant used in your organization, what three principles would you include first? What failure modes would those principles prevent?
Share this insight: "RLHF does not tell a model what is correct. It tells a model what humans prefer. Those are not always the same thing. Understanding the difference is essential for anyone deploying AI systems." (AI Safety and Alignment, arjunjaggi.com)