Instruction Tuning and RLHF

Before 2022, GPT-style models were impressive at predicting text — but they weren't actually helpful. Ask one to write a poem and it might write a poem, or it might continue writing more instructions on how to write poems. Instruction tuning changed this. RLHF made it reliable. This module explains both.

By the end of this module, you will be able to explain the three stages of RLHF, describe what a reward model does, and understand when to use simple instruction tuning versus the full RLHF pipeline.

Stage 1: Supervised Fine-Tuning on Instructions (SFT)

The first step is standard instruction fine-tuning. You take a pre-trained model and fine-tune it on a dataset of (instruction, ideal response) pairs. This teaches the model to answer questions and follow directions rather than just continue text.

The SFT step on its own produces a genuinely useful model. Wei et al. (arXiv:2109.01652) showed that fine-tuning on a diverse set of instructions makes models better at following new, unseen instructions — a property called zero-shot generalization. This is what the FLAN series of models demonstrated.

For many practical applications, SFT alone is enough. You need RLHF when you need the model to learn preferences that are hard to specify with examples: "be helpful but not harmful," "be honest but not blunt," "be concise but complete."

Stage 2: Training a Reward Model

The reward model is a separate neural network trained to predict human preference. You collect data by having human raters compare two model responses to the same prompt and choose which one they prefer. The reward model learns to assign a high score to responses humans prefer and a low score to responses they don't.

Stage 3: Reinforcement Learning from Human Feedback (PPO)

The third stage uses the reward model to fine-tune the LLM via reinforcement learning. The algorithm most commonly used is called PPO (Proximal Policy Optimization). The loop works like this:

1. Generate a response from the current LLM
2. Score the response using the reward model
3. Update the LLM's weights to make higher-scoring responses more likely
4. Repeat thousands of times

There's a crucial safeguard: a KL divergence penalty prevents the LLM from drifting too far from its SFT starting point. Without this, the model can "game" the reward model by generating responses that get high scores but are incoherent or unhelpful to actual humans — a phenomenon called reward hacking.

DPO: The Simpler Alternative to RLHF

Training a separate reward model and running PPO is complex and unstable. Direct Preference Optimization (DPO, Rafailov et al., arXiv:2305.18290) achieves similar results by reformulating the RLHF objective as a classification problem on preference pairs. The model is trained directly to prefer the chosen response over the rejected one, without ever training a separate reward model.

DPO is faster, more stable, and often matches or exceeds RLHF quality on many tasks. For most teams starting with alignment, DPO is the practical starting point.

from trl import DPOTrainer, DPOConfig
from transformers import AutoModelForCausalLM, AutoTokenizer
from datasets import Dataset

# DPO is simpler than full RLHF — no separate reward model needed
# Dataset format: each row has prompt, chosen response, rejected response
preference_data = Dataset.from_dict({
    "prompt": [
        "Explain why the sky is blue.",
        "How do I reset my API key?"
    ],
    "chosen": [
        "Light from the sun contains all colors. Shorter blue wavelengths scatter more when they hit air molecules, so we see blue.",
        "Go to Settings > API Keys > Revoke, then click Generate New Key. Copy it immediately — you can't view it again."
    ],
    "rejected": [
        "The sky is blue due to Rayleigh scattering of solar electromagnetic radiation by atmospheric gaseous molecules.",
        "You can reset your API key in the settings section of the application interface."
    ]
})

model = AutoModelForCausalLM.from_pretrained("your-sft-model")
tokenizer = AutoTokenizer.from_pretrained("your-sft-model")
ref_model = AutoModelForCausalLM.from_pretrained("your-sft-model")  # frozen reference

dpo_config = DPOConfig(
    beta=0.1,           # KL penalty — higher = stays closer to reference model
    learning_rate=5e-5,
    num_train_epochs=1,
    per_device_train_batch_size=4,
    output_dir="./dpo-output"
)

trainer = DPOTrainer(
    model=model,
    ref_model=ref_model,
    args=dpo_config,
    train_dataset=preference_data,
    tokenizer=tokenizer,
)
trainer.train()

When Do You Need the Full RLHF Pipeline?

For most teams: you don't. DPO achieves competitive results at far lower complexity. Use the full PPO-based RLHF pipeline only when:

• You need online learning — the model generates responses and gets scored in real time as the training loop runs
• The preference data is too sparse for DPO's classification approach
• You're building a foundation model and need to scale the alignment process across billions of parameters

For a custom enterprise model serving a specific domain, SFT followed by DPO is the standard modern approach. It requires fewer compute resources and is easier to debug.