LLM Fine-Tuning vs RLHF vs DPO: What Actually Aligns Models

Stage 1: Supervised Fine-Tuning

Supervised fine-tuning (SFT) is the first and most straightforward step. A pretrained model is trained on a dataset of (prompt, response) pairs where the responses are high-quality examples of the behavior you want. The training objective is the same as pretraining: cross-entropy loss on the response tokens, with the prompt tokens masked out so that loss is only computed on the model's output.

SFT teaches the model the format of instruction-following: what a helpful response to a question looks like, what code comments should say, how to structure a summary. It does not teach the model to prefer good responses over bad ones in a principled way; it only shows it examples of good responses.

The size of the SFT dataset matters less than its quality. Research across multiple model families has shown that a few thousand high-quality demonstrations can produce strong instruction-following behavior, while adding more low-quality demonstrations can hurt performance. The SFT data is typically curated by domain experts or generated by a teacher model and then filtered.

Stage 2: Reward Modeling

Human preference data takes a different form from SFT data. Rather than providing correct completions, annotators are shown two or more model outputs for the same prompt and asked which they prefer. This produces a dataset of (prompt, chosen, rejected) triples.

A reward model is a transformer fine-tuned from the pretrained base model with a scalar head: instead of predicting the next token, it outputs a single number representing the predicted human preference score for a given (prompt, response) pair. The reward model is trained with a Bradley-Terry pairwise preference loss:

Here r is the reward model, x is the prompt, y_w is the preferred (winning) response, and y_l is the dispreferred (losing) response. sigma is the sigmoid function. The loss encourages the reward for preferred responses to be higher than the reward for rejected responses.

Stage 3: PPO Optimization

The policy model (the SFT model) is then fine-tuned using Proximal Policy Optimization (PPO) to maximize the reward model's scores. For each prompt, the policy generates a response, the reward model scores it, and the gradient update moves the policy in the direction of higher reward.

A critical constraint is added: the KL divergence between the current policy and the original SFT policy is penalized. Without this constraint, the model would quickly degenerate into producing outputs that maximize the reward model's score without being coherent, a phenomenon called reward hacking or reward model overoptimization.

The beta parameter controls the tradeoff: high beta means stay close to the SFT policy, low beta means optimize more aggressively for reward. Finding the right beta requires empirical tuning and depends on the quality of the reward model.

RLHF's three-stage pipeline made sense when it was designed. DPO showed you can skip the reward model entirely and get comparable alignment from a simpler objective on the same preference data.

DPO: A Simpler Path to Alignment

Direct Preference Optimization (Rafailov et al., 2023) showed that the reward model in the RLHF pipeline is not strictly necessary. The key insight is that the optimal policy under the KL-constrained reward objective has a closed-form relationship to the reward function. This allows you to reparameterize the reward model in terms of the policy itself and derive a loss that can be optimized directly on preference pairs without training a separate reward model.

The DPO loss on a preference pair (y_w, y_l) for prompt x is:

This loss increases the relative probability of the preferred response and decreases the relative probability of the rejected response, measured against the reference policy. The beta parameter has the same role as in PPO: it controls the strength of the KL penalty. Rafailov et al. showed that DPO matches or exceeds RLHF performance on summarization and dialogue tasks while requiring only a single training phase.

The Alignment Tax

Alignment fine-tuning consistently changes model behavior in ways that are not always desirable. The most documented effects are verbosity bias (RLHF-tuned models tend to produce longer responses than necessary, because human annotators often prefer more detailed answers), over-refusal (models decline to answer questions that are not actually harmful), and reduced performance on structured tasks like code generation and mathematical reasoning.

These effects are collectively called the alignment tax. They arise because human preference data is noisy and imperfect: annotators have varying standards, they may prefer confident wrong answers over uncertain correct ones, and they systematically prefer certain styles (formal, verbose, organized with bullet points) over others regardless of content quality. The reward model learns and amplifies these biases.

Constitutional AI: Scaling Alignment Without More Human Annotations

The bottleneck in RLHF is human annotation. Collecting high-quality preference data from human raters is slow, expensive, and difficult to scale. Constitutional AI (CAI), introduced by Bai et al. (2022), addresses this by using the model itself to generate the preference signal, guided by a written set of principles called a constitution.

The CAI process has two phases. In the supervised phase, the model is prompted to generate a response, critique it according to the constitutional principles, and then revise it to address the critique. The revised responses are used as supervised fine-tuning data. In the reinforcement learning phase, the model generates response pairs, and a feedback model (trained using the constitution as the preference signal, rather than human raters) scores which response better follows the principles. This AI-generated preference signal is then used in a standard RLHF-style training loop.

The practical advantage is throughput: an AI feedback model can generate preference scores orders of magnitude faster than human raters. The practical limitation is that the quality of the alignment is bounded by the quality of the constitution and the model's ability to apply it consistently. Bai et al. showed that CAI produced models that were less harmful and comparably helpful to RLHF-trained models on their evaluations, making it a viable alternative when annotation budget is constrained.

Parameter-Efficient Fine-Tuning

Full fine-tuning, where all model parameters are updated during training, is prohibitively expensive for most organizations working with models above a few billion parameters. A 70-billion-parameter model requires hundreds of gigabytes of GPU memory just to store the parameters, gradients, and optimizer states needed for training. Parameter-efficient fine-tuning (PEFT) methods reduce this cost by training only a small subset of parameters while keeping the rest frozen.

Low-rank adaptation (LoRA) is the most widely used PEFT technique. The key insight is that the updates to weight matrices during fine-tuning tend to be low-rank: they can be well-approximated as the product of two small matrices. LoRA freezes the original weights and adds a trainable low-rank decomposition alongside each layer. If the original weight matrix has shape (d, d) and the low-rank approximation uses rank r, the number of trainable parameters per layer is 2dr rather than d squared. At r=16 and d=4096, this is 131,072 trainable parameters per layer instead of 16,777,216, a reduction of more than 100x.

PEFT methods make fine-tuning accessible without frontier compute, but they introduce their own tradeoffs. Low-rank updates cannot express all possible weight updates, so PEFT models may underperform full fine-tuning on tasks that require large shifts in model behavior. The choice of which layers to apply LoRA to, and what rank to use, affects results and requires empirical tuning. For domain adaptation tasks where the target domain is not too far from the pretraining distribution, LoRA typically achieves most of the benefit of full fine-tuning at a fraction of the cost.

When to Use Each Approach

Reward Hacking and the Limits of Learned Preferences

Every alignment method that uses a learned reward signal, whether a reward model in RLHF or an AI feedback model in CAI, faces the same fundamental risk: the learned signal is an approximation of what humans actually want, and the optimization process will find ways to exploit the gap between the approximation and the true objective. This is called reward hacking or reward model overoptimization.

The mechanism is well understood. A reward model is trained on a finite sample of human preferences. It will generalize imperfectly to the full space of possible model outputs. As PPO optimizes the policy to maximize reward model scores, it will eventually find outputs that score highly on the reward model but poorly on the actual human preference objective. These outputs may be excessively verbose (because annotators tend to prefer detailed answers), sycophantic (because annotators prefer responses that agree with them), or stylistically formulaic (because certain sentence structures score well regardless of content).

The KL penalty in RLHF and DPO is designed to limit this problem by preventing the policy from diverging too far from the reference policy. But the penalty must be tuned carefully: too weak, and the policy overfits the reward model; too strong, and the policy does not move far enough from the SFT baseline to benefit from the preference signal. Finding the right beta requires iterative experimentation and human evaluation of outputs, not just automated metrics.

The practical lesson is that alignment is not a one-time operation. Models that are deployed and monitored will accumulate evidence about which failure modes were not addressed by the initial alignment process. The most effective teams treat fine-tuning as a continuous process: collecting evidence of failures, augmenting preference data to address them, and updating the model on a regular cadence. This requires both the infrastructure to collect and process feedback at scale and the evaluation machinery to confirm that each update improves the target behaviors without degrading others.

Evaluation: How to Know Whether Alignment Worked

A model that passes an alignment training pipeline is not necessarily aligned. The evaluation methodology determines what counts as success, and many evaluation approaches in common use measure only a subset of the behaviors that matter in deployment.

Automated benchmarks like MT-Bench and AlpacaEval measure the quality of responses on a fixed set of prompts, often using a stronger model as the judge. These benchmarks are useful for rapid iteration but have known limitations: model-as-judge evaluation introduces the biases and preferences of the judge model, models can be inadvertently trained to optimize for judge preferences rather than genuine quality, and fixed benchmark sets become less informative as models are fine-tuned on data that overlaps with the evaluation set.

Human evaluation remains the most reliable signal for measuring alignment quality, but it is expensive and slow to run at scale. The most effective evaluation programs combine automated benchmarks for rapid iteration with targeted human evaluation on distributions that are representative of the actual deployment use case. Evaluating on the intended use case matters: a model aligned for customer service interactions may show different strengths and weaknesses than the same model evaluated on general instruction-following benchmarks.

Red-teaming, where evaluators actively attempt to elicit harmful, incorrect, or undesirable outputs, complements standard benchmarks by finding failure modes that routine evaluation misses. The most systematic version of this is adversarial probing: presenting the model with inputs specifically designed to expose the gap between its stated alignment and its actual behavior. Perez et al. (arXiv:2202.03286) formalized red-teaming methodology and showed that models trained to refuse harmful requests in standard evaluation settings often comply when the same request is embedded in a more complex context. Alignment that holds only on surface-level evaluations is not alignment that can be trusted in deployment.

Building a robust evaluation program requires committing resources before deployment, not after. The teams that discover alignment failures through production monitoring are reacting to problems that have already reached users. The teams that discover them through pre-deployment red-teaming and structured evaluation can address them before they cause harm. The investment required is not large relative to the cost of a post-incident response, and the technical infrastructure for systematic evaluation, including diverse prompt suites, automated scoring pipelines, and human evaluation panels, scales efficiently once established. Treating evaluation as a first-class engineering concern, rather than an afterthought to training, is the defining characteristic of mature AI deployment practice.

The progression from pretraining to SFT to preference-based alignment represents the current consensus architecture for building models that are both capable and useful. It is not the final word. Research into online RL from human feedback, reward model distillation, and scalable oversight methods continues to evolve rapidly, and methods that seem experimental today are likely to be standard practice within a few years. Understanding the underlying objectives, the tradeoffs between pipeline complexity and alignment quality, and the failure modes of each approach is the foundation needed to evaluate new methods as they emerge and to make sound adoption decisions for any specific deployment context.

For practitioners working with existing aligned models rather than training their own, the most relevant insight is this: the alignment of a model is not a binary property. It is a set of behaviors that were reinforced through a particular combination of data, objectives, and evaluation, and that combination was designed by the lab that trained the model for their specific goals and constraints. Those goals may not perfectly match your deployment requirements. Understanding the alignment methods used by a model provider, and evaluating whether the resulting behaviors match your use case, is the due diligence that responsible deployment requires.