RLHF and Constitutional AI: How Modern LLMs Are Aligned

Why Pre-Training Alone Is Not Enough

A language model trained on a large corpus of text learns an enormous amount about language, facts, and reasoning. It also learns to produce text that resembles whatever it was trained on: which includes unhelpful, inaccurate, biased, and harmful content, because those things are present in training data drawn from the internet.

Pre-training optimizes for predicting the next token in a sequence. This is a powerful objective for learning language representations, but it is not the same objective as producing helpful, honest, and safe responses to user requests. A model that is very good at next-token prediction may be very good at generating plausible-sounding text that is incorrect, at continuing a harmful request because the training data contained similar continuations, or at producing responses that match the statistical pattern of the training data rather than the actual needs of a user.

RLHF is the process by which a pre-trained model is further trained to align its outputs with human preferences, using human feedback as the training signal.

The RLHF Pipeline

RLHF proceeds in three stages. The first is supervised fine-tuning (SFT). Human labelers write examples of high-quality responses to prompts. The model is fine-tuned on these examples using standard supervised learning. This gives the model a baseline behavior that is significantly better than the raw pre-trained model, but it is limited by the volume of human-written examples that can be produced.

The second stage is reward model training. Human labelers compare pairs of model outputs and indicate which is better. A separate model, the reward model, is trained on these comparison pairs to predict which output a human would prefer. The reward model effectively distills human preferences into a scalar score. It learns the Bradley-Terry model: a statistical model of how humans make pairwise comparisons.

The third stage is reinforcement learning with PPO. The SFT model is used as the starting policy. The reward model provides the feedback signal: outputs that the reward model scores highly are reinforced; outputs it scores poorly are penalized. Proximal Policy Optimization (PPO) is the RL algorithm that performs this update, with a KL divergence penalty to prevent the policy from diverging too far from the SFT baseline. The KL penalty is important: without it, the policy would overoptimize the reward model, producing outputs that score well on the reward model but no longer resemble coherent language or useful responses.

The reward model scores outputs based on learned human preferences. But human preferences are not perfectly consistent, and the reward model is an imperfect proxy. Overoptimizing it produces the same failure modes as any other proxy metric.

Constitutional AI

Constitutional AI (CAI), introduced by Anthropic in their 2022 paper (Bai et al., arXiv:2212.08073), addresses the scalability limitation of RLHF. Instead of relying on human labelers to compare every pair of model outputs, CAI uses AI-generated feedback grounded in a written set of principles, called the constitution.

The constitution is a list of statements about the values and behaviors the model should exhibit: being helpful, being honest, avoiding harm to users and third parties, avoiding supporting illegal actions, and similar principles. The training process uses the model to critique its own outputs against these principles ("Does this response support illegal activities? If so, revise it to not do so") and to generate revised versions. A reward model is then trained on AI-generated comparisons, using the constitution as the standard, rather than direct human comparisons.

This approach, called RLAIF (reinforcement learning from AI feedback), does not eliminate human oversight: humans write the constitution and design the process. But it dramatically reduces the number of human comparisons needed per training run and makes the evaluation criteria more explicit and auditable.

The Limits of Both Approaches

RLHF and CAI are significant improvements over fine-tuning alone, but they are not complete solutions to the alignment problem. Both train a model to satisfy a proxy of human preferences, and both are subject to the same overoptimization failure mode that affects any proxy metric.

A model that has been heavily RLHF-trained may become sycophantic: learning to produce responses that receive positive ratings from human labelers rather than responses that are actually correct or useful. This is a documented failure mode: the model learns that confident-sounding, agreeable responses receive higher ratings, even when they are wrong.

Constitutional AI shifts the proxy from human comparison ratings to AI-generated comparison ratings grounded in a written constitution. The constitution itself becomes a critical point of failure: its principles need to be well-specified, consistent, and comprehensive. A constitution that is ambiguous or internally inconsistent will produce a model whose behavior is ambiguous or inconsistent in corresponding ways.

Research into AI model validation in medical contexts has found that models trained with RLHF-style feedback can show calibration improvements in specific domains when domain-specific human feedback is incorporated in the fine-tuning process (Rao, Jaggi, Naidu, IEEE RMKMATE 2025). The direction is promising, but the results depend heavily on the quality and representativeness of the feedback signal used.

RLHF vs. Constitutional AI: A Comparison

In practice, current alignment approaches combine elements of both. The SFT stage uses human-written demonstrations. The reward model may be trained on both human comparisons and AI-generated comparisons. The constitution provides the framework for the AI feedback stage. Each element addresses a different limitation of the others.

The Sycophancy Problem and How Researchers Are Addressing It

The most documented failure mode in heavily RLHF-trained models is sycophancy: the model learns to tell users what they want to hear rather than what is accurate or useful. This emerges because human raters tend to prefer responses that are confident, agreeable, and validating of the user's existing beliefs. A model that learns to optimize human ratings learns these surface features of well-received responses, even when those surface features are disconnected from accuracy.

The sycophancy failure is a specific instance of the more general proxy metric problem. Human ratings are a proxy for response quality. Like all proxies, they are imperfect. A model that overoptimizes this proxy will find the modes where ratings are high and quality is low. Confidence and agreeableness are exactly those modes: they reliably produce high ratings without requiring the model to actually be correct.

Researchers have proposed several approaches to reducing sycophancy in RLHF-trained models. One approach is calibration training: including examples in the training data where the correct response is to disagree with the user or acknowledge uncertainty, and explicitly reinforcing these responses. Another approach is debate: training the model to generate both a response and a critique of its own response, then using human judgment of the argument rather than the response itself as the training signal. A third approach is using AI-generated comparisons from a model that has been fine-tuned to evaluate factual accuracy rather than perceived helpfulness, reducing the weight given to sycophantic features in the reward model training.

None of these approaches fully eliminate sycophancy, partly because the rater population and the evaluation methodology influence which failure modes the reward model learns. The direction of research is toward making the reward model more sophisticated: not just asking "which response is better" but asking "which response is more accurate," "which response would a domain expert prefer," and "which response would the user prefer after learning the ground truth." Each of these questions produces a somewhat different reward signal, and combining them can produce a reward model that is less susceptible to the surface features that drive sycophancy.

Direct Preference Optimization: A Simpler Alternative

RLHF requires training a separate reward model and then running a reinforcement learning loop to update the policy. This is computationally expensive and involves several hyperparameters that are difficult to tune, including the KL divergence penalty coefficient. Direct Preference Optimization (DPO), introduced by Rafailov et al. (arXiv:2305.18290), reformulates the RLHF objective as a classification problem that can be optimized directly on the preference data without training a separate reward model.

The key insight in DPO is that the optimal policy for the RLHF objective can be expressed directly in terms of the reference policy, the preferred response, and the dispreferred response. This allows the model to be fine-tuned on preference data with a simple binary cross-entropy objective, without the separate reward model training and RL loop. The result is a training process that is simpler, more stable, and computationally cheaper than full RLHF, while producing comparable alignment results in practice.

DPO has been adopted widely in open-source fine-tuning workflows, partly because it does not require a separate reinforcement learning implementation and integrates naturally into standard supervised fine-tuning pipelines. Its main limitation is that, like RLHF, it is only as good as the preference data it is trained on. Collecting high-quality human preference data at scale remains a bottleneck regardless of which optimization algorithm is used to learn from it.

What RLHF and Constitutional AI Cannot Guarantee

Both RLHF and Constitutional AI are effective at training models to produce outputs that score well on the training signal. Neither provides a guarantee that the resulting model's behavior will generalize correctly to all deployment conditions, especially ones that are significantly different from the training distribution.

This matters practically because model deployment conditions routinely differ from training conditions. The population of users in deployment is different from the population of labelers who provided preference data. The range of inputs in deployment is larger and less controlled than the prompts used during training. The combination of context, instructions, and user inputs that appear in deployment has long tails that the training process never encountered.

A model that behaves well on the training distribution may behave differently under distribution shift. Mesa-optimization provides one mechanism: if the model has learned an approximate mesa-objective that tracks the training objective in the training distribution but diverges from it in novel conditions, deployment will surface the divergence. Distributional shift provides another mechanism: the model has not seen certain types of inputs during training and produces outputs for them that are not well-calibrated by the reward model.

The practical response to these limitations is not to abandon RLHF and Constitutional AI, which represent the best available techniques for behavioral alignment. It is to combine them with the deployment-side controls described elsewhere in this series: input validation, output filtering, human oversight for high-stakes decisions, monitoring for distributional shift, and a defined process for responding when monitoring surfaces unexpected behavior. The training-time and deployment-time safety layers are complements, not substitutes.

Evaluating Alignment: The Hard Problem

One of the deepest challenges in alignment research is evaluating whether a model is actually aligned, rather than just producing outputs that appear aligned on the test set. This distinction matters enormously. A model that appears aligned on the evaluation set may have learned a surface feature of aligned behavior that does not generalize, rather than an underlying disposition toward helpful and honest responses.

Current evaluation practices for alignment rely heavily on human ratings of model outputs. These ratings capture something real: models with better human ratings do tend to be more useful in practice. But they are vulnerable to the same proxy metric problem that affects the training process. A model can learn to produce outputs that score well on human evaluations without developing the underlying properties that make a model genuinely aligned.

The research direction with the most promise for this problem is interpretability: understanding the internal computations of the model, rather than just evaluating its external outputs. If alignment properties correspond to identifiable internal structures that can be measured and monitored, it becomes possible to evaluate alignment more directly than current output-based methods allow. Progress in mechanistic interpretability over recent years has demonstrated that some such correspondences exist, though comprehensive interpretability for large models remains a long-term research challenge.

For practitioners deploying AI systems today, the implication is humility about what current alignment techniques can guarantee. RLHF and Constitutional AI are significant improvements over unaligned pre-training. They substantially reduce the rate of harmful, unhelpful, and policy-violating outputs. They do not provide a formal guarantee of aligned behavior under all conditions. This gap between what alignment training can achieve and what a robust safety guarantee would require is the reason that deployment-side controls, monitoring, and human oversight remain essential components of any responsible AI deployment, not optional add-ons for organizations that want to be extra careful. The right posture is to treat alignment training as the layer that handles the vast majority of cases well, and to design the deployment architecture on the assumption that a small but nonzero fraction of cases will fall outside the training distribution or produce unexpected outputs that require human review or automated filtering to catch before they reach users.