From Zero to Model: A Complete Course in AI Development
Module 6 of 7

Fine-tuning and Alignment

Pre-training gives a model general capability. Fine-tuning gives it a job. This module explains how to adapt a foundation model to a specific task without training from scratch, how researchers teach models to be helpful and safe, and why alignment is one of the hardest open problems in AI.

By the end of this module you will be able to

The Problem with a Raw Pre-trained Model

Back in Module 3, we introduced transfer learning as the most powerful shortcut in modern AI: starting with a model already trained on enormous amounts of data and adapting it to your specific task. Now we get into how that actually works.

A model that has been pre-trained on a large text corpus has absorbed a remarkable amount of knowledge: grammar, facts, reasoning patterns, coding conventions, writing styles. But it has learned one specific thing: how to predict the next token. Ask it a question and it will, by default, try to continue the text in whatever direction seems most statistically likely based on its training data. That might produce a useful answer. It might also produce a plausible-sounding continuation that ignores your question entirely, rambles, or invents facts.

A raw pre-trained model is not a useful assistant. It is a very capable text-continuation engine. To turn it into something useful for a specific purpose, you need to fine-tune it.

What Fine-tuning Is

Fine-tuning is the process of continuing to train a pre-trained model on a smaller, targeted dataset, adjusting its weights to make it better at a specific task. The pre-trained weights are not discarded. They are the starting point: the model begins with all the knowledge it absorbed during pre-training and is then nudged by the fine-tuning data toward the specific behaviour you want.

This is why fine-tuning requires far less data and compute than pre-training from scratch. You are not teaching the model language, reasoning, or general knowledge. It already knows those things. You are teaching it how to apply what it knows to your specific domain, format, or style of task.

Analogy: the medical residency

A doctor who graduates from medical school already knows biology, chemistry, anatomy, pharmacology, and clinical reasoning. They have not specialised yet, but the foundational knowledge is there. A residency in cardiology does not re-teach them biology. It trains them to apply what they already know to a specific domain: reading ECGs, interpreting cardiac imaging, managing arrhythmias.

Fine-tuning works the same way. The pre-trained model is the medical school graduate. The fine-tuning dataset is the residency. You are not teaching general intelligence. You are directing existing capability toward a specific purpose.

Three Approaches to Fine-tuning

Not all fine-tuning is the same. There are three main approaches, each with different trade-offs in cost, data requirements, and how deeply the model is changed.

Full fine-tuning updates all of the model's weights on the new dataset. Every parameter can shift. This produces the strongest adaptation to the target task but requires the most compute, the most memory, and carries the greatest risk of catastrophic forgetting: the model gets better at your task but loses some of its general capabilities.

Instruction fine-tuning trains the model on a dataset of instruction-and-response pairs: human-written examples of good questions and good answers. The model learns the format of being a helpful assistant. This is how a raw pre-trained model becomes a chat-capable model. The dataset is typically much smaller than the pre-training corpus, often tens of thousands to hundreds of thousands of examples rather than trillions of tokens.

Parameter-efficient fine-tuning (PEFT) freezes most of the pre-trained weights and adds a small number of new trainable parameters. LoRA (Low-Rank Adaptation), introduced by Hu et al. in arXiv:2106.09685, is the most widely used technique: it inserts small adapter matrices into the model's layers and trains only those, leaving the original weights unchanged. The result is a fine-tuned model with a tiny fraction of the compute cost, whose original capabilities are preserved because the base weights are never modified.

Why LoRA matters for enterprise deployment A LoRA adapter for a large model might be just a few hundred megabytes, while the base model itself is many gigabytes. You can swap adapters to switch the model between tasks, store multiple specialisations cheaply, and update them without re-training the whole model. This makes domain-specific fine-tuning practical for organisations that cannot afford full training runs.

What Alignment Means

A fine-tuned model that is good at its task is not necessarily a safe or trustworthy one. A model might be highly capable at generating text and still produce harmful content, make up facts with great confidence, or follow instructions in ways that technically fulfil the letter of the request while violating the intent.

Alignment is the field concerned with making AI systems behave in accordance with human values and intentions. For a language model, alignment typically targets three properties, sometimes described as the three Hs:

These three goals are not always easy to satisfy simultaneously. A model optimised purely for helpfulness might produce harmful content when asked. A model trained to be harmless might refuse too many legitimate requests. A model trained to express appropriate uncertainty might feel less useful than one that answers confidently, even when confidence is not warranted. Alignment research is largely the study of how to hold all three properties in balance.

Reinforcement Learning from Human Feedback

The dominant technique for alignment is RLHF: reinforcement learning from human feedback. The core idea was developed in research by Christiano et al. (arXiv:1706.03741) and applied at scale in the InstructGPT work by Ouyang et al. (arXiv:2203.02155). It works in three stages.

Stage 1: Supervised fine-tuning. The pre-trained model is first fine-tuned on a dataset of examples where human labellers wrote high-quality responses to prompts. This gives the model a baseline sense of what good responses look like.

Stage 2: Training a reward model. Human labellers are shown two or more responses to the same prompt and asked which one they prefer. These preference judgements are used to train a separate model, called the reward model, that learns to predict human preferences. Given any response, the reward model assigns it a score representing how much a human rater would prefer it.

Stage 3: Reinforcement learning. The language model is treated as an agent making decisions. For each prompt it receives, it generates a response. That response is scored by the reward model. The language model's weights are then updated to make responses that received high scores more likely in the future. Over many iterations, the model learns to generate responses that humans tend to prefer.

Analogy: the editorial team

Imagine a new writer at a newspaper who has read millions of articles and is technically capable of writing in any style. The editor does not rewrite every article from scratch. Instead, the editor reads drafts, marks which versions of a passage are better, and the writer gradually learns the editorial standards through this feedback.

In RLHF, the language model is the writer. The human labellers are the editorial team. The reward model is a distillation of those editorial preferences into a system that can score any draft automatically. And the reinforcement learning step is the writer updating their instincts based on the accumulated feedback.

Beyond RLHF: DPO and Constitutional AI

RLHF requires training an additional reward model and running an expensive reinforcement learning loop. Researchers have developed alternatives that aim for similar alignment outcomes with less complexity.

Direct Preference Optimisation (DPO), introduced by Rafailov et al. (arXiv:2305.18290), reframes RLHF mathematically and shows that the same objective can be achieved by training directly on preference data, without needing a separate reward model or RL training loop. The model is trained to increase the probability of preferred responses and decrease the probability of dispreferred ones, in a single stable supervised step.

Constitutional AI (CAI), from Bai et al. (arXiv:2212.06950), takes a different approach. Instead of relying entirely on human preference labels, it gives the model a set of written principles (a "constitution") and has the model critique and revise its own outputs according to those principles. This reduces the dependence on human labellers for harmlessness training, since the model can generate its own feedback using the constitutional rules.

The Alignment Tax

Alignment training sometimes comes at a cost. A model that has been trained to refuse harmful requests may also become more likely to refuse borderline-but-legitimate requests. A model trained to express uncertainty may hedge even when it is actually correct. These degradations in raw capability from the alignment process are sometimes called the alignment tax.

The alignment tax is not inevitable or fixed. Researchers continue to find ways to improve alignment while preserving capability. But it illustrates a fundamental tension: optimising strongly for one objective (safety, caution, harmlessness) can shift the model away from other objectives (helpfulness, accuracy, directness). Getting the calibration right, per task and per deployment context, is one of the central engineering challenges in applied AI.

What comes next You now have the complete picture of how a language model is built: from raw data and tokenization, through the training loop and Transformer architecture, through scaling and evaluation, to fine-tuning and alignment. The Capstone module brings it all together with a toy language model you can train in your browser.
Interactive 1 — Fine-tuning Method Comparison Select a method

Click each fine-tuning approach to see what it changes, when to use it, and what it costs. These trade-offs determine which approach is practical for your organisation.

Interactive 2 — RLHF Pipeline Click each stage

Reinforcement learning from human feedback turns a raw text predictor into an aligned assistant in three stages. Click each step to understand what happens and why it is necessary.

1 Supervised Fine-tuning (SFT) Starting point
Human labellers write high-quality responses to a diverse set of prompts. The pre-trained model is then fine-tuned on these examples using standard supervised learning: the model sees the prompt, predicts the response token by token, and is trained to match the human-written answer. This gives the model a baseline sense of what a good, helpful response looks like. Without this step, the raw model would not know that a question is being asked or that it is expected to answer it helpfully.
2 Reward Model Training Learning preferences
The SFT model generates multiple responses to the same prompt. Human labellers rank or compare these responses, indicating which they prefer. A separate neural network, called the reward model, is trained on these preference pairs to predict human judgements. Given any (prompt, response) pair, the reward model outputs a score. This score is not hand-programmed. It is learned from thousands of human comparisons, capturing subtle preferences around helpfulness, honesty, tone, and safety that would be very hard to specify as explicit rules.
3 Reinforcement Learning (PPO) Optimising for human preference
The SFT model is now treated as an agent. For each prompt it receives, it generates a response. The reward model scores that response. The language model's weights are updated using a reinforcement learning algorithm called Proximal Policy Optimisation (PPO) to make higher-scoring responses more likely in the future. A key detail: a penalty is added to stop the model from drifting too far from the SFT checkpoint. Without this constraint, the model would quickly learn to produce responses that game the reward model's scoring without actually being more helpful, a phenomenon called reward hacking.
4 The Result: an Aligned Assistant What changes
After RLHF, the model's responses shift in measurable ways: it follows instructions more reliably, declines genuinely harmful requests more consistently, expresses appropriate uncertainty rather than confabulating, and generally produces outputs that humans find more useful and trustworthy. The underlying Transformer architecture and pre-trained knowledge are unchanged. What has changed is the distribution of responses the model generates: it has learned to prefer the kinds of outputs that human raters preferred during training.
Module 6 Quiz · 4 Questions
1. A company wants to adapt a large general-purpose language model to answer customer support questions about their specific product. They have 50,000 examples of good support conversations. Which approach is most practical?
2. Why does fine-tuning require far less data and compute than pre-training from scratch?
3. In RLHF, what is the purpose of the reward model?
4. What is the "alignment tax" and why does it occur?
← Module 5: Scaling and Evaluation Capstone: Build Your Own Model →