How to Fine-Tune a Language Model: A Practical Guide

Fine-tuning is one of the most misunderstood concepts in applied AI. Organizations either treat it as a universal solution to every model quality problem or as an exotic technique that requires a research team. Neither is right. Fine-tuning is a specific tool with a specific job: adapting a general pretrained model to behave differently on a specific task or in a specific style, when prompting alone does not produce consistent enough results.

The question of when to fine-tune versus when to use prompting techniques is practical and answerable. The answer depends on how consistent the behavior needs to be, how many examples of the desired behavior you have, and how much compute you are willing to spend. This post covers that decision, then walks through how fine-tuning works in detail, including the parameter-efficient techniques that have made it accessible without large compute budgets.

Module 6 of the free Model Development Course, linked at the end of this post, covers fine-tuning and alignment in full, including the relationship between fine-tuning and RLHF, how instruction tuning works, and what the capstone project involves. This post provides the conceptual foundation for that module.

What Fine-Tuning Actually Does

A pretrained language model has learned a broad statistical model of language from its training data. It can complete sentences, answer questions, write code, and do many other things because those patterns appeared in its pretraining corpus. What it has not learned is your specific task, your preferred output format, your domain terminology, or your behavioral constraints.

Fine-tuning continues the training process on a smaller, curated dataset that represents the target behavior. The model's weights, which were frozen at the end of pretraining, are unfrozen and updated through additional gradient descent steps on the fine-tuning data. After fine-tuning, the model's weights encode both the general language model knowledge from pretraining and the specific patterns from the fine-tuning data.

The key insight is that fine-tuning does not erase pretraining knowledge. It builds on top of it. A model fine-tuned on medical question-answering still knows about history, science, and literature from pretraining; it has just been given additional training that makes it more reliable and better calibrated on medical questions. This is why fine-tuning on a small domain-specific dataset can produce large improvements on domain tasks even though the dataset is tiny relative to the pretraining corpus.

Fine-tuning builds on top of pretraining. It teaches specific behavior without erasing general knowledge.

When to Fine-Tune and When to Use Prompting

The most important decision in applied language model work is whether to invest in fine-tuning or to achieve the target behavior through prompt engineering, few-shot examples in the context, or retrieval-augmented generation. Fine-tuning is not always the right answer, and understanding when it is and is not worth the investment prevents wasted effort.

A practical way to make this decision: start with prompting. Iterate on the prompt until you hit a quality ceiling. If that ceiling is below your requirement and you have high-quality labeled data, then fine-tuning is likely to help. If you hit the quality ceiling with a small dataset, fine-tuning is premature. The data problem must be solved first.

Full Fine-Tuning: What It Means and What It Costs

Full fine-tuning updates all of the model's parameters during the fine-tuning training run. For a large language model, this means updating hundreds of billions of parameters, which requires substantial compute and memory. Full fine-tuning of a 7 billion parameter model requires roughly 14-28 GB of GPU memory just to hold the model weights in the formats needed for training, before adding the optimizer states that gradient descent requires.

The practical consequence is that full fine-tuning of large models requires multiple high-end GPUs or access to cloud compute, which translates to real cost. For many applications, this cost is justified by the quality improvement. For prototyping and smaller organizations, it is often prohibitive. Parameter-efficient fine-tuning methods were developed specifically to address this constraint.

LoRA: Low-Rank Adaptation

LoRA, introduced by Hu et al. (arXiv:2106.09685), is the most widely used parameter-efficient fine-tuning technique in practice. The key observation it builds on is that the changes to the model's weight matrices during fine-tuning can be well-approximated by low-rank matrices: matrices with far fewer independent values than a full weight matrix. Instead of updating all weights directly, LoRA adds small trainable matrices alongside the original weights and trains only those additions, keeping the original weights frozen.

The dimension reduction is substantial. A weight matrix in a large language model might have dimensions of 4096 by 4096, giving over 16 million parameters. LoRA represents the update to this matrix as the product of two smaller matrices, one of dimension 4096 by 8 and one of dimension 8 by 4096, which together have roughly 65,000 parameters. The ratio of trainable parameters to total parameters drops from 100% to under 0.35% for GPT-3-scale models, according to the original paper.

The practical effects are significant. Training memory requirements drop substantially because optimizer states, which are large, only need to be maintained for the LoRA parameters. Training time decreases accordingly. Multiple fine-tuned variants of the same base model can be maintained with low storage overhead: rather than storing a full copy of the fine-tuned model for each task, you store the base model once and a small set of LoRA adapter weights for each variant.

Instruction Tuning: Teaching Models to Follow Instructions

Most of the language models you interact with today are not base pretrained models: they are instruction-tuned variants. Instruction tuning is a form of fine-tuning where the training data consists of pairs of instructions and desired responses. The fine-tuning process teaches the model to interpret instructions and produce appropriate responses, rather than simply continuing text in the style of its pretraining corpus.

The difference between a base model and an instruction-tuned model is substantial. A base GPT-3, when given the prompt "What is the capital of France?", would likely continue the text in a way that adds more context or questions rather than simply answering "Paris." An instruction-tuned model recognizes the instruction format and produces the expected answer. This shift in behavior comes entirely from the fine-tuning data, not from any architectural change.

The quality and format of instruction-tuning data matters enormously. Effective instruction-tuning datasets are diverse (covering many different types of instructions), high quality (the responses are genuinely good), and balanced (no single instruction type dominates). Early open-source instruction-tuning efforts like Stanford Alpaca used synthetic data generated from existing language models, which introduced quality and consistency constraints. Later efforts used more carefully curated human-written responses, which generally produced better-behaved models.

RLHF: Reinforcement Learning from Human Feedback

Reinforcement learning from human feedback (RLHF) is a further adaptation technique used to align model behavior with human preferences. It was a key component of the training process for ChatGPT and described in detail by Ouyang et al. (arXiv:2203.02155). RLHF works in three stages.

The first stage is supervised fine-tuning on human-written responses to a diverse set of prompts, establishing a baseline instruction-following behavior. The second stage trains a separate reward model: a model that takes a prompt and a response as input and outputs a score representing how much a human reviewer would prefer that response. Human annotators compare pairs of model responses and indicate which they prefer; these preferences are used to train the reward model.

The third stage uses reinforcement learning (specifically, a variant called proximal policy optimization or PPO) to update the language model's policy so that it produces responses the reward model scores highly. A KL penalty term prevents the model from drifting too far from the supervised fine-tuning baseline, which would produce outputs that game the reward model without actually being better responses.

RLHF is computationally expensive and requires careful implementation to avoid reward hacking: the model finding ways to score well on the reward model without actually producing better outputs. Simpler alignment approaches like Direct Preference Optimization (DPO, Rafailov et al., arXiv:2305.18290) have emerged as alternatives that achieve similar behavioral improvements without the reinforcement learning loop.

What You Need to Fine-Tune Successfully

Success with fine-tuning requires three things to be right simultaneously: the right data, the right starting model, and the right evaluation.

The right data

Fine-tuning data quality determines fine-tuning outcome more than any other factor. For instruction fine-tuning, the data should consist of diverse, representative instruction-response pairs where the responses exemplify exactly the behavior you want. Inconsistent responses, responses that partially follow the desired behavior, or responses that include patterns you want to suppress will all be learned by the model. Garbage in, garbage out applies with even more force here than in pretraining, because the fine-tuning signal is meant to override pretraining patterns and a noisy signal will produce inconsistent behavior.

The right starting model

Fine-tuning works best when the base model already has the general capabilities needed for the target task. Fine-tuning can specialize and shape behavior; it cannot install capabilities the base model does not have. A 1 billion parameter model fine-tuned on legal text is unlikely to match the legal reasoning performance of a 70 billion parameter model, regardless of fine-tuning data quality. Choose a starting point that has demonstrated general capability in the domain, then fine-tune to shape the specific behavior you need.

The right evaluation

Fine-tuning without rigorous evaluation is building blind. Before starting, define what success looks like in measurable terms: accuracy on a held-out test set, human preference ratings on a sample of outputs, or task-specific metrics. Run this evaluation on the base model before fine-tuning to establish a baseline. Run it after fine-tuning to confirm improvement. Run it on a test set that was completely held out from the fine-tuning process to get an unbiased estimate of the model's behavior on genuinely unseen inputs.

QLoRA: Fine-Tuning on Consumer Hardware

QLoRA, introduced by Dettmers et al. (arXiv:2305.14314), extends LoRA with quantization to make fine-tuning large models possible on consumer-grade hardware. Quantization reduces the precision of the model's weights from 32-bit or 16-bit floating point to 4-bit integers, which dramatically reduces memory requirements. QLoRA combines 4-bit quantization of the frozen base model with 16-bit LoRA adapters, allowing the adapters to be trained at full precision while the base model occupies less memory.

The practical implication is that QLoRA makes it feasible to fine-tune a 7 billion parameter model on a single consumer GPU with 24 GB of memory, or even on an older 16 GB GPU with careful configuration. This opened fine-tuning to practitioners who do not have access to multi-GPU cloud infrastructure, which was a meaningful shift in what was practically achievable outside large organizations.

The tradeoff is modest quality loss relative to full fine-tuning or standard LoRA. For most applied use cases, this quality tradeoff is acceptable given the large reduction in compute cost. For tasks requiring maximum quality, full fine-tuning on higher-precision hardware is still preferable. As with most engineering decisions in ML, the right choice depends on the specific quality requirements and compute constraints of the application.

Prompt tuning and soft prompts

Prompt tuning, introduced by Lester et al. (arXiv:2104.08691), is a parameter-efficient alternative to LoRA that trains a small set of learnable "soft prompt" tokens prepended to every input, while keeping all model weights frozen. The soft prompt vectors are learned through gradient descent to elicit the desired behavior from the model. Prompt tuning requires even fewer trainable parameters than LoRA and works well when the task can be framed as a prompted completion problem. Its main limitation is that it tends to underperform LoRA on tasks that require substantial behavioral change from the pretrained baseline.

Catastrophic Forgetting and How to Mitigate It

Catastrophic forgetting is the tendency of a neural network to lose knowledge it had previously acquired when it is trained on new data. In the context of fine-tuning, it manifests as the model becoming better at the fine-tuning task while becoming worse at general tasks it could handle before. A model fine-tuned aggressively on legal text might become less reliable on general conversational tasks.

Several techniques reduce catastrophic forgetting. Low learning rates during fine-tuning limit how much the weights move from their pretrained values. Mixing a small proportion of general instruction-following data into the fine-tuning dataset helps the model retain general capabilities. LoRA inherently reduces catastrophic forgetting because the original weights are frozen: only the adapters change, so the base model's capabilities are preserved by design.

For applications where broad general capability must be preserved alongside domain specialization, LoRA is generally the better choice over full fine-tuning. The tradeoff is that LoRA has somewhat less capacity to specialize behavior than full fine-tuning, particularly for tasks that require substantial behavioral change from the pretrained baseline.