How to Fine-Tune an LLM Step by Step

Most fine-tuning tutorials show you a code block and call it done. This guide shows you the full process: every step from raw data to a deployed endpoint, with the reasoning behind each decision so you can adapt it to your own use case.

Fine-tuning a language model is not a single operation. It is a five-stage workflow, and the quality of each stage determines whether the final model is actually useful. You can write perfect training code on top of a bad dataset and get a confidently wrong model. You can build a great model and deploy it incorrectly and lose half the performance to serving overhead. This guide covers all five stages in the order they actually happen, with the most common mistakes at each stage and how to avoid them.

The Five-Step Process

Fine-tuning a language model follows a repeatable process. The steps in order are: define the task and collect training data; clean and format the dataset; configure and run LoRA fine-tuning; evaluate against a held-out test set; merge, quantize, and deploy. Each step feeds into the next. A shortcut at step two creates problems you will spend hours debugging at step four.

1. Define the task and collect training data
2. Clean and format the dataset
3. Configure and run LoRA fine-tuning
4. Evaluate against a held-out test set
5. Merge, quantize, and deploy

Step 1: Define the Task and Collect Data

Before writing a single line of code, write one sentence that describes your task in the form: "Given [input], produce [output]." For example: "Given a customer support email, classify it as billing, technical, or general and suggest a response."

This sentence determines what your examples look like and what metrics you use to measure success. Vague task definitions produce vague datasets that produce models with inconsistent behavior. "Make responses better" is not a task definition. "Reduce average response length from 400 to 150 words while maintaining the answer quality on the company's support topic list" is a task definition.

For data sources, the best option is real examples from your domain: actual inputs and outputs that represent what you want the model to do. If you have existing outputs from human experts, annotated logs, or structured records of good responses, those are gold. The second option is synthetic examples: you generate them using a capable general model and your real examples as seeds, then review each one for quality before including it. The third option is public datasets that approximate your task, filtered and reformatted to match your specific requirements.

How many examples do you need? The answer depends on the task. Zhou et al. in the LIMA paper showed that alignment tasks can be achieved with 1,000 high-quality examples. Narrow classification or extraction tasks often work with 200 to 500. Format-enforcement tasks (teaching the model to always output a specific JSON schema) can work with as few as 50 to 100 if they are highly consistent. The key insight is that quality beats quantity at every scale.

Step 2: Clean and Format the Dataset

Every training example needs two components: an instruction that describes what to do, and a response that is the ideal output. Some tasks also have an input field: the specific piece of content the instruction applies to. This three-field structure is called the Alpaca format, and it is the most common format for instruction fine-tuning.

# Format for instruction fine-tuning (Alpaca-style)
example = {
    "instruction": "Classify this support ticket.",
    "input": "My card was charged twice for the same order.",
    "output": "billing"
}

# Convert to training text
def format_example(ex):
    text = f"### Instruction:\n{ex['instruction']}\n"
    if ex.get('input'):
        text += f"### Input:\n{ex['input']}\n"
    text += f"### Response:\n{ex['output']}"
    return text

Before formatting, run five quality filters on every example in your dataset. First, remove duplicates: hash each instruction-output pair and discard any example where the hash has already been seen. Duplicates cause the model to over-weight those specific examples, producing a model that handles common cases well but fails on anything unusual.

Second, filter by response length. Responses that are too short (fewer than five tokens) often represent malformed or incomplete training examples. Responses that exceed your context window will be truncated during training in ways that introduce noise. Set upper and lower bounds based on what a normal response looks like for your task.

Third, check for contradictions: the same instruction with different outputs. These are especially common when your dataset was assembled by multiple annotators with different standards. A model trained on contradictions will produce inconsistent output, appearing to randomly choose between the contradictory behaviors.

Fourth, assess coverage balance. If 80 percent of your examples are one sub-type of the task, the model will perform well on that sub-type and poorly on the others, even if the others are equally important in production. Check the distribution across all categories your task involves.

Fifth, screen for PII and toxicity. Training data that contains personal information or harmful content will cause the model to reproduce those patterns in its outputs. This is both a safety issue and a compliance issue for most organizations.

See the full data cleaning script with all five filters in Module 2 of the free course.

Step 3: Configure and Run LoRA Fine-Tuning

LoRA (Low-Rank Adaptation, Hu et al. arXiv:2106.09685) adds small trainable matrices to the frozen base model. Instead of updating all parameters in the model, you add pairs of small matrices A and B to specific linear layers. The update to each layer is computed as the product of B and A (rank-r approximation). Only A and B are trained; the original weights are frozen.

This reduces the number of parameters you actually train from billions to tens of millions, making fine-tuning feasible on consumer hardware. A 7-billion-parameter model with LoRA rank 8 on the query and value projection matrices trains roughly 8 million parameters instead of 7 billion: less than 0.12 percent of the total parameters.

from peft import LoraConfig, get_peft_model, TaskType
from transformers import AutoModelForCausalLM, TrainingArguments
from trl import SFTTrainer

model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-hf")

lora_config = LoraConfig(
    r=8,              # rank: start here for most tasks
    lora_alpha=16,    # alpha = 2 * r is a solid default
    target_modules=["q_proj", "v_proj"],
    task_type=TaskType.CAUSAL_LM
)
model = get_peft_model(model, lora_config)

training_args = TrainingArguments(
    output_dir="./ft-output",
    num_train_epochs=3,
    per_device_train_batch_size=4,
    learning_rate=2e-4,
    warmup_steps=100,
    logging_steps=50,
    eval_strategy="epoch",
    save_strategy="epoch"
)

trainer = SFTTrainer(
    model=model,
    args=training_args,
    train_dataset=train_data,
    eval_dataset=eval_data,
    dataset_text_field="text"
)
trainer.train()

The two most important hyperparameters in LoRA are rank (r) and alpha. Rank controls how many parameters the adapter matrices contain. Higher rank means more expressiveness but more parameters to train and more risk of overfitting. A rank of 8 is a reasonable starting point for most tasks. Alpha controls the scaling of the LoRA updates relative to the base model. Setting alpha to twice the rank (alpha = 16 when r = 8) is a commonly used default that produces stable training.

For very low-resource settings (12 GB VRAM or less), use QLoRA (Dettmers et al. arXiv:2305.14314). QLoRA quantizes the base model to 4-bit precision before loading it, which reduces the memory footprint of the base model by roughly 4x, leaving most of the available VRAM for the LoRA adapter training. The final model quality is comparable to standard LoRA in most benchmarks.

Monitor training loss over epochs. Healthy training loss decreases smoothly and levels off. If it decreases to near zero and then starts to increase on the validation set, the model is overfitting. Reduce the number of epochs or increase regularization. If it does not decrease meaningfully, the learning rate may be too low, the dataset may have quality issues, or the task may not be well-suited to fine-tuning.

The most common fine-tuning mistake is evaluating on training data and calling it done. Real evaluation uses examples the model has never seen. That gap is where models fail in deployment.

Step 4: Evaluate

Never evaluate on your training data. Use a held-out test set of at least 50 examples that the model never saw during training. The test set should represent the actual distribution of inputs the model will encounter in deployment, not just the clean examples you chose for training.

Run two parallel evaluation tracks. The first is task-specific metrics: F1, exact match, ROUGE-L, or whatever is appropriate for your task. These tell you whether fine-tuning improved the specific capability you were targeting. The second is a general capability check: run the base model and the fine-tuned model on 20 to 30 general questions that your training data does not cover. This detects catastrophic forgetting, the phenomenon where fine-tuning on a narrow task causes the model to degrade on general capabilities it had before training.

A drop of more than 5 percentage points on general benchmark questions is a warning sign. It suggests the model has overfit to your fine-tuning dataset in ways that degraded its broader capabilities. Common fixes: reduce the number of training epochs, reduce the learning rate, or add a small number of general instruction-following examples to your training dataset to anchor the base behavior.

For tasks where automated metrics are insufficient (open-ended generation, subjective quality assessment), use LLM-as-judge evaluation. A capable general model evaluates each output on criteria you define (accuracy, tone, format adherence) on a 1 to 5 scale. Zheng et al. (arXiv:2306.05685) showed that GPT-4-based judges agree with human evaluators at a high rate on relative quality rankings, making this a scalable alternative to full human evaluation.

Maintain a set of regression test cases: examples where the base model got the answer right. After fine-tuning, verify that the model still passes these. A fine-tune that improves your target metric by 15 percent while breaking 10 previously-working examples may not be a net improvement, depending on the relative importance of those cases.

Step 5: Merge, Quantize, and Deploy

Once evaluation passes, the LoRA adapter exists as a separate set of weights alongside the frozen base model. Before deploying, you typically merge the adapter back into the base model weights using merge_and_unload(). The merged model has the same architecture as the base model and can be served with any standard inference framework, without requiring the PEFT library at runtime.

from peft import PeftModel
from transformers import AutoModelForCausalLM

base = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-hf")
model = PeftModel.from_pretrained(base, "./ft-output/checkpoint-best")
merged = model.merge_and_unload()
merged.save_pretrained("./merged-model")

After merging, quantize the model for inference efficiency. The two main formats are GPTQ (for GPU serving, arXiv:2210.17323) and GGUF (for CPU serving, used by llama.cpp). GPTQ 4-bit quantization reduces the model size by roughly 75 percent with minimal quality loss on most tasks, enabling faster inference and lower GPU memory requirements. GGUF enables CPU inference at reasonable speeds on modern hardware, which matters for deployment scenarios where GPU availability is limited or where data cannot leave on-premises infrastructure.

For serving, the simplest approach is a FastAPI application that loads the model at startup (not per-request) and exposes a generation endpoint. Hugging Face Inference Endpoints provides a managed alternative if you prefer not to manage infrastructure. For high-traffic use cases, vLLM provides efficient batched inference with continuous batching and PagedAttention, significantly improving throughput compared to naive sequential inference.

The Hidden Costs Most Teams Underestimate

The compute cost of running LoRA fine-tuning is relatively predictable. The costs most teams underestimate are the data costs. Collecting 500 high-quality examples from domain experts takes time. If those experts are billing at consulting rates, the data collection phase can cost more than the GPU compute. Annotation consistency review, contradiction detection, and format validation all add hours before the first training job runs.

The second hidden cost is iteration. Most first fine-tuning runs are not the final fine-tuning run. The first run tells you that your dataset has coverage gaps you did not anticipate. The second run tells you that your evaluation metrics do not capture the failure mode you actually care about. The third run is where you start to get results that feel genuinely useful. Budget for multiple iterations when planning a fine-tuning project.

The third hidden cost is maintenance. A fine-tuned model trained on data from six months ago may drift from your current needs as your product, policies, or domain evolves. Build a lightweight re-evaluation pipeline from the start so you can detect when the model's performance on your benchmarks degrades and trigger a re-training run.

For deeper coverage of every step with interactive exercises and real Python code, the Fine-Tuning LLMs free course walks through each stage with diagrams, quizzes, and working code examples you can run immediately.

When Fine-Tuning Is Not the Right Tool

This guide has focused on how to fine-tune, but the most valuable skill is knowing when not to. Fine-tuning is time-intensive and requires a data preparation investment that is not justified for every use case. Skip fine-tuning if the task changes frequently (frequent changes mean frequent re-training), if the failure mode is about missing information rather than inconsistent behavior (use RAG instead), or if you have fewer than 50 high-quality examples and no clear path to getting more.

Fine-tuning is also not a substitute for good prompting. Some teams rush to fine-tuning when a better system prompt would solve the problem faster. The test: can you describe the desired behavior in a paragraph? Can a capable model produce acceptable outputs with that description as context, and five examples? If yes, build the prompt and monitor it. Fine-tune only if the prompt fails consistently at scale, or if the volume makes per-request prompt length economically significant.

The FrugalGPT framework (Chen, Zaharia, Zou; arXiv:2310.11409) provides a useful lens here: the optimal inference strategy depends on cost, latency, and quality requirements simultaneously. Fine-tuning a smaller specialized model may be cheaper and faster than prompting a larger general model, but only if the quality delta justifies the preparation time. Build a cost-quality comparison before committing to a full fine-tuning pipeline.

Next in this series: Fine-tuning for beginners: the same concepts, explained for someone who has never run a GPU job before.