LoRA and QLoRA: Fine-Tuning Without a Supercomputer

In 2021, fine-tuning a 7-billion-parameter model required 8 high-end GPUs and roughly $4,000 for a single training run. By 2023, the same model could be fine-tuned on a single consumer GPU for under $10. LoRA made this possible. This module explains exactly how.

By the end of this module, you will be able to explain what LoRA does to a weight matrix, choose appropriate rank and alpha values for your task, and understand how QLoRA extends LoRA with quantization to further reduce memory use.

The Problem: Full Fine-Tuning is Expensive

A language model stores its knowledge in weight matrices. A 7-billion-parameter model has roughly 7 billion numbers, each stored as a 16-bit float. That's about 14 GB just to store the weights. During full fine-tuning, you also need gradients and optimizer states — typically 3-4x the model size. Full fine-tuning a 7B model needs roughly 56–80 GB of GPU memory.

Consumer GPUs typically have 8–24 GB. Enterprise GPUs (A100) have 40–80 GB and cost thousands of dollars per hour on cloud services. Full fine-tuning is simply out of reach for most teams.

The Insight Behind LoRA

Hu et al. (arXiv:2106.09685) made a key observation: when you fine-tune a pre-trained model on a downstream task, the changes to the weight matrices have a low intrinsic rank. In plain terms, the meaningful changes during fine-tuning live in a much smaller mathematical space than the full weight matrix.

Instead of updating the full weight matrix W (which might be 4096 × 4096 = 16 million parameters), LoRA adds two small matrices A and B, where the product B×A approximates the weight update. If rank r = 8, then A is 4096×8 and B is 8×4096 — totaling only 65,536 parameters instead of 16 million. That's a 244x reduction.

The Two LoRA Hyperparameters You Control

Rank (r)

Rank controls how many parameters the adapter matrices contain. Higher rank means more expressive adapters but more memory and compute. Common values are 4, 8, 16, and 32. Start with r=8 for most tasks. Use r=16 or r=32 for complex reasoning or multi-task fine-tuning.

Alpha (α)

Alpha scales the LoRA update relative to the original weights. The effective learning rate of the adapter is proportional to α/r. A common rule of thumb is to set α = 2×r. If r=8, use α=16. This keeps the adapter update magnitude stable as you change rank.

QLoRA: Adding Quantization to LoRA

QLoRA (Dettmers et al., arXiv:2305.14314) extends LoRA by storing the frozen base model weights in 4-bit precision instead of 16-bit. This cuts the base model's memory use by 4x with minimal quality loss on most tasks.

The combination is powerful: with QLoRA, a 7B model that would normally need 56 GB can be fine-tuned on a single 12 GB consumer GPU. This is what made fine-tuning accessible to individual researchers and small teams.

from peft import LoraConfig, get_peft_model, TaskType
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
import torch

# Optional: load base model in 4-bit (QLoRA)
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",           # NormalFloat4 — optimal for LLM weights
    bnb_4bit_compute_dtype=torch.float16,
    bnb_4bit_use_double_quant=True        # nested quantization saves ~0.4 bits extra
)

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-2-7b-hf",
    quantization_config=bnb_config,       # remove this line for standard LoRA
    device_map="auto"
)

# Configure LoRA
lora_config = LoraConfig(
    r=8,                                  # rank: start here, raise to 16/32 if needed
    lora_alpha=16,                        # scaling: rule of thumb = 2 * r
    target_modules=["q_proj", "v_proj"], # apply LoRA to attention projections
    lora_dropout=0.05,
    bias="none",
    task_type=TaskType.CAUSAL_LM
)

model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# Trainable params: 4,194,304 || All params: 6,742,618,112 || Trainable: 0.06%

Which Layers Should LoRA Target?

LoRA can be applied to any weight matrix in the model. The most common choice is the attention projection matrices: q_proj (query) and v_proj (value). Some practitioners also add k_proj and o_proj. Adding more target modules increases the number of trainable parameters, which can improve performance but also increases memory and training time.

A good default for instruction fine-tuning: target q_proj and v_proj with rank 8. If you see poor performance after 2 epochs, try adding k_proj and o_proj, or increase rank to 16.