Why Fine-Tune? When Prompts Are Not Enough

By the end of this module, you will be able to explain the difference between a pre-trained model and a fine-tuned one, name three situations where fine-tuning beats prompt engineering, and three situations where it doesn't.

What a Pre-Trained Model Actually Knows

When a company trains a large language model on the internet, the model learns patterns from billions of text samples. It learns grammar, logic, facts, code, and a wide range of writing styles. This is called pre-training.

Pre-training is expensive. It can cost millions of dollars in compute. The result is a model with broad, general knowledge — the kind of model that can write a poem, explain quantum physics, and translate Spanish, all without any additional training.

But "broad" is the key word. The model has never seen your internal documents. It does not know your company's terminology. It has no idea how your customers phrase support requests. And it has been trained to be helpful across all topics, which means its outputs tend toward the generic middle, not the specific edge you need.

What Fine-Tuning Does

Fine-tuning takes a pre-trained model and trains it further on a smaller, task-specific dataset. You are not teaching the model from scratch. You are adjusting its existing weights to perform better on your specific inputs and outputs.

Think of it like a doctor who went to medical school (pre-training) and then did a residency in cardiology (fine-tuning). The residency didn't teach them how to read. It sharpened their ability to notice the specific patterns that matter in one domain.

Three Situations Where Fine-Tuning Wins

1. You have proprietary vocabulary

Every industry has words that mean something specific to insiders. A model trained on the internet learns the common meaning. Your contracts, medical records, or engineering specs use those words differently. Fine-tuning teaches the model your vocabulary.

2. You need consistent output format

Suppose every customer support response must follow a five-part structure: acknowledgment, apology (if warranted), root cause, fix, and next steps. You can prompt for this. But prompts get forgotten in long conversations, and different agents phrase the prompt differently. A fine-tuned model internalized the format. You stop fighting it in every prompt.

3. You need lower latency or cost per token

A general model needs a long system prompt to set context. A fine-tuned model already has that context baked in. Shorter prompts mean fewer tokens processed per request. Research by Chen, Zaharia, and Zou (arXiv:2310.11409, the FrugalGPT paper) shows that routing to smaller, specialized models can cut inference costs significantly while maintaining quality on in-domain tasks.

Three Situations Where Fine-Tuning Loses

1. You have fewer than 50 good examples

Fine-tuning needs examples to learn from. If you have fewer than 50 high-quality input-output pairs, the model will overfit. It will memorize your examples rather than learning the underlying pattern. In this case, a good few-shot prompt outperforms fine-tuning every time.

3. The task requires up-to-date information

Fine-tuning bakes knowledge into weights at training time. If your task requires current information (stock prices, recent news, live inventory), fine-tuning cannot help. Use retrieval-augmented generation (RAG) instead. The model fetches fresh data at query time.

3. You need to change behavior frequently

Fine-tuning takes time and compute. If the correct behavior changes every week (new pricing, updated policies), you would need to re-fine-tune constantly. Prompt engineering with a well-maintained system prompt is far more agile in this scenario.

The Spectrum: Prompting, RAG, Fine-Tuning, Pre-Training

It helps to see these four approaches on a spectrum from least to most customization cost:

1. Prompt engineering: Zero training cost. Works well for generic tasks and tasks with few examples.
2. RAG (retrieval-augmented generation): No weight changes. Adds a retrieval step that fetches relevant documents at inference time. Best for factual, up-to-date, document-heavy use cases.
3. Fine-tuning: Moderate training cost. Changes model weights on task-specific data. Best for format, tone, terminology, and specialized reasoning.
4. Pre-training from scratch: Enormous cost. Only for organizations building foundation models for specific modalities (e.g., protein sequences, satellite imagery, specialized code).

Most organizations choosing between RAG and fine-tuning should ask: "Is my problem about access to information, or about how the model responds to information?" If information access is the issue, use RAG. If response quality, format, or reasoning style is the issue, use fine-tuning.

# Simple check: is fine-tuning worth it for your use case?

def should_fine_tune(
    num_examples: int,
    domain_specificity: float,   # 0.0 to 1.0
    change_frequency: str        # "daily", "weekly", "rarely"
) -> dict:
    if num_examples < 50:
        return {
            "recommendation": "prompt_engineering",
            "reason": "Too few examples. Fine-tuning would overfit."
        }
    if change_frequency == "daily":
        return {
            "recommendation": "rag_or_prompting",
            "reason": "Behavior changes too fast. Fine-tuning can't keep up."
        }
    if domain_specificity > 0.7 and num_examples > 200:
        return {
            "recommendation": "fine_tuning",
            "reason": "High domain specificity + sufficient data = strong fine-tuning case."
        }
    return {
        "recommendation": "try_prompting_first",
        "reason": "Marginal case. Start with prompts, measure, then decide."
    }

# Example usage
result = should_fine_tune(
    num_examples=500,
    domain_specificity=0.85,
    change_frequency="rarely"
)
print(result)
# {"recommendation": "fine_tuning", "reason": "High domain specificity..."}