Dataset Preparation and Formatting

A team at a major bank spent three months fine-tuning a model on their compliance documents. The model got worse. The problem wasn't the training process — it was that 40% of their examples contradicted each other. Garbage in, garbage out. This module teaches you to prepare data the right way.

By the end of this module, you will be able to format a dataset for instruction fine-tuning, apply the five most important data quality filters, and avoid the three most common dataset mistakes that cause fine-tuning to fail.

The Three Formats You Need to Know

Fine-tuning datasets take different shapes depending on your task. The most common format today is the instruction-response pair, sometimes called a "prompt-completion" pair.

Format 1: Instruction-Response (most common)

Each example has two fields: an instruction (what you ask the model to do) and a response (the ideal output). This format teaches the model to follow instructions in your domain.

{
  "instruction": "Classify this customer complaint as billing, technical, or general.",
  "input": "My invoice shows a charge I don't recognize from last Tuesday.",
  "output": "billing"
}

Format 2: Conversational (for chat models)

Uses a list of turns with roles: system, user, and assistant. This format trains the model to maintain context across a conversation.

{
  "messages": [
    {"role": "system", "content": "You are a helpful assistant for TechCorp."},
    {"role": "user", "content": "How do I reset my password?"},
    {"role": "assistant", "content": "Go to Settings > Security > Reset Password. You'll receive an email within 2 minutes."}
  ]
}

Format 3: Completion (for older-style models)

A single text string where the model continues from a prompt. Less common for modern instruction-following fine-tuning, but still used for text generation tasks where you want to extend content in a particular style.

The Five Quality Filters

Before you format your data, you need to clean it. These five filters catch 90% of the problems that cause fine-tuning to fail.

1. Deduplication

Remove exact duplicates and near-duplicates. A model trained on 1,000 examples where 200 are copies of the same 10 examples will behave as if it has only 800 unique examples — but with those 10 over-represented. Use MinHash or exact-match hashing for deduplication.

2. Length filtering

Responses shorter than 10 tokens teach the model nothing useful. Responses longer than your model's context window will be truncated, which teaches the model to start answers it cannot finish. Set minimum and maximum token lengths appropriate for your task.

3. Consistency checking

If two examples have nearly identical inputs but contradictory outputs, the model receives contradictory training signal. It will average between them, satisfying neither. This is the hardest filter to automate. Embedding-based clustering helps: group similar inputs and check whether their outputs agree.

4. Coverage analysis

Make sure your dataset covers all the sub-tasks you need the model to handle. A customer support dataset that has 1,000 billing examples and 5 technical examples will produce a model that is great at billing and helpless at technical. Use stratified sampling to balance coverage.

5. Toxicity and PII removal

Any toxic content or personally identifiable information in training data ends up in model outputs. Run a classifier for toxicity (e.g., Perspective API or a local model) and use regular expressions for common PII patterns (emails, phone numbers, social security numbers).

The JSONL Format: What the Training Script Actually Reads

Most fine-tuning frameworks (Hugging Face Transformers, Axolotl, OpenAI fine-tuning API) expect data in JSONL format: one JSON object per line, each object being one training example.

# Step 1: Load and validate your raw examples
import json
import hashlib
from pathlib import Path

def load_and_clean_dataset(raw_path: str, min_resp_tokens: int = 10, max_resp_tokens: int = 512):
    examples = []
    seen_hashes = set()

    with open(raw_path) as f:
        for line in f:
            ex = json.loads(line.strip())

            # Filter: require instruction and output fields
            if not ex.get('instruction') or not ex.get('output'):
                continue

            # Filter: length
            output_len = len(ex['output'].split())
            if output_len < min_resp_tokens or output_len > max_resp_tokens:
                continue

            # Dedup: hash the instruction+output together
            key = hashlib.md5((ex['instruction'] + ex['output']).encode()).hexdigest()
            if key in seen_hashes:
                continue
            seen_hashes.add(key)

            examples.append(ex)

    return examples

# Step 2: Format for instruction fine-tuning (Alpaca-style)
def format_example(ex: dict) -> str:
    prompt = f"### Instruction:\n{ex['instruction']}\n"
    if ex.get('input'):
        prompt += f"### Input:\n{ex['input']}\n"
    prompt += f"### Response:\n{ex['output']}"
    return prompt

# Step 3: Write to JSONL
def write_jsonl(examples, output_path):
    with open(output_path, 'w') as f:
        for ex in examples:
            f.write(json.dumps({"text": format_example(ex)}) + "\n")

# Usage
raw = load_and_clean_dataset("data/raw.jsonl")
write_jsonl(raw, "data/train.jsonl")
print(f"Wrote {len(raw)} clean examples.")

How Much Data Do You Need?

This is the most common question and the honest answer is: it depends on task complexity. Research by Wei et al. (arXiv:2109.01652) on instruction tuning showed that relatively small, high-quality datasets can produce strong instruction-following behavior in models that were already large and well-pre-trained.

Practical ranges used by practitioners:

• Simple classification tasks: 100–500 examples often sufficient
• Extraction with a fixed schema: 200–1,000 examples
• Open-ended generation in a specific style: 1,000–5,000 examples
• Complex multi-step reasoning: 5,000–50,000 examples