Deploying Your Fine-Tuned Model

You've trained the model. It passes evaluation. Now it needs to serve real users in under 500 milliseconds. This module takes you from a folder of model weights to a live API endpoint — and covers the quantization step that makes it fast enough to deploy affordably.

By the end of this module, you will be able to merge LoRA adapter weights into the base model, apply GPTQ or GGUF quantization for inference speed, and serve the model via a FastAPI endpoint or Hugging Face Inference Endpoint.

Step 1: Merge the LoRA Adapter

After LoRA fine-tuning, you have two separate sets of weights: the frozen base model and the trained adapter (matrices A and B). Before serving, merge them. The merged model has the same architecture as the original — no adapter overhead at inference time.

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

# Load the base model
base_model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-2-7b-hf",
    torch_dtype="float16",
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf")

# Load LoRA adapter on top of base model
model = PeftModel.from_pretrained(base_model, "./lora-adapter-output")

# Merge: fold adapter weights into the base model
merged_model = model.merge_and_unload()

# Save the merged model — same architecture as base, adapter incorporated
merged_model.save_pretrained("./merged-model")
tokenizer.save_pretrained("./merged-model")

print("Merged model saved. No adapter needed at inference time.")

Step 2: Quantize for Inference Speed

The merged model is in 16-bit float precision. For deployment, you often want to reduce this further. Two formats dominate:

GPTQ (GPU-optimized)

GPTQ (Frantar et al., arXiv:2210.17323) quantizes the model to 4-bit integers using a calibration dataset. The result is a model that runs about 3-4x faster on GPU at roughly the same quality as the 16-bit version. Use GPTQ when you're serving from a GPU server.

GGUF / llama.cpp (CPU and consumer hardware)

GGUF is a file format used by llama.cpp that supports 4-bit and 8-bit quantization optimized for CPU inference. If your deployment target is a laptop, an edge device, or a server without a GPU, GGUF lets you run a 7B model in about 4 GB of RAM at roughly 10-20 tokens per second on a modern CPU.

Step 3: Serve via FastAPI

A FastAPI server wraps your model in an HTTP endpoint that accepts a prompt and returns a generated response. This is the simplest way to expose a fine-tuned model to an application.

from fastapi import FastAPI
from pydantic import BaseModel
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline
import torch

app = FastAPI()

# Load at startup — not per request
tokenizer = AutoTokenizer.from_pretrained("./merged-model")
model = AutoModelForCausalLM.from_pretrained(
    "./merged-model",
    torch_dtype=torch.float16,
    device_map="auto"
)
generator = pipeline("text-generation", model=model, tokenizer=tokenizer)

class Request(BaseModel):
    prompt: str
    max_new_tokens: int = 256
    temperature: float = 0.1

class Response(BaseModel):
    generated_text: str
    tokens_used: int

@app.post("/generate", response_model=Response)
async def generate(req: Request):
    outputs = generator(
        req.prompt,
        max_new_tokens=req.max_new_tokens,
        temperature=req.temperature,
        do_sample=req.temperature > 0,
        return_full_text=False
    )
    text = outputs[0]["generated_text"]
    return Response(
        generated_text=text,
        tokens_used=len(tokenizer.encode(req.prompt + text))
    )

# Run: uvicorn serve:app --host 0.0.0.0 --port 8000

Hugging Face Inference Endpoints: The Managed Option

If you don't want to manage infrastructure, Hugging Face Inference Endpoints lets you deploy any model from the Hugging Face Hub with a few clicks. You push your merged model to the Hub, create an endpoint, and get a URL you can call like any API. Pricing is per-second of compute used, so you only pay when your endpoint handles requests.

For a pilot deployment, this is often the right choice: no DevOps, no GPU management, and you can shut it down when not in use.