Module 6: Deploying RAG · RAG

Your RAG prototype works flawlessly in a Jupyter notebook. Latency feels fine, retrieval looks accurate, the LLM responses are grounded. Then 10,000 users hit it simultaneously, and everything breaks. Notebook RAG and deployed RAG are two different engineering problems. This module covers what changes: latency budgets, caching layers, metadata filtering, observability, and the advanced retrieval patterns that make a deployed system reliable and affordable at scale.

The Latency Budget

A RAG request moves through three sequential stages, each contributing to the total response latency the user experiences. Understanding each stage separately is the first step toward optimizing the overall system.

Embedding stage (target: 10ms): The user query is encoded into a dense vector. With a model like Sentence-BERT running on GPU, this typically takes 5-15ms for a single query. Batching requests can improve throughput but adds queuing latency for individual users.

Retrieval stage (target: 20ms): The query vector is compared against the index to find top-k chunks. With FAISS on a million-document index running locally, ANN search typically takes 5-30ms depending on index type and the value of k. Managed vector databases like Pinecone add network round-trip time but handle horizontal scaling automatically.

Generation stage (target: 200-500ms): The assembled context and query are sent to the language model for response generation. This dominates the latency budget. Token generation speed depends on model size, hardware, and context length. For conversational RAG, users typically tolerate up to 3-5 seconds end-to-end; for batch workflows, throughput matters more than latency.

Where to invest optimization effort Generation is 80-90% of your latency. Shaving 5ms off retrieval while generation takes 800ms is rounding-error improvement. Optimize in order: reduce context length sent to the generator, cache repeated queries, then address retrieval speed if it exceeds 50ms.

Caching Strategies

Caching is the highest-leverage optimization available for a deployed RAG system. Real-world usage shows significant query repetition: users in the same organization tend to ask similar questions, and many queries repeat verbatim across sessions.

Layer 1

Query Cache

Exact-match cache on the raw query string. Redis or Memcached with TTL. If the same query appears again, return the cached answer immediately. Cache hit rate of 15-30% is typical for organizational knowledge bases.

Layer 2

Embedding Cache

Store computed query embeddings by query hash. Eliminates the embedding stage for repeated queries even when the answer needs refreshing. Particularly useful when the knowledge base updates frequently but queries repeat.

Layer 3

Semantic Cache

Compare the new query embedding against cached query embeddings. If cosine similarity exceeds a threshold (typically 0.95+), return the cached answer. Catches paraphrases: "What is the refund policy?" and "How do I get a refund?" resolve to the same cached response.

Semantic caching is the most powerful layer but also the most complex to tune. A threshold too high misses paraphrases; a threshold too low conflates distinct questions. Start with exact-match caching for immediate gains, then layer in semantic caching once you have query logs to validate threshold choices.

Metadata Filtering

Vector similarity search compares every indexed chunk against the query vector by default. For a 5-million-chunk index, this is computationally expensive. Metadata filtering limits the search to a relevant subset before similarity comparison.

Consider a legal document system with chunks tagged with document_type, jurisdiction, and year. A query about California employment law in 2024 should search only chunks where jurisdiction="CA" and document_type="employment". This pre-filter might reduce the search space from 5 million chunks to 40,000, dramatically reducing retrieval cost and improving precision.

Most managed vector databases support metadata filtering natively. FAISS requires maintaining a separate metadata store and filtering chunk IDs before search. Weaviate and Pinecone integrate filtering directly into the vector search query, pushing the filter to the index level for maximum efficiency.

Try this

Audit the metadata currently attached to your chunks. Common high-value metadata fields: source_url, created_date, document_type, department, language. Pick one field that would meaningfully reduce your search scope for 70% of real user queries, and add a filter on that field to your retrieval call.

Observability

A RAG system without observability is a black box. You cannot improve what you cannot measure, and problems invisible in development often surface at scale.

Log every retrieved chunk alongside the query and generated answer. This creates a dataset for offline evaluation: you can run RAGAS (Module 5) over production logs weekly to detect faithfulness drift before users notice it.

Track retrieval latency per stage separately: embedding time, vector search time, generation time, and total response time. Sudden increases in any single stage point directly to the bottleneck.

Monitor faithfulness as a moving average. Compute faithfulness scores on a random 5% sample of production queries. A drop in faithfulness from 0.88 to 0.72 over two weeks indicates either document drift (the knowledge base is stale) or prompt drift (a prompt change caused the model to hallucinate more).

Flag no-retrieval answers. When the retriever returns zero relevant chunks (all similarity scores below threshold), log the query. Clusters of these queries indicate coverage gaps in your knowledge base that need new content, not engineering fixes.

What metrics should go on a RAG operations dashboard?

A minimal RAG operations dashboard should show: (1) p50/p95/p99 end-to-end latency broken down by stage, (2) cache hit rate by cache layer, (3) 7-day rolling faithfulness score sampled from production, (4) zero-retrieval query rate as a percentage of total queries, (5) knowledge base coverage: queries where max chunk similarity is below 0.6 suggest the corpus needs expansion. These five metrics catch the majority of deployment problems.

Advanced Retrieval Patterns

Standard dense retrieval works well for direct questions but struggles with abstract or hypothesis-driven queries. Three advanced patterns address specific failure modes.

HyDE (Hypothetical Document Embeddings): Instead of embedding the query directly, ask the LLM to generate a hypothetical document that would answer the query. Embed that document and use it as the search vector. HyDE shifts the embedding from query-space into document-space, which can dramatically improve recall for complex or abstract questions. The cost is one extra LLM call per query before retrieval.

FLARE (Forward-Looking Active Retrieval): FLARE generates the response token by token and triggers a new retrieval whenever the model is uncertain about an upcoming claim (measured by token probability). It is a dynamic retrieval pattern rather than a one-shot retrieve-then-generate approach. FLARE is particularly effective for multi-hop questions requiring information from multiple documents.

Self-RAG: A fine-tuned model that generates special tokens during generation to decide whether to retrieve, evaluate the quality of retrieved chunks, and assess whether its own output is grounded. Self-RAG internalizes the retrieval decision rather than relying on the pipeline structure. It requires a specialized fine-tuned model but reduces unnecessary retrieval calls for queries the model can answer from parametric knowledge.

Fig 6 · RAG Deployment Architecture

FastAPI RAG server with query cache and metadata filtering

from fastapi import FastAPI
from pydantic import BaseModel
import hashlib, json
import redis
import faiss
import numpy as np
from sentence_transformers import SentenceTransformer

app = FastAPI()
embedder = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
cache = redis.Redis(host="localhost", port=6379, db=0)
CACHE_TTL = 3600  # 1 hour

# Assume index and chunk metadata loaded at startup
# index: faiss.Index
# chunks: list[dict] with keys "text", "metadata", "id"

class QueryRequest(BaseModel):
    query: str
    filters: dict = {}  # e.g. {"department": "legal", "year": 2024}

def embed(text: str) -> np.ndarray:
    vec = embedder.encode([text], normalize_embeddings=True)
    return vec.astype("float32")

def cache_key(query: str, filters: dict) -> str:
    payload = json.dumps({"q": query, "f": filters}, sort_keys=True)
    return hashlib.sha256(payload.encode()).hexdigest()

def metadata_filter(chunks, filters):
    if not filters:
        return list(range(len(chunks)))
    ids = []
    for i, chunk in enumerate(chunks):
        if all(chunk["metadata"].get(k) == v for k, v in filters.items()):
            ids.append(i)
    return ids

@app.post("/query")
async def query_rag(req: QueryRequest):
    # Layer 1: exact query cache
    ck = cache_key(req.query, req.filters)
    cached = cache.get(ck)
    if cached:
        return {"answer": cached.decode(), "cache": "hit"}

    # Embed query
    q_vec = embed(req.query)

    # Metadata pre-filter
    candidate_ids = metadata_filter(chunks, req.filters)
    if not candidate_ids:
        return {"answer": "No matching documents found.", "cache": "miss"}

    # Build sub-index from filtered candidates
    sub_vecs = np.stack([chunk_vectors[i] for i in candidate_ids])
    sub_index = faiss.IndexFlatIP(sub_vecs.shape[1])
    sub_index.add(sub_vecs)

    _, sub_ranks = sub_index.search(q_vec, k=min(5, len(candidate_ids)))
    top_chunks = [chunks[candidate_ids[r]] for r in sub_ranks[0] if r >= 0]

    # Assemble context
    context = "\n\n".join(c["text"] for c in top_chunks)
    answer = call_llm(req.query, context)  # your LLM call here

    # Cache result
    cache.setex(ck, CACHE_TTL, answer)

    return {"answer": answer, "cache": "miss", "chunks_used": len(top_chunks)}

Try this

Add a timing decorator to each stage of the request in the code above: embed time, filter time, search time, LLM time. Log all four values to your monitoring system for every request. After 100 requests, compute p95 latency for each stage. You will almost certainly discover that generation dominates, but the exact split guides where to invest optimization effort next.