Module 1: What Is RAG? · Retrieval-Augmented Generation

Imagine asking an AI assistant about a policy that changed last month. Its training data ended before that change happened, so it either refuses to answer or confidently gives you the old answer. RAG fixes this without retraining the model. By the end of this module you will understand exactly what RAG does, how its three components fit together, and when it beats fine-tuning.

The Problem RAG Solves

A language model learns from a snapshot of text. That snapshot has a cutoff date. After training ends, the model's knowledge is frozen. Ask it about something that happened after the cutoff and it either says "I don't know" or, worse, fills in a plausible-sounding wrong answer from patterns it already learned.

The obvious fix is to retrain the model every time the knowledge base changes. But training large models costs hundreds of thousands of dollars and takes weeks. For most organizations, that is not a realistic cycle.

RAG takes a different approach. Instead of baking knowledge into the model's weights, it retrieves relevant documents at the time of each query and injects them into the model's context window. The model then generates an answer grounded in those documents. The knowledge lives in your document store, not in the model's parameters. When knowledge changes, you update the documents, not the model.

The one-sentence definition RAG is an architecture that retrieves relevant documents at inference time, injects them into the prompt, and asks the model to generate an answer grounded in those documents rather than in its training memory.

The Three-Component Architecture

Every RAG system has three components working in sequence. Understanding each one is the foundation for everything else in this course.

Component 1

Retriever

Takes the user query, finds the most semantically relevant chunks from a document index, and returns a ranked list of candidate passages.

Component 2

Context Assembler

Takes the retrieved chunks, formats them, truncates to fit the context window, and constructs the final prompt that will be sent to the language model.

Component 3

Generator

Receives the assembled prompt (query + context) and produces a grounded answer. The model is instructed to use only what is in the provided context.

These three components are arranged in a pipeline. A query enters at the retriever, passes through context assembly, and exits as a generated response with the retrieved documents as its factual grounding.

Fig 1 · The RAG Pipeline

When RAG Beats Fine-Tuning

Both RAG and fine-tuning help a language model work with specific knowledge. They solve different problems. Choosing the wrong one wastes time and money.

Use RAG when: your knowledge changes frequently (daily or weekly), your knowledge base is large (thousands of documents), you need citations so users can verify answers, or you have multiple knowledge bases that swap depending on the query context.

Use fine-tuning when: you need to change how the model responds, not what it knows, your knowledge is stable and will not change often, or you need the model to adopt a specialized style, format, or reasoning pattern.

The original RAG paper by Lewis et al. (arXiv:2005.05611) showed that retrieval-augmented models significantly outperformed parametric-only models on knowledge-intensive tasks, particularly on questions requiring current or specific factual information. The key finding was that combining retrieval with generation produces more accurate and more verifiable answers than storing all knowledge in model weights.

Common misconception "RAG is just prompt stuffing." This misses the point. Prompt stuffing would mean manually finding the right documents and pasting them in. RAG automates retrieval at inference time using semantic similarity search over potentially millions of documents. The magic is in the retriever knowing which documents are relevant to which queries without human curation.

A Concrete Example: Policy Q&A

A company stores its 800-page employee handbook in a vector database, chunked into paragraphs. An employee asks: "What is our policy on remote work for employees on a performance improvement plan?"

The RAG system embeds the query, searches the vector database for the 5 most semantically similar paragraphs, assembles them into a prompt that reads "Use only the following policy excerpts to answer...", sends it to the LLM, and returns an answer that cites the specific policy section. If the HR team updates the handbook next month, they re-index the changed document and the system immediately reflects the new policy, with no model retraining.

Without RAG, the LLM would either refuse to answer (it has no handbook) or hallucinate a plausible policy that might contradict the actual one. The cost of that error in an HR context is obvious.

Why does RAG produce more verifiable answers than a fine-tuned model? ▼

When a RAG system answers a question, the retrieved chunks are visible in the prompt. The answer can be traced back to specific source documents. A fine-tuned model produces answers from its weights, which are opaque: you cannot easily inspect which training examples contributed to a particular answer. RAG makes the evidence chain explicit, which is why regulated industries (legal, medical, financial) favor it for compliance-sensitive use cases.

The Knowledge Base Size and Freshness Matrix

The choice between RAG, fine-tuning, or both depends heavily on two dimensions: how large your knowledge base is and how often it changes. Use the interactive below to explore how different combinations of size and freshness push the recommendation.

Python · Basic RAG Loop

# Basic RAG loop: embed query, search, assemble context, generate
from sentence_transformers import SentenceTransformer
import faiss, numpy as np

model = SentenceTransformer('all-MiniLM-L6-v2')

# --- INDEXING (done once, then persisted) ---
def build_index(chunks: list[str]):
    embeddings = model.encode(chunks, convert_to_numpy=True)
    index = faiss.IndexFlatIP(embeddings.shape[1])
    # Normalize for cosine similarity
    faiss.normalize_L2(embeddings)
    index.add(embeddings)
    return index, chunks

# --- RETRIEVAL ---
def retrieve(query: str, index, chunks, k=5):
    q_vec = model.encode([query], convert_to_numpy=True)
    faiss.normalize_L2(q_vec)
    scores, indices = index.search(q_vec, k)
    return [chunks[i] for i in indices[0]]

# --- CONTEXT ASSEMBLY ---
def assemble_prompt(query: str, retrieved: list[str]) -> str:
    context = "\n\n".join(f"[{i+1}] {c}" for i, c in enumerate(retrieved))
    return (
        f"Use only the context below to answer the question.\n\n"
        f"Context:\n{context}\n\nQuestion: {query}\nAnswer:"
    )

# --- GENERATION ---
def rag_query(query, index, chunks, llm_client):
    retrieved = retrieve(query, index, chunks)
    prompt = assemble_prompt(query, retrieved)
    return llm_client.complete(prompt)  # your LLM call here

Try this

Pick a document you use frequently at work: a policy manual, a product spec, or a FAQ. Write down three questions a new colleague might ask about it. Then trace each question through the three RAG components: what would the retriever search for, what chunks would the context assembler select, and what would you want the generator to say? This exercise reveals which component is doing the most work for your use case.