What Is RAG? Retrieval-Augmented Generation Explained

Every large language model has a knowledge cutoff. Ask it about something that happened after training and it either refuses or invents an answer. Retrieval-Augmented Generation, RAG, solves this by connecting the model to a live knowledge base at inference time. Here is exactly what that means, how the architecture works, and why it has become the most widely deployed pattern in applied AI.

The Core Problem RAG Solves

Language models are trained on static snapshots of text. A model trained through late 2023 cannot tell you about regulatory changes published in 2024. It cannot access your internal company documentation, your product database, your customer records, or anything that was not in its training corpus. It does not know what it does not know: it will often generate a plausible-sounding answer to a question it cannot actually answer, because generating plausible text is exactly what it was trained to do.

The traditional solution to knowledge gaps was fine-tuning: retrain the model on your domain data to bake the knowledge in. This works for stable knowledge but fails for anything that changes. You cannot fine-tune a model daily to keep up with document updates, regulatory filings, or product inventory changes. The retraining cost, in time and compute, makes that approach impractical for dynamic knowledge.

RAG takes a different approach: instead of baking knowledge into model weights, retrieve the relevant information at query time and inject it into the model's context window. The model sees the retrieved text as part of its input and uses it to generate a grounded answer. The knowledge lives outside the model, in a searchable store, and can be updated without touching the model at all.

"The model's weights are the reasoning engine. The knowledge base is the filing cabinet. RAG connects the two at the moment a question is asked."

The Three-Component Architecture

Every RAG system has three core components working in sequence. Understanding each one is essential for building, debugging, and improving a RAG pipeline.

The Retriever

The retriever finds the documents or passages most relevant to the user's query. Modern retrievers typically work with dense vector representations: both the query and every document chunk in the knowledge base are encoded as high-dimensional vectors, and retrieval finds the chunks whose vectors are most similar to the query vector. This is semantic search: a query about "employee termination process" will find documents about "offboarding procedures" even if those exact words do not appear in the query.

The retriever is the most critical component of the pipeline. A generator that receives the right context can almost always produce a good answer. A generator that receives the wrong context, or no context at all, will fail regardless of how capable it is. Retrieval quality is the primary determinant of RAG system quality.

The Context Assembler

The context assembler takes the retrieved chunks and packages them into a structured prompt for the language model. This typically includes: the user's original question, the retrieved context passages (usually 3-10 chunks), and a system instruction telling the model to answer only from the provided context. The assembler also handles token budget management: if retrieved chunks exceed the model's context window, they must be truncated or ranked by relevance.

The Generator

The generator is the language model that synthesizes an answer from the assembled context. With a well-designed retriever and assembler, the generator's job is straightforward: read the provided context, identify the parts that answer the question, and produce a coherent response. The generator should not need to recall anything from training for the answer, because the answer should be present in the context. This is what makes RAG grounded: the model is not guessing from parametric memory, it is reading from a document.

A Concrete Example

Consider a company with 5,000 internal policy documents, updated monthly. An employee asks: "What is the approval process for expenses over $10,000?"

Without RAG, the language model has no idea what this company's policy is. It might produce a generic description of an enterprise approval process that sounds plausible but has nothing to do with the actual policy.

With RAG: the query is embedded as a dense vector, the retriever finds the three chunks in the policy knowledge base most semantically similar to the question, those chunks are assembled into the prompt alongside the question, and the model reads the actual policy text and produces an answer grounded in it. The answer is correct, current, and traceable to the source document.

When the expense policy changes next month, the knowledge base is updated with the new document chunks. The model itself is untouched. Every new query reflects the current policy immediately.

What RAG Is Not

RAG is not fine-tuning. Fine-tuning updates the model's weights to embed new knowledge. RAG leaves the model's weights unchanged and supplies knowledge externally at inference time. The two approaches can be combined, but they solve different problems.

RAG is not just semantic search. Semantic search finds relevant documents. RAG uses those documents as context for a language model that synthesizes a direct answer. Semantic search returns ranked results; RAG returns a generated response grounded in those results.

RAG is not a solution to hallucination in general. A language model can still hallucinate even when given correct context, particularly if the context is ambiguous, the model's instruction following is weak, or the model decides to supplement context with parametric knowledge. RAG reduces hallucination by providing ground-truth context, but evaluation is still required to measure how faithfully the model uses it.

When to Use RAG

RAG is the right choice when the knowledge you need is dynamic, private, or too large to fit in a model's context window. It is particularly well-suited to: internal knowledge bases and documentation systems, customer support over proprietary product catalogs, legal or compliance search over regulation libraries, research assistants over document collections, and any application where answers need to be traceable to source documents.

RAG is less appropriate when the knowledge is stable and small enough to include directly in a system prompt, when latency requirements cannot tolerate retrieval overhead, or when the knowledge is already well-represented in model training data and does not require grounding in specific documents. For these cases, see the RAG vs fine-tuning comparison in Part 3 of this series.

How Retrieval Quality Determines Everything

The most important insight in RAG engineering is that the quality of the generated answer is bounded by the quality of the retrieval. If the correct information is not in the context window, the generator cannot produce a correct answer, regardless of how capable the model is. Improving the generator when retrieval is failing is a waste of effort.

This means the first thing to optimize in any RAG system is the retriever. Standard dense retrieval using bi-encoder models like Sentence-BERT works well for most semantic queries, but has two known weaknesses. First, it struggles with queries that contain very specific identifiers: product codes, clause numbers, proper nouns, or highly technical terminology that the embedding model did not encounter during pretraining. For these queries, sparse retrieval methods like BM25, which perform exact term matching, are more reliable. Second, dense retrieval is sensitive to the quality of the embeddings: a general-purpose embedding model trained primarily on web text will perform worse on clinical or legal corpora than one trained on domain-specific text.

Hybrid retrieval addresses the first weakness by combining dense and sparse retrieval. The practical implementation uses Reciprocal Rank Fusion to merge the ranked lists from both retrievers into a single ranked list without requiring score normalization. The second weakness is harder to address without a domain-specific embedding model, but can often be partially offset by careful chunking: shorter, more focused chunks produce more semantically coherent vectors that embed more reliably even with general-purpose models.

The Role of Chunking in Retrieval Quality

Chunking is how you split the raw documents in your knowledge base into the units that will be embedded and retrieved. The chunking strategy is one of the most consequential decisions in a RAG system, and it is also one of the least intuitive.

The intuitive approach is to use very small chunks, on the theory that more precise units will retrieve more precisely. This fails in practice for two reasons. First, small chunks lose context: a chunk that says "The deadline is 30 days from the date of notice" is useless without knowing what deadline and what notice it refers to. Second, the embedding of a small, decontextualized chunk is less reliable: the model has less signal to work with, and the vector may not accurately capture the semantic content of the passage.

The intuitive approach in the other direction, using very large chunks or even full documents, also fails. A chunk that covers 5,000 words on multiple topics produces an embedding that averages over all those topics and retrieves poorly for any specific question. The generator also has to read more irrelevant text to find the relevant answer, which reduces answer quality and increases latency and cost.

The empirical answer for most corpora is chunks of 256-512 tokens with 10-20% overlap between adjacent chunks. The overlap ensures that information near chunk boundaries does not fall through the cracks. The exact numbers should be tuned for each corpus by evaluating retrieval precision and recall on a sample of test questions.

Evaluating Whether RAG Is Working

The most common failure mode for RAG pilots is trusting the fluency of the output. Language models produce fluent, confident text regardless of whether that text is grounded in the retrieved context. An answer that reads well is not necessarily correct, and an answer that cites a source is not necessarily faithful to that source.

The RAGAS framework (Es et al., arXiv:2309.15217) provides four metrics that together characterize the health of a RAG pipeline. Context precision measures whether the retrieved chunks are relevant to the question. Context recall measures whether the retrieved chunks contain the information needed to answer. Faithfulness measures whether the generated answer is supported by the retrieved context. Answer relevance measures whether the generated answer addresses the question asked.

A healthy RAG system scores above 0.75 on all four metrics. The lowest-scoring metric indicates where to focus improvement effort. Low context precision points to retrieval noise: the retriever is returning off-topic chunks. Low context recall points to retrieval coverage gaps: the relevant documents may not be indexed, or the query phrasing is too different from the document language. Low faithfulness points to generator hallucination: the model is supplementing context with parametric knowledge. Low answer relevance points to prompt design issues: the generator is drifting off-topic.

RAG in Practice: Getting Started

The fastest path to a working RAG system is to start with a small, well-curated knowledge base: 50-200 documents that you know well, covering a narrow domain. This makes it easy to evaluate whether the system is working: you can ask questions you already know the answers to and check whether the retrieved chunks contain the right information and the generated answers are accurate.

The technology stack for an initial RAG implementation is minimal: a sentence-transformers model for embedding, FAISS for the vector index, and any language model API for generation. LangChain and LlamaIndex both provide higher-level abstractions that can accelerate initial development. The important thing is to build the evaluation loop alongside the pipeline, not after: you should be measuring retrieval quality from the first day.

Once the system is working on the small corpus, expand to the full knowledge base incrementally, measuring retrieval quality at each step. Scale issues, indexing problems, and chunking failures that are invisible with 50 documents often become visible at 5,000. Building the evaluation infrastructure early makes these problems discoverable before they reach users.

The six-module RAG course on this site covers each stage of the pipeline in detail with interactive demos and working code examples. Module 3 covers chunking strategies with an interactive slider that shows how chunk size and overlap affect precision and recall. Module 4 covers hybrid retrieval with a live comparison of sparse, dense, and hybrid retrieval precision. Module 5 covers RAGAS evaluation with a score explorer that lets you see how each metric responds to different pipeline configurations.