How RAG Works: Embeddings, Retrieval, and Generation

RAG sounds conceptually simple: retrieve relevant documents, inject them into the prompt, generate an answer. The implementation requires four technical decisions that determine whether your pipeline works well or fails silently. This article walks through each step with enough detail to understand what can go wrong and how to fix it.

Step 1: Preparing the Knowledge Base

Before retrieval can happen, every document in your knowledge base must be indexed. This preparation phase has three stages: chunking, embedding, and indexing.

Chunking

Most documents are too long to embed as a whole. A 50-page policy document has many distinct topics. Embedding the entire document produces a single vector that averages over all topics and retrieves poorly for specific questions. Chunking splits documents into smaller pieces, each of which can be embedded and retrieved independently.

Chunk size is a critical parameter. Small chunks (128-256 tokens) are highly specific but may lack context: a chunk that says "This applies to all employees hired after January 1st" is useless without the surrounding clause it refers to. Large chunks (1024+ tokens) preserve context but produce noisier embeddings and consume more of the model's context window. Most production RAG systems use chunks of 256-512 tokens with a 10-20% overlap between adjacent chunks to preserve cross-boundary context.

Embedding

Each chunk is encoded by an embedding model into a dense vector, typically 384 to 1536 dimensions depending on the model. The vector captures the semantic meaning of the chunk: similar meanings produce similar vectors regardless of exact wording. Sentence-BERT models (Reimers and Gurevych, arXiv:1908.10084) are a standard choice for English text; multilingual models are available for cross-language retrieval.

The embedding model you choose for indexing must be the same model you use to embed queries at inference time. A query vector from one model cannot be meaningfully compared to document vectors from a different model.

Indexing

The chunk vectors are loaded into a vector index that enables fast approximate nearest-neighbor search. FAISS (Johnson et al., arXiv:1702.08734) is the most widely used library for this, offering multiple index types with different speed-accuracy tradeoffs. HNSW (Hierarchical Navigable Small World) graphs offer strong recall with low latency for datasets under 10 million vectors. IVF (Inverted File Index) scales better for very large corpora but requires a training step.

Step 2: Query-Time Retrieval

When a user submits a query, the pipeline embeds the query using the same model that was used to index the knowledge base, then searches the vector index for the k most similar chunks. The similarity measure is typically cosine similarity for normalized vectors, which is mathematically equivalent to dot product after L2 normalization.

Cosine similarity between two vectors measures the angle between them. Vectors pointing in similar directions, regardless of their magnitude, have high cosine similarity. This makes it robust to documents of different lengths: a short chunk about a topic and a long article about the same topic produce vectors in a similar direction.

"Cosine similarity asks: are these two pieces of text pointing in the same conceptual direction? Not: do they share words?"

Dense retrieval has a known weakness: it performs poorly when the query contains very specific terms like product codes, proper nouns, or technical identifiers that the embedding model was not exposed to during training. For these cases, sparse retrieval methods like BM25, which match exact terms, perform better. Hybrid retrieval combines both approaches using Reciprocal Rank Fusion to merge the ranked results, and consistently outperforms either method alone. See Part 4 of this series for implementation details.

Step 3: Context Assembly

The top-k retrieved chunks are assembled into the prompt. A typical assembly looks like this: a system instruction explaining the task, the retrieved context passages labeled with their source, and the user's question. The instruction typically includes: "Answer only from the provided context. If the context does not contain enough information to answer, say so."

Context assembly has two underappreciated challenges. First, chunk ordering matters: language models attend differently to content at different positions in the context window, with content near the beginning and end getting more attention than content in the middle. If retrieval returns 5 chunks, placing the most relevant chunk first or last improves generation quality. Second, token budget management is required: if the total context exceeds the model's context window, less relevant chunks must be truncated or dropped.

Step 4: Grounded Generation

The assembled prompt is sent to the language model. With good retrieval and assembly, the generator's job is to read the context and synthesize an answer. The key instruction is that the model should answer from context, not from its parametric memory. A model that supplements context with memorized knowledge may produce plausible-sounding answers that are not grounded in the retrieved documents.

Faithfulness is the metric that measures how well the model stays within the retrieved context. A faithfulness score of 1.0 means every claim in the answer is directly supported by the context. A score below 0.7 typically indicates either the retriever is returning irrelevant context (causing the model to fall back on parametric knowledge) or the system prompt is not sufficiently constraining the model to context-only answers.

Where RAG Pipelines Fail

Understanding failure modes is as important as understanding the happy path. Most RAG failures fall into one of four categories.

Retrieval misses: The correct information is in the knowledge base but the retriever does not return it. Causes include chunks that are too large (diluting the signal), an embedding model that was not trained on domain-specific vocabulary, or a query phrasing that differs significantly from the document language. The fix is usually better chunking, a more domain-appropriate embedding model, or adding hybrid retrieval.

Retrieval noise: The retriever returns the correct chunk plus several irrelevant chunks that confuse the generator. Reranking (a cross-encoder pass after initial retrieval) reduces this significantly and is the standard fix for high-precision applications.

Context assembly failures: The assembled prompt exceeds the model's context window, causing truncation of the most relevant chunks. The fix is more aggressive relevance filtering before assembly and reducing chunk count or size.

Generator faithfulness failures: The generator produces an answer that supplements or contradicts the retrieved context with parametric knowledge. The fix is a stronger system prompt instruction to answer only from context, combined with faithfulness evaluation (see Part 5 of this series) to detect this in production.

Embedding Models: What They Learn and Why It Matters

The quality of a RAG system is bounded by the quality of its embeddings. An embedding model learns to compress a passage of text into a fixed-length vector such that passages with similar meaning land near each other in the resulting space. The model achieves this through contrastive training: it sees millions of sentence pairs labeled as "related" or "unrelated" and adjusts its weights until similar pairs map to nearby vectors and dissimilar pairs map to distant ones.

The practical consequence is that the embedding space captures semantic relationships that keyword matching cannot. The query "How do I reset my account access?" will retrieve a passage containing "To regain entry to your account, click the Forgot Password link" because their vectors are geometrically close, even though they share no keywords. This is the central capability that makes RAG useful for real-world knowledge bases, where users phrase questions in their own language rather than the document's language.

Different embedding models learn different things. A general-purpose model trained on web data performs well on everyday queries but may underperform on scientific, legal, or medical documents where vocabulary is highly specialized. A domain-specific embedding model trained on PubMed abstracts, for example, will embed the word "lesion" near "tumor" and "necrosis" rather than near other common uses of the word. When your knowledge base contains specialized vocabulary, the choice of embedding model is one of the highest-leverage decisions in the pipeline.

Sentence-BERT (Reimers and Gurevych, arXiv:1908.10084) established the modern approach to sentence embedding by extending BERT with a Siamese network architecture specifically designed for semantic similarity tasks. The resulting embeddings are fast to compute (the 384-dimensional MiniLM variant processes thousands of sentences per second on a single GPU) and accurate enough for most retrieval applications. Larger models like text-embedding-3-large produce 3,072-dimensional vectors with higher precision at greater cost.

How Cosine Similarity Drives Retrieval Decisions

Once the knowledge base is indexed as vectors, retrieval reduces to a nearest-neighbor search: given the query vector, find the document vectors that are most similar. The most common similarity metric is cosine similarity, which measures the angle between two vectors rather than the distance between their endpoints. This matters because cosine similarity is invariant to vector magnitude: a long document and a short document covering the same topic will produce vectors pointing in similar directions, even if one vector is larger in scale than the other.

The cosine similarity between two vectors A and B is computed as their dot product divided by the product of their magnitudes. A score of 1.0 means the vectors point in the same direction (identical meaning), 0.0 means they are orthogonal (completely unrelated), and negative values indicate opposite meaning. In practice, a threshold of 0.7 or higher is commonly used to identify relevant matches, though the right threshold varies by embedding model and application.

The challenge is that computing cosine similarity between the query vector and every document vector in the knowledge base becomes expensive as the corpus grows. A knowledge base with one million chunks requires one million similarity computations per query. For small corpora this is acceptable, but at enterprise scale it introduces unacceptable latency. This is why approximate nearest neighbor (ANN) indexing exists: algorithms like HNSW (Hierarchical Navigable Small World) and IVF (Inverted File Index) build data structures that allow the search to skip most of the corpus and find the nearest neighbors in milliseconds rather than seconds, with only a small accuracy tradeoff.

Sparse Retrieval and the Case for Hybrid Search

Dense embedding-based retrieval is not the only option. Sparse retrieval methods like BM25 (a descendant of TF-IDF) work by indexing exact terms and scoring documents based on term frequency and inverse document frequency. BM25 excels at finding documents that contain specific, rare terms: product codes, proper nouns, chemical names, regulatory identifiers. It does not require any embedding computation.

Neither approach dominates the other. Dense retrieval handles paraphrase and synonym matching well but can miss exact-match queries where a rare term is the key signal. Sparse retrieval handles exact-term matching well but fails on semantic paraphrase queries. The standard solution in high-performance RAG systems is hybrid retrieval: run both BM25 and dense vector search in parallel, then combine the results using Reciprocal Rank Fusion (RRF).

RRF assigns a score to each document based on its rank in each result list using the formula 1/(k+rank), where k is typically 60. Documents that rank highly in both lists receive the highest combined scores. This fusion approach consistently outperforms either method alone and is now the default in most enterprise RAG implementations. The Dense Passage Retrieval paper (Karpukhin et al., arXiv:2004.04906) established the foundations of dense retrieval, and subsequent work on hybrid fusion confirmed that combining sparse and dense signals is the most reliable path to high recall.

The Generator's Role: Staying Faithful to Context

After retrieval, the generator (typically a large language model) receives a prompt containing the assembled context and the user's question. Its job is narrower than unconstrained question answering: it should synthesize an answer from the provided context and nothing else. This is harder than it sounds, because large language models have absorbed enormous amounts of parametric knowledge during pretraining and will draw on that knowledge if the system prompt allows it.

The critical design decision is the system prompt instruction. A weak instruction like "use the provided context to answer" leaves room for the model to supplement context with parametric knowledge. A strong instruction like "answer only using information in the provided context; if the context does not contain the answer, say so" constrains the model much more effectively. Testing faithfulness explicitly using the RAGAS framework (Es et al., arXiv:2309.15217) is the only reliable way to confirm the generator is staying within context bounds.

Temperature also matters here. A temperature of 0 makes the generator deterministic and more likely to extract rather than generate, which is appropriate for factual retrieval applications. A higher temperature introduces more creative synthesis, which is appropriate for applications where the generator needs to combine multiple retrieved passages into a coherent explanation rather than simply extracting a single answer.

End-to-End Latency and How to Hit It

A RAG system has a latency budget composed of several stages: query embedding (typically 5-15ms for a lightweight model), vector search (10-50ms depending on index size and infrastructure), optional reranking (50-200ms for a cross-encoder over the top-k candidates), context assembly (negligible), and generation (200-800ms depending on the model and output length). The total round-trip for a well-optimized system lands around 300-600ms, which is within acceptable range for most applications.

The largest latency levers are model size and caching. Switching from a full-size embedding model to a distilled model like MiniLM-L6 cuts embedding time by roughly 80% with modest precision loss. Caching frequently issued queries (storing the query embedding and retrieved chunks) eliminates the retrieval cost entirely for repeated questions, which is highly effective for knowledge bases where users ask a concentrated set of common questions. Many enterprise RAG deployments find that 30-50% of queries are repetitions of previously answered questions, making a query cache one of the highest-ROI optimizations available.