Building a RAG Pipeline: A Step-by-Step Guide

Most RAG tutorials stop at "embed your docs and do a similarity search." Real pipelines need chunking that preserves context, hybrid retrieval that works for both keyword and semantic queries, reranking that filters noise, and evaluation that catches regressions before users do. Here is the complete build sequence.

Phase 1: Chunking Your Knowledge Base

Chunking decisions affect every downstream metric. Too-small chunks miss context. Too-large chunks dilute embeddings and consume context window. The goal is chunks that are semantically coherent: each chunk should represent a complete thought that can be retrieved and understood without the surrounding document.

Start with LangChain's RecursiveCharacterTextSplitter at 512 tokens with a 64-token overlap. This splits on paragraph boundaries when possible, falling back to sentence boundaries, then word boundaries. It is the most reliable general-purpose splitter for mixed-format documents.

For structured document types, switch to domain-aware splitters: MarkdownHeaderTextSplitter for Markdown documentation (splits on heading boundaries, preserving the header as metadata on each chunk), and SentenceTransformersTokenTextSplitter for corpora where sentence-level granularity improves retrieval. Each chunk should carry metadata: source, page, section, and created_date at minimum. Metadata enables filtering, which reduces retrieval noise significantly for multi-domain corpora.

Phase 2: Embedding and Indexing

Use sentence-transformers/all-MiniLM-L6-v2 as your starting embedding model. It produces 384-dimensional vectors, runs quickly on CPU for batched indexing, and performs strongly across general English text. For domain-specific corpora (medical, legal, financial), evaluate whether a domain-adapted model improves retrieval quality before committing to it for production.

Normalize embeddings to unit length before indexing. This allows you to use inner product search (FAISS IndexFlatIP), which is mathematically equivalent to cosine similarity for normalized vectors but runs faster. For corpora over 1 million chunks, switch to an approximate index: FAISS IndexHNSWFlat gives strong recall (typically above 98%) with sub-millisecond search latency.

Phase 3: Hybrid Retrieval with RRF

Dense retrieval misses exact-match queries for product codes, proper nouns, and rare technical terms. Sparse retrieval (BM25) misses paraphrases and semantic equivalences. Hybrid retrieval combines both and is almost always strictly better than either alone.

The implementation is straightforward. Run both retrievers against the query, get ranked lists from each, then combine using Reciprocal Rank Fusion. For each chunk, the RRF score is the sum of 1/(k+rank) from each retriever, where k=60 is the standard smoothing constant. Re-sort by RRF score descending. The top-k results from this merged list go to the next stage.

"Hybrid retrieval is not an optimization. It is a correctness fix for a class of queries that dense retrieval gets fundamentally wrong."

Phase 4: Cross-Encoder Reranking

The initial retrieval returns candidate chunks. Reranking applies a more expensive but more accurate model to re-score each candidate relative to the query. Cross-encoders (such as cross-encoder/ms-marco-MiniLM-L-6-v2) read the query and each chunk together as a pair, which allows them to model relevance much more precisely than bi-encoder embeddings.

Retrieve 20-50 candidates from hybrid retrieval, rerank with a cross-encoder, and pass only the top 3-5 chunks to the generator. This two-stage approach amortizes the cost of the expensive reranker over a small candidate set while dramatically improving precision.

Phase 5: Context Assembly and Generation

The assembled prompt has three sections. First, the system instruction: include a clear statement that the model should answer only from the provided context, and should say "I don't have enough information" when the context does not contain the answer. Second, the retrieved context: number each chunk and include its source metadata. Third, the user's question, repeated at the end for models that attend more strongly to later positions.

Track token count before sending. If total tokens exceed 80% of the model's context window, drop the lowest-scored chunks until you are within budget. Never silently truncate in the middle of a chunk: always drop complete chunks from the bottom of the ranked list.

Evaluation Before You Ship

Build a test set of 20-50 question-answer pairs before launching. For each pair, run the full pipeline and record the retrieved chunks, the generated answer, and the ground-truth answer. Run RAGAS on this set with all four metrics. Set minimum thresholds: faithfulness above 0.8, context precision above 0.7, answer relevance above 0.75. These are starting baselines, not performance targets, but they will catch obvious pipeline failures before users encounter them.

After launch, run the same RAGAS evaluation on a 5% random sample of production traffic weekly. Faithfulness drift over two weeks is the strongest early signal of either knowledge base staleness (old documents are returning for queries about new topics) or retrieval quality regression (an index update caused precision to drop).

Choosing the Right Vector Store

The vector store is the infrastructure component that persists your embeddings and handles nearest-neighbor search at query time. The right choice depends on your scale, operational constraints, and query latency requirements.

FAISS (Johnson et al., arXiv:1702.08734) is the starting point for most teams because it is open-source, runs in memory, and requires no additional infrastructure. FAISS supports both exact search (IndexFlatL2) and approximate search (IndexIVFPQ, IndexHNSWFlat). For knowledge bases under 500,000 chunks, a FAISS flat index fits comfortably in the RAM of a standard server instance and returns results in under 5ms. The limitation is that FAISS is not a managed service: you are responsible for persistence (serialize the index to disk regularly), updates (add new embeddings when the knowledge base changes), and scaling (FAISS is single-node without additional engineering).

Chroma is the most popular choice for early-stage applications because it bundles embedding, storage, and retrieval into a single Python library with minimal configuration. Its developer experience is excellent: you can build and query a persistent index in under 20 lines of code. It supports metadata filtering, which is critical for multi-tenant applications where each user should only see their own documents.

Pinecone and Weaviate are fully managed vector databases designed for enterprise scale. Pinecone handles infrastructure, replication, and index updates automatically, with a query API that returns results in single-digit milliseconds at billion-vector scale. Weaviate adds a built-in BM25 hybrid search layer and a GraphQL API for complex filtered queries. Both charge based on the number of stored vectors and query volume, with costs that are manageable at typical enterprise scale but require forecasting.

pgvector extends PostgreSQL with a native vector column type and HNSW indexing. Its key advantage is that it integrates vector search into an existing Postgres database, eliminating the need for a separate vector store and its associated operational overhead. For applications where the metadata (document title, author, date, access control) is already in Postgres, pgvector enables metadata filtering through standard SQL WHERE clauses joined with the vector search, which is more expressive than most dedicated vector stores.

Metadata Filtering: The Missing Piece of Most RAG Tutorials

Most introductory RAG implementations treat retrieval as a pure semantic search: embed the query, find the nearest k chunks, return them. This works for small, homogeneous knowledge bases but breaks down in production, where knowledge bases contain documents with different access permissions, different date ranges, different document types, and different relevance scopes.

Metadata filtering solves this by attaching structured attributes to each chunk at index time and applying SQL-style filters before or alongside vector search. A legal firm's knowledge base might filter by client matter number and document date. An enterprise knowledge base might filter by department and confidentiality level. A multi-tenant SaaS product must filter by tenant ID so each customer sees only their own data.

The implementation requires populating metadata fields when chunks are embedded: for each chunk, store not just the vector but also the document ID, document date, author, category, and any other filtering attributes relevant to your application. At query time, apply the filter as a pre-filter (restrict the search to a subset of the index before running vector search) or a post-filter (run vector search over all chunks, then filter the results). Pre-filtering is more efficient at large scale; post-filtering is simpler but can return fewer than k results when the filter is restrictive.

Document Processing: Getting Clean Text Into the Pipeline

The chunking and embedding steps assume you already have clean text. In practice, extracting clean text from real-world documents is often the most time-consuming part of building a RAG pipeline. PDFs are the dominant format in enterprise knowledge bases and the most problematic: PDF rendering libraries vary in how they handle multi-column layouts, tables, footnotes, headers, and scanned content. A poorly parsed PDF can produce chunks that interleave text from different columns, split table cells incorrectly, or include page number artifacts in the middle of sentences.

Tools like PyMuPDF (for text-layer PDFs), pdfplumber (for table extraction), and Unstructured.io (for mixed-content documents) each handle different failure modes. For document-heavy RAG pipelines, it is worth testing your parser on a representative sample of your actual documents before committing to an implementation. Common issues include footnote text interleaved with body text, headers and footers repeated in every chunk, and mathematical notation lost in conversion.

HTML and Markdown documents are generally easier: both have predictable structure and can be parsed reliably with standard libraries. For HTML, BeautifulSoup can extract main content while stripping navigation, ads, and boilerplate. For Markdown, the structure itself (headings, code blocks, lists) provides natural boundaries for document-aware chunking.

From Prototype to Pilot Deployment

The gap between a working RAG notebook and a reliable pilot deployment is primarily an engineering gap, not an AI gap. The RAG algorithm itself is straightforward; the surrounding infrastructure for document ingestion, index maintenance, query logging, and evaluation is what differentiates a research prototype from a useful system.

A minimal pilot deployment needs four components beyond the core RAG logic: an ingestion pipeline that watches for new and updated documents and re-embeds them automatically, a query logging layer that records every question and the chunks retrieved, a basic evaluation harness that runs RAGAS metrics weekly on a sample of logged queries, and an alerting threshold that flags when faithfulness drops below a configured minimum. These components do not need to be elaborate; a simple cron job, a Postgres log table, and a weekly email report satisfy the requirements for most pilots.

The most common mistake teams make after a successful prototype is deploying without the evaluation harness. A RAG system that is not monitored will degrade silently as the knowledge base grows, document quality varies, and query patterns shift. The evaluation investment pays back within the first month in the form of early warnings that would otherwise reach users as incorrect or hallucinated answers.