Module 4: Retrieval and Reranking · RAG

Top-5 dense retrieval returns five chunks. One is exactly right, three are loosely related, and one is completely wrong. Your generator gets all five and produces an answer that blends truth with noise. This is the retrieval precision problem, and it is the most common reason RAG systems give inconsistent answers. By the end of this module you will know how sparse retrieval, dense retrieval, hybrid fusion, and cross-encoder reranking each address it, and when to use each.

Two Families of Retrieval

There are two fundamentally different ways to score documents against a query: matching on words (sparse retrieval) or matching on meaning (dense retrieval). Each has strengths the other lacks.

Sparse retrieval uses term frequency statistics. BM25, the most widely used sparse method, scores documents based on how often query terms appear in the document, adjusted for document length and term rarity across the corpus. A query for "BM25 weighting function" will score highly any document containing those exact words or close variants. BM25 is fast, interpretable, and surprisingly strong on keyword-heavy queries. It fails when the query and document use different vocabulary for the same concept.

Dense retrieval uses neural embeddings. The DPR (Dense Passage Retrieval) approach by Karpukhin et al. (arXiv:2004.04906) trains separate encoders for queries and passages. Both are mapped to the same embedding space, and retrieval finds passages whose embeddings are closest to the query embedding. Dense retrieval captures synonyms, paraphrases, and conceptual similarity that BM25 misses entirely. Its weakness is that it may deprioritize exact keyword matches that matter for technical queries.

The vocabulary mismatch problem A regulatory document refers to "elevated cardiovascular risk." A user asks about "heart disease danger." BM25 scores this match near zero: no shared terms. A dense retriever maps both to nearby points in embedding space and returns the correct document. This is why dense retrieval dominates for knowledge-intensive Q&A, while BM25 remains strong for code search and exact-term technical lookups.

Hybrid Retrieval with RRF

The practical solution is to run both and combine results. Reciprocal Rank Fusion (RRF) takes the ranked lists from each retriever and assigns each document a score based on its rank position, then sums across retrievers. Documents ranked highly by both retrievers score highest in the merged list.

The RRF formula for document d given a set of ranked lists R is: score(d) = sum over each list of 1 / (k + rank(d, list)), where k is a smoothing constant (typically 60). Documents missing from one list get a penalty proportional to how far down they would fall if they appeared.

Hybrid retrieval consistently outperforms either sparse or dense retrieval alone on mixed-vocabulary corpora, because it captures both exact-term matches (BM25) and semantic matches (dense). REPLUG (Shi et al., arXiv:2301.12652) demonstrates that augmenting LMs with strong retrieval pipelines produces measurable gains across knowledge-intensive benchmarks.

Fig 4 · Hybrid Retrieval with Cross-Encoder Reranking

The Reranking Stage

The retriever's job is recall: return the top 20-50 candidates quickly. The reranker's job is precision: pick the 3-5 most relevant from that candidate set.

A bi-encoder (what your embedding retriever uses) encodes the query and each document independently, then computes similarity. This is fast because document embeddings can be precomputed. But it is approximate: the query and document representations never directly interact during encoding.

A cross-encoder concatenates the query and document into one input and runs them together through the model. This allows every query token to attend to every document token, which produces much higher-precision relevance scores. The cost is that cross-encoders cannot precompute document representations, making them 10-100x slower per pair. This is why rerankers are used as a second pass over a small candidate set, not as the primary retriever.

In practice, running a cross-encoder reranker over the top-20 candidates from hybrid retrieval takes 200-500ms and substantially improves precision@3, which is what the language model actually uses.

Python · BM25, Dense Retrieval, and Cross-Encoder Reranker

from rank_bm25 import BM25Okapi
from sentence_transformers import SentenceTransformer, CrossEncoder
import numpy as np

chunks = ["patient hypertension elevated...", "revenue targets exceeded...", ...]

# BM25 sparse retrieval
tokenized = [c.lower().split() for c in chunks]
bm25 = BM25Okapi(tokenized)

def sparse_retrieve(query, k=20):
    scores = bm25.get_scores(query.lower().split())
    top_ids = np.argsort(scores)[::-1][:k]
    return [(i, scores[i]) for i in top_ids]

# Dense retrieval (already indexed in FAISS from Module 2)
bi_encoder = SentenceTransformer('all-MiniLM-L6-v2')

# RRF fusion
def rrf_merge(sparse_ids, dense_ids, k=60):
    scores = {}
    for rank, (idx, _) in enumerate(sparse_ids):
        scores[idx] = scores.get(idx, 0) + 1 / (k + rank + 1)
    for rank, idx in enumerate(dense_ids):
        scores[idx] = scores.get(idx, 0) + 1 / (k + rank + 1)
    return sorted(scores.items(), key=lambda x: x[1], reverse=True)

# Cross-encoder reranker
reranker = CrossEncoder('cross-encoder/ms-marco-MiniLM-L-6-v2')

def rerank(query, candidate_ids, top_k=5):
    pairs = [(query, chunks[i]) for i in candidate_ids]
    scores = reranker.predict(pairs)
    ranked = sorted(zip(candidate_ids, scores), key=lambda x: x[1], reverse=True)
    return [idx for idx, _ in ranked[:top_k]]

Try this

Take 20 passages from a document in your domain and write 5 test queries. Run BM25 retrieval on each query and note the top-3 results. Then design a query where BM25 clearly fails: a query that uses different vocabulary than the relevant passages. This exercise builds intuition for when you need dense retrieval and when BM25 is actually the right choice.