Module 5: RAG Evaluation · RAG

Your RAG pipeline answers questions fluently. Users seem satisfied. But three weeks later, you discover it has been giving confident wrong answers for a specific category of questions, and nobody caught it because the answers sounded plausible. Evaluation is not a post-launch checklist. It is a continuous measurement system. By the end of this module you will understand the four RAGAS dimensions, how to measure each one, and how to build an evaluation loop that catches problems before users do.

The Three Evaluation Axes

A RAG system can fail in three distinct places. Each failure mode requires a different metric to detect it.

Retrieval quality measures whether the retriever found the right documents. If the right documents are not in the context, the generator cannot produce a correct answer regardless of how capable it is. Retrieval failure is invisible in end-to-end accuracy: you see a wrong answer, but you do not know if it came from bad retrieval or bad generation.

Generation faithfulness measures whether the generated answer is supported by the retrieved context. A faithfully grounded answer only makes claims that are directly stated or logically implied by the context. An unfaithful answer introduces facts that the context does not support, which is the definition of hallucination in RAG systems.

Answer relevance measures whether the answer actually addresses the question asked. A perfectly faithful answer to the wrong question is useless. This dimension catches cases where the retriever found relevant context but the generator went off-topic or answered a different aspect of the question.

The critical distinction: faithfulness vs fluency A language model can generate a fluent, confident, well-structured answer that contains no information from the retrieved context. It sounds correct. It reads well. It is completely wrong. Fluency evaluations catch nothing here. Faithfulness evaluation asks: is every claim in this answer supported by the provided context? That is the question that matters for RAG reliability.

The RAGAS Framework

RAGAS (Retrieval Augmented Generation Assessment), introduced by Es et al. (arXiv:2309.15217), provides a standardized set of four metrics for RAG evaluation. All four can be computed without human-labeled ground truth for the evaluation examples, using an LLM as the evaluator.

Metric 1

Context Precision

Of the retrieved chunks, what fraction are actually relevant to the question? High context precision means the retriever is not polluting the context with noise.

Metric 2

Context Recall

Of the information needed to answer the question, what fraction was present in the retrieved chunks? Low context recall means the retriever missed key information.

Metric 3

Faithfulness

What fraction of the claims in the generated answer are directly supported by the retrieved context? The hallucination detector for RAG systems.

Metric 4

Answer Relevance

Does the generated answer actually address the question asked? Penalizes answers that are faithful to the context but miss the point of the question.

These four metrics cover the full failure space of a RAG pipeline. Context precision and recall evaluate the retriever. Faithfulness evaluates the generator's grounding. Answer relevance evaluates end-to-end usefulness.

LLM-as-Judge for RAG Evaluation

Human annotation is accurate but slow and expensive. RAGAS uses an LLM (typically GPT-4 class) as the evaluator. For faithfulness, the LLM is asked to decompose the generated answer into atomic claims, then verify each claim against the retrieved context. For context precision, it is asked to label each retrieved chunk as relevant or irrelevant to the question.

This approach scales to thousands of test cases automatically. The LLM-as-judge methodology has known limitations: the judge model itself can make errors, and it tends to favor fluent answers. But for the specific structural questions RAGAS asks (is this claim in the context? is this chunk relevant?), judge accuracy is high enough to be practically useful.

Research on clinical AI systems, including work like Rao, Jaggi, Naidu (IEEE RMKMATE 2025, DOI:10.1109/RMKMATE64574.2025.11042816), highlights that domain-specific evaluation is essential: a general evaluator may not catch domain-specific hallucinations that a clinical or legal expert would immediately identify. RAGAS provides the framework; domain knowledge determines what to measure.

Fig 5 · RAGAS Evaluation Dimensions

Python · RAGAS Evaluation Loop

from datasets import Dataset
from ragas import evaluate
from ragas.metrics import (
    context_precision,
    context_recall,
    faithfulness,
    answer_relevancy,
)

# Evaluation dataset: questions + ground truth + RAG outputs
data = {
    "question": [
        "What is the refund policy for digital products?",
        "How do I reset my password?",
    ],
    "answer": [
        rag_system.query("What is the refund policy for digital products?"),
        rag_system.query("How do I reset my password?"),
    ],
    "contexts": [
        retrieve("What is the refund policy for digital products?"),
        retrieve("How do I reset my password?"),
    ],
    "ground_truth": [
        "Digital products are non-refundable within 30 days of purchase.",
        "Visit the login page and click 'Forgot Password' to receive a reset email.",
    ]
}

dataset = Dataset.from_dict(data)
result = evaluate(
    dataset=dataset,
    metrics=[context_precision, context_recall, faithfulness, answer_relevancy],
)
print(result.to_pandas())

Try this

Write 5 questions for a knowledge base you know well. For each question, also write the ideal answer. Now imagine a RAG system retrieved the wrong context chunks and the model generated an answer that sounds correct but does not match your ideal. Score each of the four RAGAS dimensions by hand on a 0-10 scale. This manual scoring exercise makes the metrics concrete and builds intuition for what each dimension actually catches in practice.