RAG Evaluation: RAGAS, Faithfulness, and LLM-as-Judge

Your RAG pipeline has been live for three weeks. Users seem satisfied. Then someone notices the system has been confidently wrong about a specific topic category the entire time, and every answer sounded so plausible that nobody questioned it. Evaluation is not a launch checklist. It is the continuous measurement system that catches this before three weeks pass.

Why Standard Accuracy Metrics Fail for RAG

End-to-end accuracy tells you whether the final answer is correct. It does not tell you why. A wrong answer might come from bad retrieval (the correct document was never found), bad generation (the correct document was retrieved but the model misread it), or bad context assembly (the correct chunk was found but truncated). Without decomposed metrics, you cannot fix what you cannot locate.

RAG also fails in a mode that accuracy misses entirely: fluent hallucination. A language model can produce a fluent, confident, well-structured answer that introduces facts not present in any retrieved document. To a reader unfamiliar with the actual content, the answer appears correct. End-to-end accuracy only catches this if the evaluator happens to know the correct answer independently. Faithfulness evaluation catches it systematically.

The RAGAS Framework

RAGAS (Retrieval Augmented Generation Assessment), introduced by Es et al. (arXiv:2309.15217), defines four metrics that together cover the full failure space of a RAG pipeline. Each metric can be computed without human-labeled ground truth for the specific generated answers, using an LLM as the evaluator.

Context Precision

Of the retrieved chunks, what fraction are actually relevant to the question? An LLM evaluator labels each chunk as relevant or irrelevant. Context precision = relevant chunks / total retrieved chunks. Low precision means the retriever is flooding the context with noise. Typical causes: chunk size too large, embedding model not suited to the domain, or missing metadata filters that would have excluded off-topic content.

Context Recall

Of the information needed to answer the question correctly, what fraction was present in the retrieved chunks? This requires a reference answer (from a human or a trusted source) to determine what information "should" be present. Low recall means the retriever missed key documents. Typical causes: the right documents were not chunked, the query phrasing differs too much from document phrasing (the fix is hybrid retrieval), or the relevant content is in a document type that was excluded from indexing.

Faithfulness

What fraction of the claims in the generated answer are directly supported by the retrieved context? The evaluator LLM decomposes the answer into atomic claims, then checks each claim against the context. Faithfulness = supported claims / total claims. This is the hallucination detector for RAG: a faithfulness score below 0.8 means the model is regularly supplementing context with parametric knowledge, which cannot be relied on for correctness.

"Faithfulness measures whether the model stayed in the lane of the retrieved context. A fluent, well-written answer can have faithfulness zero."

Answer Relevance

Does the generated answer address the question that was actually asked? The evaluator generates several questions that the answer could be answering, then measures how similar those generated questions are to the original. Low answer relevance means the model answered a different aspect of the question or went off-topic despite having relevant context. Typical causes: a system prompt that encourages verbose responses, or context that pulls the model toward a related but not identical topic.

LLM-as-Judge: Scaling Evaluation

Human annotation is the gold standard but costs approximately $0.50-2.00 per evaluated example at professional annotation rates. For a system handling 10,000 queries per week, full human evaluation is impractical. RAGAS uses an LLM (typically a GPT-4 class model) as the evaluator, bringing cost to approximately $0.01-0.05 per example and enabling evaluation at scale.

The LLM judge has known biases: it favors fluent and confident-sounding answers, and it struggles with domain-specific accuracy in fields like clinical medicine, law, or quantitative finance where the judge model's parametric knowledge may itself be incorrect. Research including work at the intersection of clinical AI and evaluation methodology (Rao, Jaggi, Naidu, IEEE RMKMATE 2025, DOI:10.1109/RMKMATE64574.2025.11042816) shows that domain-specific evaluation frameworks are necessary: a general evaluator may score a clinically hallucinated answer highly because it sounds medically plausible to a non-specialist.

The practical approach: use RAGAS for continuous monitoring of the pipeline's structural properties (is the retriever returning relevant chunks? is the generator staying in context?), and reserve human evaluation or domain-expert review for samples where faithfulness is low or for evaluation of high-stakes answer categories.

Interpreting Your RAGAS Scores

When you run RAGAS on a test set and get scores, the combination of low scores tells you specifically what to fix. Low context precision means retrieval is returning noise: improve chunking, add metadata filters, or add reranking. Low context recall means retrieval is missing documents: check that all relevant document types are indexed, and consider adding hybrid retrieval if you are only using dense search. Low faithfulness means the generator is hallucinating: strengthen the system prompt instruction, check that the context is actually reachable within the token limit, or reduce k (the number of retrieved chunks) to reduce the chance that irrelevant context triggers parametric recall. Low answer relevance means the model is drifting off-topic: the most common causes are an overly permissive system prompt and retrieved context that is topically adjacent but not directly responsive to the query.

Track all four scores over time, not just the average. A declining faithfulness score with stable context precision indicates a model behavior change (possibly from a provider model update). A declining context precision with stable faithfulness indicates index quality degradation. Each pattern points to a different root cause and a different fix.

Building a Regression Test Suite for RAG

RAGAS evaluation on production samples catches regressions after they occur. A regression test suite catches them before deployment. The goal is a curated set of question-answer pairs where the expected answer is known, run automatically on every change to the knowledge base or pipeline configuration.

A useful regression suite has 50-200 questions covering the full range of query types your application handles. It should include questions with clear, retrievable answers (for measuring faithfulness), questions that test edge cases (out-of-scope queries, ambiguous phrasing, questions about recently updated content), and questions from real user sessions that previously produced incorrect answers (regression tests for known bugs).

Each test case stores the question, the expected answer summary, and the source document identifier. At test time, the pipeline runs the question, RAGAS scores the result, and the test passes if faithfulness exceeds the threshold. Any test that drops below threshold triggers a review of what changed: a new document was added that introduced a confounding chunk, the embedding model was updated, or a chunking parameter was changed. This discipline forces teams to understand the impact of every pipeline change rather than discovering degradation through user complaints.

For teams managing multiple RAG applications, the regression suite also serves as a benchmark for comparing pipeline configurations. You can run two chunking strategies, two embedding models, or two retrieval depths against the same test suite and choose the configuration that produces the highest scores. This structured evaluation approach replaces ad-hoc "let me try a few queries" testing with reproducible, quantitative evidence for configuration decisions.

Domain-Specific Evaluation: When General Metrics Are Not Enough

RAGAS faithfulness measures whether the generated answer is grounded in the retrieved context. It does not measure whether the retrieved context is factually correct in the real world. In most applications, these are the same question: if the knowledge base is authoritative, grounded answers are correct answers. In specialized domains like clinical medicine, law, and quantitative finance, they diverge.

A clinical knowledge base may contain conflicting guidelines from different years, outdated treatment protocols, or institution-specific procedures that differ from national standards. A RAG system with high faithfulness scores may still produce answers that are clinically incorrect because it faithfully reproduced content that is medically outdated. Research at the intersection of clinical AI and evaluation methodology (Rao, Jaggi, Naidu, IEEE RMKMATE 2025, DOI:10.1109/RMKMATE64574.2025.11042816) demonstrates that domain-specific evaluation is not optional in healthcare applications: a general RAGAS evaluator would miss hallucinations that a clinical expert reviewer catches because the evaluator lacks the domain knowledge to identify the error.

The solution for high-stakes domains is a layered evaluation strategy. RAGAS handles pipeline-level evaluation (is the retriever working? is the generator staying in context?). Domain expert review handles content-level evaluation (is this medically, legally, or financially correct?). The two layers are complementary: RAGAS scales to thousands of examples per week automatically, while domain expert review focuses on a smaller random sample plus any examples that RAGAS flagged as low-faithfulness. Together they provide both coverage and depth.

Evaluation in Practice: A Recommended Schedule

A practical evaluation schedule for a RAG system in regular use combines automated weekly runs with monthly human review.

Weekly: run RAGAS on a 5% random sample of production traffic from the prior week. Track context precision, context recall, faithfulness, and answer relevance. Alert if faithfulness drops below 0.75 or context precision drops below 0.70. Log all low-scoring examples for review.

Monthly: review the flagged low-scoring examples with a domain expert or senior practitioner. Identify whether failures are due to knowledge base gaps (missing documents, outdated content), chunking issues (relevant information split across chunk boundaries), retrieval failures (correct document not returned), or generation failures (model not staying within context). Update the regression test suite with any new failure patterns discovered during review.

Quarterly: run the full regression test suite against any major pipeline change (new embedding model version, significant knowledge base expansion, chunking parameter updates). Require regression suite passage before promoting changes to the pilot environment. This schedule provides continuous coverage without over-burdening the evaluation process or creating alert fatigue from too-frequent thresholds.