Skip to content

RAG Evaluation

RAG quality has two dimensions: did you retrieve the right documents? and did the LLM use them correctly? This sheet covers evaluation metrics, the RAGAS framework, and practical patterns for measuring and improving RAG systems.

Table of Contents

Installation / Setup

pip install ragas evaluate       # or: uv add ragas evaluate

Core Syntax

The two dimensions of RAG quality

Dimension Question Metric
Retrieval quality Did we find the right documents? Context precision, recall, relevance
Generation quality Did the LLM use them correctly? Faithfulness, answer relevance

Key metrics

Metric What it measures Range How to improve
Faithfulness Answer is grounded in context 0–1 Better prompting, reranking
Answer relevance Answer addresses the question 0–1 Better prompts, more context
Context precision Retrieved docs are relevant 0–1 Better retrieval, reranking
Context recall All relevant docs are retrieved 0–1 Higher K, better chunking
Context relevance Context is on-topic 0–1 Better embedding model

RAGAS evaluation

from ragas import evaluate
from ragas.metrics import (
    faithfulness,
    answer_relevancy,
    context_precision,
    context_recall,
)

# prepare evaluation data
eval_data = {
    "question": ["What is RAG?", "How does chunking work?"],
    "answer": [
        "RAG retrieves relevant documents and uses them to generate answers.",
        "Chunking splits text into segments for embedding.",
    ],
    "contexts": [
        ["RAG stands for Retrieval-Augmented Generation..."],
        ["Chunking is the process of splitting text..."],
    ],
    "ground_truth": [
        "Retrieval-Augmented Generation combines retrieval with generation.",
        "Chunking splits documents into smaller pieces.",
    ],
}

result = evaluate(
    dataset=eval_data,
    metrics=[faithfulness, answer_relevancy, context_precision, context_recall],
)
print(result)

Common Operations

Retrieval evaluation (without LLM)

def evaluate_retrieval(retriever, eval_queries):
    """Simple retrieval hit-rate."""
    hits = 0
    total = len(eval_queries)
    for query, expected_keywords in eval_queries:
        docs = retriever.invoke(query)
        text = " ".join(d.page_content for d in docs).lower()
        if any(kw.lower() in text for kw in expected_keywords):
            hits += 1
    return hits / total

# example
eval_queries = [
    ("What is Python?", ["programming", "language"]),
    ("How do I train a model?", ["gradient", "loss"]),
]
print(f"Hit rate: {evaluate_retrieval(retriever, eval_queries):.2%}")

Faithfulness check (with LLM)

from openai import OpenAI

client = OpenAI(api_key="YOUR_API_KEY")

def check_faithfulness(answer, context, llm="gpt-4o-mini"):
    response = client.chat.completions.create(
        model=llm,
        messages=[{"role": "system", "content": """Rate the faithfulness of the answer to the context.
Score 1-5:
1 = completely fabricated, not in context
2 = mostly fabricated
3 = partially supported
4 = mostly supported
5 = fully supported by context
Return only the score."""},
        {"role": "user", "content": f"Context:\n{context}\n\nAnswer:\n{answer}"}],
        temperature=0.0,
    )
    return int(response.choices[0].message.content.strip())

Answer relevance check

def check_relevance(answer, question, llm="gpt-4o-mini"):
    response = client.chat.completions.create(
        model=llm,
        messages=[{"role": "system", "content": """Rate how well the answer addresses the question.
Score 1-5:
1 = completely irrelevant
2 = tangentially related
3 = partially addresses
4 = mostly addresses
5 = fully addresses the question
Return only the score."""},
        {"role": "user", "content": f"Question:\n{question}\n\nAnswer:\n{answer}"}],
        temperature=0.0,
    )
    return int(response.choices[0].message.content.strip())

Practical Examples

Example 1 — End-to-end evaluation pipeline

import json
from openai import OpenAI

client = OpenAI(api_key="YOUR_API_KEY")

def evaluate_rag_system(rag_chain, eval_dataset):
    results = []
    for item in eval_dataset:
        answer = rag_chain.invoke(item["question"])

        # evaluate each dimension
        faithfulness = check_faithfulness(answer, item.get("context", ""))
        relevance = check_relevance(answer, item["question"])

        results.append({
            "question": item["question"],
            "answer": answer,
            "faithfulness": faithfulness,
            "relevance": relevance,
        })

    # aggregate
    avg_faith = sum(r["faithfulness"] for r in results) / len(results)
    avg_rel = sum(r["relevance"] for r in results) / len(results)
    print(f"Avg faithfulness: {avg_faith:.2f}/5")
    print(f"Avg relevance: {avg_rel:.2f}/5")
    return results

Example 2 — Build an evaluation dataset

eval_dataset = [
    {
        "question": "What is the main topic of this document?",
        "context": "...",      # optional: expected context
        "ground_truth": "...",  # optional: reference answer
    },
    {
        "question": "What are the key findings?",
        "context": "...",
        "ground_truth": "...",
    },
]
# save for reuse
with open("eval_dataset.json", "w") as f:
    json.dump(eval_dataset, f, indent=2)

Example 3 — RAGAS automated evaluation

from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_precision

# build eval dataset from your RAG system
eval_data = {
    "question": questions,
    "answer": answers,
    "contexts": contexts,
    "ground_truth": ground_truths,   # optional for some metrics
}

result = evaluate(
    dataset=eval_data,
    metrics=[faithfulness, answer_relevancy, context_precision],
)
print(result)

Common Patterns

Pattern: A/B testing RAG configurations

def compare_configs(rag_chain_a, rag_chain_b, eval_dataset):
    results_a = evaluate_rag_system(rag_chain_a, eval_dataset)
    results_b = evaluate_rag_system(rag_chain_b, eval_dataset)

    faith_a = sum(r["faithfulness"] for r in results_a) / len(results_a)
    faith_b = sum(r["faithfulness"] for r in results_b) / len(results_b)
    print(f"Config A faithfulness: {faith_a:.2f}")
    print(f"Config B faithfulness: {faith_b:.2f}")

Pattern: regression testing

def regression_test(rag_chain, eval_dataset, threshold=4.0):
    """Ensure quality stays above threshold."""
    results = evaluate_rag_system(rag_chain, eval_dataset)
    avg_score = sum(r["faithfulness"] for r in results) / len(results)
    assert avg_score >= threshold, f"Quality dropped: {avg_score:.2f} < {threshold}"
    print(f"✓ All scores above {threshold}")

Best Practices

  • Always evaluate — you can't improve what you don't measure.
  • Build a ground-truth dataset — even 20–50 examples help.
  • Track both retrieval AND generation — they fail in different ways.
  • Use RAGAS for automated evaluation at scale.
  • A/B test changes — chunking, retrieval, reranking all affect quality.
  • Monitor in production — log questions, answers, and retrieval scores.
  • Faithfulness is the most important metric — hallucination is the #1 RAG failure.

Common Mistakes

Mistake Fix
Not evaluating at all Build a test set and measure regularly
Only measuring answer quality Also measure retrieval quality
Using only automated metrics Add human evaluation for critical applications
Not tracking changes over time Log metrics with each deployment
Evaluating on trivial examples Include edge cases, ambiguous questions

Troubleshooting

Symptom Cause / Fix
Low faithfulness LLM ignoring context — improve prompt or reranking
Low answer relevance Bad retrieval — improve chunking/embeddings
Low context precision Retrieving irrelevant docs — add reranking
Low context recall Missing relevant docs — increase K, improve chunking
RAGAS errors Check data format — each field must be a list

Official Documentation

Last Updated

2026-09-06 — Initial version.