07. RAG Evaluation Frameworks

• Primary References:

1. 1
   Es, S., James, J., Espinosa-Anke, L., & Schockaert, S. (2024). RAGAS: Automated Evaluation of Retrieval Augmented Generation. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (EACL 2024: System Demonstrations), pp. 150–158.
2. 2
   Saad-Falcon, J., Khattab, O., Potts, C., & Zaharia, M. (2024). ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems. In NAACL 2024.
3. 3
   Järvelin, K., & Kekäläinen, J. (2002). Cumulated Gain-Based Evaluation of IR Techniques. ACM Transactions on Information Systems (TOIS), 20(4), 422–446.
4. 4
   Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2024). Lost in the Middle: How Language Models Use Long Contexts. Transactions of the ACL (TACL), 12, 157–173.

• Why I Wrote This Note: As someone who bridges AI engineering and ML research, I noticed that most engineering teams tweak chunk sizes, embedding models, or hybrid BM25 weights by "eyeballing" \(5\) queries. I wrote this note to establish a reproducible, decomposed metric suite that isolates retriever bottlenecks from generator hallucinations.

• Never Evaluate RAG End-to-End Alone: A RAG pipeline is a composite function \(a = g_\phi\big(q,\, R_\theta(q, \mathcal{K})\big)\). You must evaluate the retriever \(R_\theta(q, \mathcal{K}) \to C_K\) and the generator \(g_\phi(q, C_K) \to a\) independently.
• Order-Unaware vs. Rank-Aware Retrieval Metrics: \(\text{Recall}@K\) measures whether the gold evidence made it into the top-\(K\) context window at all; \(\text{MRR}\) and \(\text{nDCG}@K\) penalize placing gold evidence at rank \(k=9\) instead of \(k=1\), which directly impacts LLM attention due to the Lost-in-the-Middle effect (Liu et al., 2024).
• Atomic Claim Decomposition for Faithfulness: Measuring hallucination at the paragraph level is noisy. Decomposing the generated answer \(a\) into atomic statements \(\mathcal{S}(a) = \{s_1, \dots, s_M\}\) and verifying whether retrieved context \(C_K\) entails each claim (\(C_K \models s_m\)) yields an interpretable Faithfulness Score in \([0, 1]\).
• Prediction-Powered Inference (ARES): LLM-as-a-Judge scores exhibit systematic bias; calibrating them against a small human-annotated gold set (\(150\text{–}300\) examples) via Prediction-Powered Inference (PPI) produces statistically valid confidence intervals.

4.1 Component 1: Retrieval Evaluation Metrics

Given a query \(q\), let \(C_K = (d_1, d_2, \dots, d_K)\) denote the ordered list of top-\(K\) document chunks returned by the retriever (and optional reranker), and let \(\mathcal{G}_q\) denote the set of ground-truth relevant chunks.

1. Recall@K and Precision@K (Order-Insensitive)

2. Mean Reciprocal Rank (\(\text{MRR}\), First-Hit Sensitive)

Let \(\text{rank}_1(q)\) denote the \(1\)-based position of the first relevant document in \(C_K\) (\(\text{rank}_1(q) = \infty\) if none exists). Over a query set \(\mathcal{Q}\):

3. Normalized Discounted Cumulative Gain (\(\text{nDCG}@K\), Graded & Rank-Sensitive)

When chunks have graded relevance scores \(r_i \in \{0, 1, 2, 3\}\) (e.g., \(3 = \text{exact answer}\), \(1 = \text{partial background}\)), \(\text{DCG}@K\) applies a logarithmic discount \(\log_2(i+1)\) to lower ranks:

where \(\text{IDCG}@K(q)\) is the Ideal \(\text{DCG}@K\) obtained by sorting the ground-truth relevance scores in descending order \(r_{(1)} \ge r_{(2)} \ge \dots \ge r_{(K)}\).

4.2 Component 2: The RAG Generation Triad

Once \(C_K\) is passed to the LLM generator \(g_\phi(q, C_K) \to a\), we evaluate three orthogonal axes linking Query (\(q\)), Context (\(C_K\)), and Answer (\(a\)):

flowchart TD
    Q["User Query (q)"]
    C["Retrieved Contexts (C_K)"]
    A["Generated Answer (a)"]
    Q <-- "1. Context Relevance / Precision@K<br/>Is C_K focused on q without noise?" --> C
    C <-- "2. Faithfulness / Groundedness<br/>Does C_K entail every claim in a? (C_K ⊨ s_m)" --> A
    Q <-- "3. Answer Relevance<br/>Does a directly address q's intent?" --> A

1. 1
   Context Precision (Signal-to-Noise Ratio of \(C_K\)):

Let \(\text{rel}_k \in \{0, 1\}\) indicate whether chunk \(d_k\) is needed to answer \(q\). RAGAS defines rank-weighted Context Precision as:

1. 1
   Faithfulness (Hallucination-Free Grounding):

Decompose the generated answer \(a\) into a set of atomic factual statements \(\mathcal{S}(a) = \{s_1, \dots, s_M\}\). An NLI verifier (or structured LLM judge) checks whether each claim \(s_m\) can be inferred strictly from \(C_K\):

1. 1
   Answer Relevance (Query Alignment):

Prompt an auxiliary generator to synthesize \(N_{\text{gen}}\) probable questions \(\{\hat{q}_1, \dots, \hat{q}_{N_{\text{gen}}}\}\) that answer \(a\) would naturally satisfy, and compute the mean cosine similarity in embedding space \(e(\cdot)\) with the original query \(q\):

4.3 Python Implementation of Retrieval & Faithfulness Evaluators

import math
from typing import Sequence

def compute_ir_metrics(
    retrieved_ids: Sequence[str],
    relevant_grades: dict[str, int],  # maps doc_id -> graded relevance in {1, 2, 3}
    k: int = 5,
) -> dict[str, float]:
    # Compute Recall@K, Reciprocal Rank (RR@K), and nDCG@K for a single query
    top_k = list(retrieved_ids[:k])
    relevant_set = {doc_id for doc_id, grade in relevant_grades.items() if grade > 0}

    # 1. Recall@K
    hits = sum(1 for doc_id in top_k if doc_id in relevant_set)
    recall_at_k = hits / max(len(relevant_set), 1)

    # 2. Reciprocal Rank (RR@K)
    rr = 0.0
    for idx, doc_id in enumerate(top_k):
        if doc_id in relevant_set:
            rr = 1.0 / float(idx + 1)
            break

    # 3. nDCG@K
    dcg = sum(
        (2 ** relevant_grades.get(doc_id, 0) - 1.0) / math.log2(idx + 2)
        for idx, doc_id in enumerate(top_k)
    )
    ideal_grades = sorted(relevant_grades.values(), reverse=True)[:k]
    idcg = sum((2 ** grade - 1.0) / math.log2(idx + 2) for idx, grade in enumerate(ideal_grades))
    ndcg_at_k = dcg / idcg if idcg > 0 else 0.0

    return {"recall@k": recall_at_k, "rr@k": rr, "ndcg@k": ndcg_at_k}

def compute_atomic_faithfulness(claim_entailment_scores: Sequence[float], threshold: float = 0.5) -> float:
    # Given NLI entailment probabilities P(C_K |= s_m) for each atomic claim s_m in answer a
    if not claim_entailment_scores:
        return 1.0
    supported = sum(1 for p in claim_entailment_scores if p >= threshold)
    return supported / float(len(claim_entailment_scores))

In my systems engineering and AI research, RAG is where Information Retrieval meets Uncertainty Quantification:

• Standard RAG pipelines treat retrieval scores and NLI faithfulness scores as deterministic heuristics. By combining RAG Faithfulness NLI scores with Conformal Prediction (Risk-Constrained AI Decision-Making) and Predictive Entropy (Understanding Predictive Uncertainty), we can build Self-Abstaining Clinical / Technical RAG Assistants that refuse to emit an ungrounded claim when \(\max_k \text{sim}(q, d_k) < \tau_{\text{ret}}\) or when atomic claim entailment drops below a calibrated threshold.

• Why LLM-as-a-Judge Fails Without Atomic Decomposition: Asking an LLM judge "Rate the faithfulness of this 300-word answer from 1 to 5" suffers from severe position bias, verbosity bias, and central tendency bias (clustering around \(4/5\)). Breaking the answer into \(8\) atomic sentences and scoring binary entailment \(\{0, 1\}\) per sentence via a specialized DeBERTa-v3 NLI cross-encoder is \(20\times\) cheaper, deterministic, and correlates much higher with human annotators.
• Embedding Cosine Similarity Saturation: In AnswerRelevance, raw cosine similarity of OpenAI/BGE embeddings compresses into \([0.72, 0.94]\) even for unrelated sentences due to anisotropy of the embedding cone. Always normalize against a baseline negative-pair distribution on your corpus.

• Implement compute_ir_metrics (\(\text{Recall}@K, \text{MRR}, \text{nDCG}@K\)) and atomic claim faithfulness scoring.Verified
• Integrate Reciprocal Rank Fusion (\(\text{RRF}(d) = \sum_{m} \frac{1}{60 + \text{rank}_m(d)}\)) benchmark comparing dense-only vs. BM25+Dense+Cross-Encoder on technical documentation.Pending
• Implement ARES Prediction-Powered Inference (PPI) confidence intervals to debias automated NLI/LLM judges using \(150\) gold human labels.Pending

"If your RAG evaluation only looks at the final generated string, you cannot tell whether you have a search engine problem or a language model hallucination problem."

Reciprocal Rank Fusion (RRF) & Atomic Faithfulness:

• Connected Research Notes:
• Understanding Predictive Uncertainty — Measuring token-level and semantic entropy of generated RAG responses (Kuhn et al., ICLR 2023).
• Risk-Constrained AI Decision-Making — Turning retrieval confidence and NLI faithfulness into formal selective abstention rules.
• LoRA: Mathematical Foundations — Fine-tuning domain-specific bi-encoder retrievers and lightweight NLI verifiers with LoRA.