MTMoueen Togarvi5 min read

Retrieval Evaluation Metrics for RAG Systems

Quick overview

Updated

Choose and apply retrieval evaluation metrics: recall and precision at k, ranking measures, faithfulness, building a test set, and diagnosing whether retrieval or generation failed.

Retrieval evaluation metrics help distinguish missing evidence from an answer that misuses available evidence. Evaluate the retrieval stage and the generated response separately so a change targets the measured failure rather than a guessed cause.

Separate the two stages

Evaluate retrieval and generation independently before evaluating end to end.

Retrieval evaluation asks whether the documents needed to answer the question were returned. Generation evaluation asks whether the answer is correct and faithful given those documents.

An end-to-end score alone cannot reliably distinguish them. Microsoft's RAG evaluator documentation separates document retrieval evaluation from response groundedness and relevance. Those are different questions: a response can repeat the supplied context faithfully even when that context is outdated or irrelevant.

Retrieval metrics

The Stanford information retrieval text explains ranked retrieval evaluation, including precision at a cutoff and graded-relevance measures. Choose a metric that matches the evidence your task actually needs.

  • Recall at k. The fraction of labelled relevant items appearing in the top k results. Define whether an item is a document or a chunk before comparing runs; duplicate chunks from one document can otherwise inflate an apparent improvement.

  • Precision at k. Of the returned documents, how many were relevant. Precision matters more than intuition suggests, since irrelevant passages act as distractors and degrade answer quality.

  • Mean reciprocal rank. Where the first relevant document appeared. Useful when a single correct source exists.

  • Normalized discounted cumulative gain. Accounts for graded relevance and position, appropriate when some documents are more useful than others.

Track recall at the k you actually pass to the model, not at a larger k that flatters the number. Recall at fifty is irrelevant if you send three passages.

Generation metrics

  • Faithfulness. Whether claims in the answer are supported by the supplied context. This does not establish that the source itself is accurate or that the answer contains every required fact.

  • Answer relevance. Whether the answer addresses the question, independent of correctness.

  • Correctness against a reference answer where one exists.

  • Appropriate refusal. Whether the system declines when the answer is genuinely not in the corpus. Systems are frequently never tested on unanswerable questions, and then fabricate confidently in production.

Building the test set

This is the work that determines whether evaluation is meaningful.

Source questions from real usage: support tickets, search logs, and user interviews. Synthetic questions generated from your documents test whether retrieval finds the document a question was generated from, which is an easier task than real queries.

For each question, record the answer and the specific documents or passages that support it. This annotation is the expensive part and the part that makes diagnosis possible.

Include categories deliberately: straightforward lookups, questions requiring multiple documents, questions with near-miss distractors in the corpus, ambiguous questions, and questions with no answer in the corpus.

Keep a held-out set for the final comparison and record annotation disagreements. There is no universal minimum sample size: rare but costly failures need deliberate coverage even if common questions dominate traffic.

Worked example: recall improves while precision falls

Suppose a test question has three relevant document IDs: A, B, and C. A retriever returns A, D, B, E, and F as its top five. Two returned documents are relevant, so precision at five is 2 / 5 = 0.40 and recall at five is 2 / 3, approximately 0.67. The reciprocal rank for this query is 1, because the first result is relevant. Mean reciprocal rank averages that value across the query set.

If the context budget permits only the first two documents, the generation step receives A and D: its effective recall is 1 / 3, not the top-five score. Record both the retriever's candidate results and the passages actually supplied after reranking, deduplication, and truncation.

Now suppose C is a policy exception required for the correct answer. A system could earn a respectable average retrieval score while omitting that exception. Add a case-level check for the required evidence and evaluate the final answer against it. This example is a diagnostic calculation, not a benchmark result from a deployed client system.

Diagnosing failures

When an answer is wrong, check whether an authorised, current supporting document exists and was returned. Missing evidence can arise from ingestion, access filters, chunking, embedding, ranking, or query construction. A prompt that controls query generation can also affect retrieval, so inspect the full request path.

If the document was returned and the answer is still wrong, examine position and context size. Evidence buried among many passages is often effectively ignored, and cutting to fewer, better-ranked results frequently resolves it.

If the document was returned, well-positioned, and the answer is still wrong, then it is a generation problem worth addressing through prompting or model choice.

Record retrieved document IDs, versions, ranks, and the final context selection with production request identifiers. Store raw personal content only where necessary and governed; identifiers and versioned test fixtures can often support diagnosis without copying every document into logs.

Running it continuously

Run the suite on every change to chunking, embedding models, retrieval parameters, reranking, prompts, or generation models. Each of these changes behavior in ways that are not predictable.

Set thresholds that block deployment, and track metrics over time. Slow degradation as a corpus grows is common and invisible without trend data.

Frequently asked questions

Which single metric matters most?

For evidence-dependent answers, recall at the production context limit is a useful starting point. Pair it with answer correctness and appropriate refusal; no single metric establishes system quality.

How large should the test set be?

Enough to cover your question categories with several examples each. Annotation quality matters more than volume.

Can models evaluate the outputs?

Model-based grading is practical for faithfulness and relevance at scale, and should be validated against human judgment on a sample before being trusted.

Why does retrieval degrade over time?

Corpus growth adds near-duplicates and distractors. Periodic re-evaluation and reranking adjustments are ongoing work.

For the surrounding architecture decision, read RAG versus fine-tuning. Explore our software and web capabilities or discuss an evaluation plan using examples from your own corpus.

MT

Written by

Moueen Togarvi

Founder & CEO at Voquarn Code, focused on product engineering, search growth, and practical AI systems.

About author
Turn the insight into action

Need a practical plan for your next digital project?

Tell us what you are building. We will help you clarify the scope, technical approach, and highest-value first step.

Discuss your project