RAG Evaluation Metrics Explained - The 2026 Practical Guide

RAG feels simple until you have to prove it works. You retrieve some context, you generate an answer, and somewhere in that chain it goes wrong - but where? The retriever? The prompt? The model? RAG evaluation metrics exist to answer exactly that question.

This guide explains the metrics that matter, why they are split the way they are, and which tools ship them. Concept first, tools second.

Why RAG needs its own metrics

A plain LLM answer has one thing to judge: the output. A RAG answer has two stages, and each can fail on its own. First the retriever pulls context from your knowledge base. Then the model generates an answer from that context. A perfect model still fails if the retriever hands it garbage. A perfect retriever still fails if the model ignores what it got.

So you cannot score RAG with one number. You measure retrieval and generation separately, then read the two together to find the break. That split is the whole idea behind RAG-specific metrics.

The metrics that matter

There are four you will meet first, and they map cleanly onto the two stages.

Retrieval metrics:

• Context relevancy (or context precision). Of the chunks the retriever pulled, how many are actually relevant to the question. Low score means your retriever is pulling noise.
• Context recall. Of all the relevant chunks that exist, how many did the retriever find. Low score means it is missing information the answer needs.

Generation metrics:

• Faithfulness (or groundedness). Is every claim in the answer supported by the retrieved context. This is your hallucination check - anything the model asserts that is not in the context is invented. Reference-free, so it works in production.
• Answer relevancy. Does the answer actually address the question, regardless of whether it is correct.

The rule to remember: retrieval metrics judge the search, generation metrics judge the writing. Keep them separate and each failure points at a specific fix.

The RAG triad - reading the scores together

The reason to run three of these at once is diagnostic. It is often called the RAG triad - context relevancy, faithfulness, and answer relevancy - and the pattern of scores tells you exactly what to fix.

• Context relevancy low - your retriever is the problem. Tune chunking, embeddings or the top-k.
• Faithfulness low - the model is going past its context. Tighten the prompt or the model is over-answering.
• Answer relevancy low - the generation prompt is off, even if the facts are fine.

That is the value of measuring all three. One number tells you quality dropped. The triad tells you which stage to open up. If hallucination is your specific worry, we go deeper in how to measure LLM hallucination.

Where the tools fit

You do not write these metrics by hand. Three platforms ship them, with different strengths.

For depth, Arize Phoenix. Reviewers single out its RAG evaluation as the best in the category - it ships 50+ pre-built eval metrics with serious retrieval and answer scoring, and no judge prompts to write. It runs locally on your laptop in under a minute and is OpenTelemetry-native, so it fits any framework. The one thing to know: the server repo is Elastic License 2.0, source-available rather than OSI open source, which only matters if you plan to resell it as a hosted service. For internal RAG eval, that restriction never comes up.

For pytest-style RAG tests, DeepEval. It ships RAG metrics - faithfulness, answer relevancy and contextual scores - inside its 50+ research-backed metric library, and you run them with deepeval test run in CI. The framework is Apache-2.0 and free. The catch is that nearly all its metrics are LLM-as-judge, so a big RAG suite fires a lot of inference calls - runs take minutes and API bills add up. Cost-model it before you scale the suite.

For RAG scoring in CI gates, Braintrust. Its autoevals library ships working scorers and you add custom RAG scorers as plain functions, then its CI/CD quality gates block a merge on a statistically significant regression. If your goal is to stop a retrieval or prompt change from silently degrading answers, this is the turnkey path. Watch the processed-data billing meter - big RAG contexts stuffed into every call are exactly what burns the GB allowance fastest, and there is no hard cap.

So which one?

• You want the deepest RAG metrics and the fastest start - Arize Phoenix.
• You want RAG tests that run in CI like pytest - DeepEval, with an eye on the judge-call cost.
• You want to block merges that hurt RAG quality - Braintrust, with billing alerts on.

The takeaway: never score RAG with a single number. Measure retrieval and generation separately, read the triad together, and you will know which stage to fix instead of guessing. For the full field of platforms, see best RAG evaluation tools, and for metrics beyond RAG, LLM evaluation metrics explained.