Evaluating RAG Systems

Two systems wearing one trench coat

A retrieval-augmented generation pipeline looks like one component from the outside - question in, answer out - but it is two systems stacked. First a retriever pulls relevant chunks from your knowledge base. Then a generator writes an answer conditioned on those chunks. When the answer is wrong, the single most important question is which stage failed, and end-to-end scoring cannot tell you. This is why RAG evaluation is its own discipline and why applying the plain metrics from the last chapter without decomposition leaves you blind.

The mental model to hold - garbage retrieval guarantees a garbage answer no matter how good your generator is, and a perfect retriever is wasted if the generator ignores what it fetched. You have to measure both, separately, before you can improve either.

The RAG triad

The cleanest framework for this is the RAG triad, three metrics that cover the three relationships in the pipeline.

Context relevancy (retriever quality) - are the retrieved chunks actually relevant to the question? This scores the retriever in isolation. If it is low, your embedding model, chunking strategy, or index is the problem, and nothing downstream can save you.

Faithfulness (generator grounding) - does the answer stick to the retrieved chunks rather than inventing? You met this in the last chapter. In a RAG context it specifically catches the generator going off-script from good retrieval.

Answer relevancy (generator usefulness) - does the answer address the question? A generator can faithfully summarize retrieved chunks that do not actually answer what was asked.

If all three are high, your pipeline works. If context relevancy is high but faithfulness is low, fix the generator prompt. If context relevancy is low, no amount of generator tuning helps. This triage is the entire value of the framework.

Retrieval metrics in detail

The generator metrics are the ones from the previous chapter, so the new material is the retrieval side. Two metrics matter most.

Context precision asks - of the chunks we retrieved, how many are relevant, and are the relevant ones ranked near the top? Precision punishes a retriever that returns ten chunks where only two are useful and buries them at positions eight and nine. Ranking matters because generators weight earlier context more heavily and because you often truncate to a token budget. A precision-aware metric rewards putting the signal first.

Context recall asks the opposite - of all the chunks that should have been retrieved to answer this question, how many did we actually get? Recall is the one retrieval metric that genuinely needs ground truth, because to compute it you must know the complete set of relevant documents. If recall is low, the answer is missing information the knowledge base contained, and the generator never had a chance. This is the most common and most damaging RAG failure, and it is invisible unless you measure it.

The tension between them is familiar from classic information retrieval. Retrieve more chunks and recall rises but precision falls. Retrieve fewer and precision rises but you risk missing something. RAG evaluation makes that tradeoff visible so you can tune your top-k and your reranker against real numbers instead of vibes.

A worked example

Suppose a user asks “what is our parental leave policy for contractors?” and the system answers “contractors are not eligible for parental leave,” which is wrong - the policy changed last quarter.

• Context relevancy is high - the retrieved chunks are all about leave policy.
• Context recall is low - the retriever pulled the old policy doc and missed the updated one.
• Faithfulness is high - the answer faithfully reflects the outdated chunk it was given.

Every generation metric looks fine. Only context recall exposes the real fault, and it points you straight at the retriever and the index freshness, not the prompt. Grade end-to-end and you would have wasted a day tuning the generator on a problem it did not have.

Building the eval set

RAG evals need a dataset of question, ideal-context, and ideal-answer triples. Writing these by hand is slow, so most teams use synthetic generation - point a model at your document corpus and have it generate questions each document can answer, along with the source chunk as ground-truth context. DeepEval and other frameworks include synthesizers for exactly this. Curate the synthetic set, correct the obvious junk, and you have a reusable benchmark you can run on every retriever change.

How the tools do it

DeepEval ships the full RAG metric suite - contextual precision, contextual recall, contextual relevancy, faithfulness, and answer relevancy - as pytest cases, so a retrieval regression fails CI like any other test. Arize Phoenix is built on OpenTelemetry and shines at the tracing side - it captures every retrieval span so you can open a bad answer and see exactly which chunks were fetched and scored, then run evals over those traces. Langfuse attaches retrieval and generation scores to production traces so you can watch context recall on live traffic, not just a test set. Promptfoo lets you assert RAG metrics from YAML when you want retrieval checks in a lightweight CLI flow.

For the deeper walkthrough, how to evaluate RAG builds a pipeline end to end, RAG evaluation metrics explained has every formula, and best RAG evaluation tools compares the platforms. Since hallucination is the failure faithfulness catches, how to measure LLM hallucination pairs naturally with this chapter.

Key takeaways

• Treat RAG as two systems - grade the retriever and the generator separately or you will not know what to fix.
• The RAG triad - context relevancy, faithfulness, answer relevancy - covers the three relationships in the pipeline.
• Context precision scores whether retrieved chunks are relevant and well ranked. Context recall scores whether you retrieved everything you needed, and it needs ground truth.
• Low recall is the most common and most damaging failure, and it is invisible to end-to-end scoring.
• Build a reusable eval set with synthetic question-context-answer triples, then run it on every retriever change.

You now have the core toolkit - traces, judges, metrics, and RAG decomposition. From here the path leads into running these continuously in production and wiring them into CI, which is where evaluation stops being a one-off exercise and becomes part of how you ship.