Answer Relevancy

What it means

Answer relevancy measures whether a response stays on target for the question that was asked. It is usually computed with an LLM-as-judge that reads the question and the answer, then rates how directly the answer speaks to the intent. Some implementations generate a set of questions the answer could plausibly be responding to, embed them, and compare them against the original question. The closer the match, the higher the score.

Importantly, relevancy is not about truth. An answer can be perfectly factual and still score low because it buried the point under caveats, restated the question, or drifted into a tangent. A separate faithfulness or correctness metric handles accuracy.

Why it matters

Relevancy is the metric that most closely tracks how a real user experiences an answer. People forgive a system that says “I do not know” far more readily than one that returns a confident wall of text that never answers the question. When you are tuning prompts, falling relevancy is an early signal that you over-constrained the model or stuffed the context with noise. In a Retrieval-Augmented Generation pipeline, it pairs with context-precision to separate a retrieval problem from a generation problem.

In practice

Most eval frameworks ship answer relevancy as a built-in scorer. In DeepEval it is one of the core RAG metrics, run over a dataset of question-answer pairs so you get a distribution rather than a single number. Promptfoo can assert a minimum relevancy threshold in CI so a prompt change that starts producing rambling answers fails the build before it ships. A practical workflow is to log production traffic, sample the lowest-scoring answers, and read them - the failure modes cluster quickly, and most trace back to either a vague system prompt or retrieved chunks that pulled the model off course. See how to evaluate RAG for where relevancy fits in a full scorecard.