Core Evaluation Metrics

What makes a metric worth tracking

A metric earns its place on your dashboard only if a change in it means something changed for a user. That sounds obvious, yet most eval dashboards are cluttered with numbers that move for reasons nobody can act on. The metrics in this chapter survive that test - each one maps to a concrete way an LLM application fails the person using it. Learn these five well and you can evaluate almost any text-generation system.

Before the specifics, one framing that will save you grief. Metrics split into two layers. Component metrics score one property of one output - is this answer faithful, is it relevant. Task metrics score whether the whole interaction achieved the user’s goal. You need both. A response can be individually faithful and relevant and still fail the task, and only a task metric catches that.

Answer relevancy

Answer relevancy measures whether the output actually addresses the input. A model can produce a fluent, accurate paragraph that answers a question nobody asked. Relevancy catches that.

The common implementation is clever. You take the generated answer and ask a model to reverse-engineer the questions that answer would satisfy. Then you measure the semantic similarity between those reconstructed questions and the user’s real question. High overlap means the answer stayed on target. This is reference-free, so it works in production where you have no gold answer. DeepEval ships this as its AnswerRelevancyMetric and it is one of the most useful signals you can add on day one.

Faithfulness and hallucination

Faithfulness measures whether the claims in an output are supported by the context the model was given. It is the single most important metric for any system that grounds answers in documents, because the failure it catches - confident fabrication - is the one that destroys user trust fastest.

The mechanism is claim decomposition. A judge breaks the output into individual factual claims, then checks each one against the provided context. Faithfulness is the fraction of claims that are supported. Hallucination rate is essentially its inverse, computed against the source material rather than the retrieved context. If a summary makes eight claims and one is unsupported, faithfulness is 0.875.

The distinction that trips people up - faithfulness is about grounding, not truth. A claim can be factually correct in the real world and still count as unfaithful if the provided context did not support it, because a grounded system is supposed to stick to its sources. How to measure LLM hallucination goes deeper on the measurement mechanics.

Correctness

Correctness compares the output to a known-good reference answer. This is the closest thing to traditional testing, and you use it wherever a right answer exists - factual QA, structured extraction, math, code. Unlike exact string matching, LLM-graded correctness accepts a paraphrase that conveys the same information.

Correctness is powerful precisely because it is strict, but it costs you the effort of writing references. Reserve it for the parts of your system where wrongness is unambiguous and dangerous. For open-ended generation, reference-free metrics serve you better.

Task success rate

Task success is the metric that separates people who evaluate outputs from people who evaluate applications. It asks a binary question - did the user accomplish what they came for. For an agent, that might mean the correct tool was called and the transaction completed. For a support bot, it might mean the issue was resolved without escalation.

Task success is usually judged by an LLM against a checklist of what a successful outcome requires, or measured from downstream signals like whether the user reformulated their question or gave up. It is noisier than component metrics and harder to compute, which is exactly why so few teams track it and why it is often the number that actually predicts churn.

Safety and toxicity

The last core metric is not about quality, it is about harm. Safety metrics score whether an output contains toxic language, leaks private data, or violates a policy. Even a faithful, relevant, correct answer is a failure if it is also abusive or exposes a secret. These are typically run as classifier-based checks or narrow LLM judges, and in regulated domains they are non-negotiable gates rather than dashboard numbers.

Putting them together

Here is how a modest but honest metric set looks for a document-grounded assistant, expressed as DeepEval-style test cases:

from deepeval.metrics import (
    AnswerRelevancyMetric,
    FaithfulnessMetric,
)
from deepeval.test_case import LLMTestCase

test_case = LLMTestCase(
    input="What is our refund window?",
    actual_output=model_answer,
    retrieval_context=retrieved_docs,
)

metrics = [
    AnswerRelevancyMetric(threshold=0.7),
    FaithfulnessMetric(threshold=0.8),
]

Two component metrics, explicit thresholds, run in CI. Add a task-success judge and a safety check and you have a set that catches the failures that matter without drowning you in noise.

A note on thresholds and aggregation

A metric is useless without a threshold, and a threshold is meaningless without a baseline. Never pick 0.8 because it looks reasonable. Run your current system, record where it lands, then set the gate slightly below that and tighten over time. And be careful how you aggregate - a mean of 0.85 can hide a cluster of catastrophic 0.2 cases. Always look at the distribution and the worst cases, not just the average. The LLM evaluation metrics explained post has the full formulas, and how to evaluate LLM applications shows how these fit into a real pipeline.

The tools handle the computation. DeepEval offers fifty-plus research-backed metrics out of the box. Promptfoo lets you assert on them from YAML. Braintrust tracks them across experiments so you can see a metric regress between versions, and Langfuse attaches them to live production traces.

Key takeaways

• Track a metric only if a move in it maps to real user harm.
• Answer relevancy catches off-topic answers - reference-free, cheap, add it first.
• Faithfulness measures grounding in context, not real-world truth, and is the top metric for document systems.
• Correctness needs references and is worth the cost where wrongness is unambiguous.
• Task success rate is the number that actually predicts whether users succeed.
• Set thresholds against your own baseline and always inspect the distribution, not just the mean.

Next, in Evaluating RAG Systems, we apply these metrics to retrieval-augmented pipelines, where you have to grade the retriever and the generator separately.