LLM Evaluation Metrics Explained - A Practical 2026 Guide

If you have ever shipped an LLM feature and then wondered whether the new prompt actually made it better, you have hit the core problem evaluation metrics solve. You cannot eyeball a thousand responses. You need a number.

This guide explains what those numbers are, how the main families of metric differ, and which one to reach for first. I will name tools where they concretely fit, but the metrics come first - the tool is just what runs them.

What an LLM evaluation metric actually is

An evaluation metric is a function that takes a model output and returns a score. That is the whole idea. The input might be the response alone, the response plus the question, or the response plus a known-good answer. The output is a number - usually 0 to 1 - that says how good the response was on one specific axis.

The word “one” matters. No single metric tells you if an answer is good. Relevancy, factual accuracy, tone and safety are different questions, and you score them separately. A response can be perfectly relevant and completely made up. You measure quality as a set of scores, not one grade.

The two families that matter most

Almost every metric falls into one of two camps, and knowing which is which saves you a lot of confusion.

Statistical metrics compare text with math. Exact match, embedding similarity, BLEU and ROUGE overlap. They are fast, cheap, deterministic and repeatable - run them a hundred times, get the same answer. The weakness is that they judge surface form, not meaning. “The capital is Paris” and “Paris is the capital” can score low on word overlap while meaning the same thing.

Model-based metrics use an LLM to judge the output. This is LLM-as-judge - you hand another model the response and a rubric, and it scores it. This captures nuance a string comparison never will. The trade-off is real: DeepEval notes that nearly all of its metrics are LLM-as-judge, which means every test case fires another inference. Big suites rack up API bills and runs take minutes, not seconds. There is also the honest reliability question of using one AI to judge another. It is powerful, but you cost-model it before you scale.

For a deeper look at the judge approach, we wrote a full LLM-as-a-judge guide.

The metrics you will meet first

Whatever tool you pick, the same handful of names show up. Here is what each one means.

• Answer relevancy. Does the response actually address the question asked. Reference-free, so it works in production.
• Faithfulness / hallucination. Is the answer grounded in the context you gave the model, or did it invent facts. This is the single most-requested metric for RAG apps.
• Correctness. Does the answer match a known-good reference. Reference-based, so it needs a test set with ground-truth answers.
• G-Eval. A widely-cited LLM-as-judge metric where you write custom criteria in plain language and the judge scores against them. DeepEval ships it as one of its 50+ research-backed metrics.
• Task-specific scores. RAG retrieval quality, tool-call accuracy for agents, toxicity and bias checks. These get their own sections in most eval libraries.

The rule of thumb: reference-based metrics for CI, reference-free metrics for production. In a test suite you have known answers to compare against. On live traffic you do not, so you score relevancy and faithfulness on the output’s own terms.

Where the tools fit

You rarely write these metrics from scratch. Three platforms ship them, and they suit different workflows.

If your team thinks in tests, DeepEval. It is the closest thing to pytest for LLM apps - you write test cases, assert on a metric, and run deepeval test run in CI. It ships 50+ research-backed metrics including G-Eval, hallucination, answer relevancy and faithfulness, plus RAG, conversational and agent metrics. The open-source framework is Apache-2.0 and free. The catch is the LLM-as-judge cost and the cloud’s pricing cliff - Starter jumps from $200/mo straight to $2,000/mo Team. Live in the free framework as long as you can.

If you want the fastest start and the best RAG eval, Arize Phoenix. It runs locally in under a minute, ships 50+ pre-built eval metrics, and reviewers single out its RAG evaluation as the best in the category - serious retrieval and answer scoring with no judge prompts to write. It is OpenTelemetry-native too. The one gotcha to know: the server repo is Elastic License 2.0, source-available rather than OSI open source, which only bites if you plan to resell it as a service.

If regression testing is the point, Braintrust. Its autoevals library hands you working scorers out of the box - exact match, embedding similarity, LLM-as-judge factuality - and you drop in custom scorers as plain functions. The real draw is that its CI/CD quality gates can block a merge on a statistically significant regression, not just log that quality dropped. Watch the processed-data billing meter, which counts every byte with no hard cap.

So which metric do you start with?

• Building RAG? Start with faithfulness and answer relevancy, then add retrieval quality. Arize Phoenix has the deepest RAG metrics - and we cover the wider field in best RAG evaluation tools.
• Writing Python tests? G-Eval plus correctness in DeepEval, run in CI.
• Guarding against regressions? Pick two or three metrics that matter, wire them into Braintrust quality gates, and block the merge when they drop.
• Not sure yet? Answer relevancy and hallucination cover most first questions. Add the rest as you learn what breaks.

The mistake I see most is teams chasing a perfect single score. There is not one. Pick the two or three axes that matter for your app, score them honestly, and treat the LLM-as-judge cost as a real line item. For the full field of frameworks that run these metrics, see best LLM eval frameworks, and for the wider concept, what are LLM evals.