How to Measure LLM Hallucination in 2026 - A Practical Guide

“Measure hallucination” sounds like one task. It is not. Hallucination is a bucket word for several different failure modes, and each one is measured differently. If you try to reduce it to a single number, you will build a metric that misses half the failures.

This is a practical guide to actually scoring it. I will break hallucination into the metrics that matter, show how each is measured, and point to the eval tools that ship these metrics so you are not writing judge prompts from a blank file.

Step 1: Break “hallucination” into scoreable metrics

The first move is to stop treating hallucination as monolithic. In practice it is at least three distinct things, and which ones you need depends on whether you are doing RAG or open-ended generation.

• Faithfulness - is the answer grounded in the context you retrieved? This is the core RAG hallucination check. A RAG answer should come from the retrieved documents, not the model’s memory.
• Answer relevancy - does the answer actually address the question, or is it fluent and off-topic?
• Factuality - is the claim true against a reference or ground truth, independent of any context?

A RAG answer can be perfectly faithful to bad retrieved context and still be factually wrong. That is why you rarely want just one of these. For retrieval systems, faithfulness catches ungrounded claims and answer relevancy catches off-topic ones. For open-ended generation, factuality against a reference is the workhorse.

DeepEval ships hallucination, answer relevancy and faithfulness as named metrics, along with RAG, conversational and agent metrics - part of its library of 50+ research-backed metrics. Starting from named, documented metrics beats inventing your own definitions.

Step 2: Pick a measurement method

Almost every hallucination metric in 2026 is implemented as LLM-as-judge - you give a model the output, the context or reference, and a scoring criterion, and it returns a score. DeepEval’s widely-cited G-Eval is exactly this: LLM-as-judge with custom criteria you define.

The alternative, where it applies, is reference-based scoring against a ground-truth answer - exact match or embedding similarity when you have a known-correct target. Braintrust ships these in its autoevals library, which includes exact match, embedding similarity and LLM-as-judge factuality as working scorers out of the box, plus custom scorers you write as plain functions.

Use reference-based scoring when you have ground truth, and LLM-as-judge when you do not. Most real hallucination measurement is the latter, because you rarely have a labeled correct answer for every production query - but a held-out labeled set is what keeps the judge honest, which is Step 5.

Step 3: Build a test set with known answers

You cannot measure hallucination without something to measure against. Before you score anything, assemble a dataset of inputs paired with either the correct answer or the context the answer should be grounded in.

This is unglamorous and it is the whole game. A hallucination metric with no reference is just vibes. All three tools here treat datasets as first-class:

• DeepEval has dataset management in the Confident AI cloud, and its OSS framework runs test cases directly.
• Braintrust has dataset management wired into its eval and experiment system.
• Arize Phoenix ships datasets and experiments alongside its eval metrics.

Start small and real. A few dozen carefully labeled examples that reflect your actual failure modes beat thousands of synthetic ones.

Step 4: Score, especially for RAG

For RAG systems, this is where the strongest tooling matters, because RAG is where faithfulness lives.

Arize Phoenix ships 50+ pre-built eval metrics, and reviewers single out its RAG evaluation as the best available - you get serious retrieval and answer scoring without writing your own judge prompts. If your hallucination problem is a RAG pipeline producing ungrounded answers, this is the eval library I reach for first. It is also OpenTelemetry-native and runs locally in under a minute, so you can score against your own traces fast.

For a Python team that wants hallucination checks in a test suite, DeepEval is the natural shape - it is pytest for LLM apps. You write test cases, assert on the hallucination or faithfulness metric, and run deepeval test run in CI. The OSS framework is Apache-2.0 and free, so you can score locally with no account.

For teams that want scoring wired to regression gates, Braintrust puts the factuality scorer in the same system as its CI/CD quality gates - which is Step 6.

Step 5: Calibrate the judge before you trust it

Here is the honest caveat everyone should sit with. Nearly all these metrics are one AI judging another, and reviewers raise the reliability question fairly. The judge has its own error rate.

The mitigation is calibration. Take a labeled subset - examples where a human decided hallucination or not - and check how often the judge agrees. That agreement rate is your confidence level. If the judge agrees with humans 90% of the time, you can lean on it. If it agrees 60% of the time, your metric is barely better than a coin flip and you need a better judge prompt or a stronger judge model.

Do not ship a hallucination metric you have not checked against human labels. An uncalibrated judge produces confident numbers that mean nothing, which is worse than no metric at all.

Step 6: Gate on it in CI

Measuring hallucination once tells you where you are. Gating on it stops regressions from shipping.

Braintrust is the most turnkey here - its CI/CD quality gates can block a merge when a change causes a statistically significant regression, not just log that quality dropped. So a prompt change that quietly raises your hallucination rate gets stopped at the PR instead of discovered in production. DeepEval fits the same job through its pytest-style runner in CI, where a failing hallucination assertion fails the build.

A hallucination metric that only lives in a dashboard is a metric you will ignore. Wire it to the merge gate and it starts protecting you.

Two costs to plan for

Before you scale a hallucination suite, budget for two things.

API cost. Nearly every metric fires another LLM inference, so a large suite compounds into significant API bills and runs take minutes, not seconds - DeepEval’s own docs note CI needs explicit timeouts. Cost-model the judge calls the way you model the app’s own inference.

Billing shape. If you use Braintrust’s cloud, its meter counts “processed data” in GB - every byte of inputs, outputs, prompts and metadata - with no hard spending cap, and verbose RAG contexts are exactly what burn it. Set billing alerts on day one. On DeepEval’s Confident AI cloud, watch the pricing cliff - Starter is $200/mo and the next rung, Team, jumps 10x to $2,000/mo with nothing in between.

Bottom line

Do not measure “hallucination.” Measure faithfulness, answer relevancy and factuality, each with a method that fits - LLM-as-judge where you have no ground truth, reference-based scoring where you do. Build a labeled test set, calibrate the judge against human labels before you trust it, and gate on the metric in CI. Reach for Phoenix when RAG faithfulness is the problem, DeepEval when you want it in a Python test suite, and Braintrust when you want it wired to a merge gate. And cost-model the judge calls, because measuring hallucination well is not free.