Evaluation of LLM Applications: The 2026 Field Guide

The evaluation of LLM applications means measuring whether your system retrieves the right evidence, calls the right tools, answers correctly and finishes the task on realistic inputs. Scored the same way every run, on a dataset you own, often enough to catch regressions before your users do. Four scoring mechanisms do the work: deterministic code checks (regex, schema validation, exact match), model-based scorers (embedding similarity, NLI entailment), LLM-as-a-judge for criteria that need reasoning, and human review as the calibration signal. Production sampling is a fifth source of test cases rather than a scoring method.

Public benchmarks tell you which base model to start from. MMLU-Pro covers 12,032 questions across 14 domains; GPQA Diamond is 198 expert-written science questions. Neither says anything about whether your chunking strategy pulls the right paragraph out of your own contract archive. Confident AI makes the same point about Stanford CRFM’s HELM being effectively redundant for applications built on proprietary data (confident-ai.com).

Seven steps follow. Build the dataset and size it with real statistics, pick metrics that map to your failure modes, choose the cheapest scorer that can detect each one, evaluate RAG component by component, evaluate agents as trajectories, wire the result into CI as a quality gate, then close the loop with online scoring on sampled traffic.

What this page is. Everything comes from public documents: papers, vendor docs, changelogs, pricing pages, GitHub issues, practitioner threads. No benchmarking was performed for this article and no measured results are claimed. Where a question needs hands-on measurement, the article says so and tells you what to measure.

---

The four axes that decide your entire eval design

Four decisions set what you collect, what it costs, and what your scores can support.

Axis 1, offline versus online. Offline evaluation runs against a curated dataset before merge, like a test suite. Online evaluation scores sampled production traffic asynchronously. Braintrust argues for both rather than either (braintrust.dev), and that is correct for a structural reason: a fixed dataset cannot contain the query a user will invent next Tuesday.

Axis 2, component versus end-to-end. Score retrieval separately from generation, tool selection separately from the final answer. End-to-end scores tell you that quality dropped. Only component scores tell you where.

Axis 3, model versus product. Fine-tuning that lifts your target task can degrade unrelated behaviour, so a model that scores better on your headline metric may be worse for everyone who asks about something else.

Axis 4, reference-based versus reference-free. This sets your dataset cost more than any other choice. Reference-based scoring (BLEU, ROUGE, exact match, correctness against a gold answer) needs labelled ground truth on every row. Reference-free scoring (faithfulness against retrieved context, toxicity, JSON validity) does not. Labelling is the expensive part.

Your situation	Offline or online	Component or end-to-end	Reference-based?
Pre-launch RAG chatbot, no traffic yet	Offline only	Both, retrieval first	Yes for a small golden core, reference-free for the rest
Live app, frequent prompt edits	Both, offline gates the PR	End-to-end in CI, component on failure	Reference-free dominates
Multi-step agent with tool calls	Both, trajectory logged online	Component, per tool call, mandatory	Mostly reference-free, task completion is reference-based
Regulated domain, audit requirement	Both, with retained rationales	Both	Yes, you need a defensible expected answer
Choosing a base model	Offline, public benchmarks	End-to-end	Yes, benchmarks ship with references

Annotate each arrow with an owner and a cadence: per-PR, nightly, continuous, weekly triage. Stages composed from MLflow, Braintrust and Datadog documentation.

---

Step 1. Build the dataset and size it with actual math

Three sources feed a working set. Public benchmarks give a directional baseline for model selection. Human-annotated golden sets carry anything needing domain expertise, the cases where a lawyer, clinician or support lead has to say what “correct” means. Synthetic silver sets scale coverage cheaply.

The June 2025 arXiv survey on evaluating LLM-based applications (arxiv.org/html/2506.13023v1) describes bootstrapping silver data from a golden seed, then promoting rows into the golden set as they are reviewed. That promotion path matters. Silver data nobody reviews measures your generator, not your application. Our walkthrough of synthetic eval data covers the seeding step.

The 5 D’s

• Defined. The scope is written down, including what it deliberately excludes.
• Demonstrative. The distribution reflects production usage rather than what was easy to write.
• Diverse. Inputs vary in length, phrasing, difficulty and intent.
• Decontaminated. No overlap with model training data.
• Dynamic. The set changes as the product and its failure modes change.

How big does the set need to be?

Every vendor guide skips this. You can compute it. Treat a metric score as a proportion:

n = z² × m̂(1 − m̂) / ε²

where z is the normal quantile for your confidence level (1.96 at 95%, 1.645 at 90%), m̂ is your expected score, and ε is your target margin of error.

Check the arithmetic on one cell. At 95% confidence, an expected score of 0.80 and a ±5 point margin: n = 1.96² × 0.8 × 0.2 / 0.05² = 3.8416 × 0.16 / 0.0025 ≈ 246 samples. That figure is the worked example in the arXiv survey. Relax to 90% confidence and the same margin costs 173 rows, which is often the cheapest 30% you will ever save.

Expected score	±10 pts	±5 pts	±3 pts	±1 pt
0.60	93	369	1,025	9,220
0.80	62	246	683	6,147
0.95	19	73	203	1,825

Formula n = z²m̂(1−m̂)/ε², z = 1.96. Source: https://arxiv.org/html/2506.13023v1. Computed, not measured.

Read the curve, not the cells. Going from ±5 points to ±1 point multiplies your labelling budget by twenty-five. Tightening only as far as ±3 points costs 683 rows instead of 246, which most teams can afford.

Reconciling this with “start with 25 to 50 cases”

Braintrust recommends starting with 25 to 50 test cases drawn from production traffic, then expanding as failure patterns emerge. That is right as practice and wrong as measurement.

A 40-case set is a smoke test. It catches the prompt change that broke JSON output, the retriever that returns nothing, the model swap that made the assistant answer in Portuguese. Those show up as 40-point swings.

What 40 cases cannot do is tell an 82% system from a 78% one. At n = 40 and m̂ ≈ 0.8 the margin of error is roughly ±12.4 points. A four-point improvement on 40 examples is noise wearing a suit.

Start at 40. Then write in the README what the number can support, and grow toward the n your target margin demands.

Coverage, decontamination and row metadata

Every set needs four categories: happy path, edge cases, adversarial inputs, and off-topic requests the system should decline. Datadog and Braintrust both point at the last one. Teams forget it until a support chatbot cheerfully answers a question about a competitor’s product.

Decontamination means three habits from the arXiv survey. Keep the set private, so never in a public repo and never pasted into a chat UI that retains data. Prefer cases collected after the model’s training cutoff. Check suspicious rows with continuation testing, where you give the model the first half of an item and see whether it reproduces the rest, plus perplexity inspection, where anomalously low perplexity on a supposedly novel item suggests memorisation.

Row metadata that pays for itself: tags for intent, language, difficulty and customer segment, so you can prove the set is not 90% one query type; grounding context for the judge, sourced independently of the system under test, because a judge grading faithfulness against the app’s own retrieval measures internal consistency; and expected information, meaning the specific terms, figures or steps the answer must contain. That last one is cheaper to author than a full gold answer and far more robust to paraphrase than exact match.

---

Step 2. Choose metrics that map to your actual failure modes

Every glossary organises metrics by family. That ordering is useless when you are choosing. Organise by what breaks.

Failure you have seen	Metric that detects it	Cheapest scorer that works
Hallucinated fact not in sources	Faithfulness, groundedness	NLI entailment, or LLM judge
Wrong document retrieved	Context precision and recall, precision@k, MRR, NDCG	Code plus embeddings
Right facts, wrong question answered	Answer relevance	LLM judge
Off-brand or wrong tone	Style adherence	LLM judge with a written rubric
Unsafe or policy-violating output	Toxicity, policy adherence	Classifier plus judge
Agent loops or picks the wrong tool	Trajectory metrics, tool-call accuracy	Code over spans
Malformed output breaking downstream code	Schema validity, JSON parse rate	Code
Too slow or too expensive to ship	p95 latency, cost per resolved task	Code over telemetry

Code is near free, embedding and NLI scorers are cheap, judges are expensive, humans cost most. Synthesised from taxonomies published by Braintrust, Datadog, MLflow and https://arxiv.org/html/2506.13023v1.

Task success. Completion rate, correctness against a reference, instruction following. Braintrust and MLflow both put these first, because they track most directly with whether users got what they came for.

Faithfulness, mechanically. Datadog spells out the procedure (datadoghq.com). Break the answer into atomic claims. Ask whether the retrieved context entails each one. Return the supported fraction as a score from 0 to 1. Decomposition is what makes it useful: an answer with six correct claims and one fabricated one scores 0.86 rather than “fail”, and you can see which claim broke. More in our guide to hallucination detection in RAG systems.

Retrieval. Needle-in-a-haystack testing plants a known statement at varying depths across varying context sizes, from 4k tokens up to whatever your model advertises.

Term-overlap metrics. BLEU, SacreBLEU, NIST, METEOR and the ROUGE family still earn a place in translation, extractive summarisation, and as cheap regression tripwires where output format is stable. They collapse on paraphrase, ignore structure and react to length. Use SacreBLEU rather than hand-rolled BLEU, because tokenisation differences make raw BLEU numbers non-comparable between papers.

Semantic scorers. Embedding cosine similarity, BERTScore, MoverScore and NLI entailment through a cross-encoder occupy the middle. More tolerant of paraphrase than n-grams, far cheaper and more reproducible than a judge call. Deterministic enough to run on every commit, which is where they belong.

Perplexity. A contamination and fluency signal, weakly linked to generation quality anyone cares about (databricks.com). Do not gate a release on it.

Operational metrics belong in the same dashboard. p50 and p95 latency, tokens per request, cost per resolved task, error rate. A reranker that lifts faithfulness two points and triples p95 latency is a regression, and only a joint dashboard shows you that.

Three to five metrics, at least one of them deterministic code, and never a single composite as the release gate. Composites hide compensating movements, with safety down, fluency up and the average unchanged.

---

Step 3. Pick the cheapest scorer that can detect the failure

The decision rule is an escalation ladder. Most teams start two rungs too high.

Use code if you can. If the criterion fits a regex, a JSON schema check, a length bound, a fuzzy match or a forbidden-terms check, write the function. Free, instant, and it returns the same answer forever. A surprising share of what teams pay a judge to assess is “did the model return valid JSON with these five keys”.

Use a model-based scorer when the criterion needs semantics but not reasoning. Embedding similarity for “is this roughly the same answer”. NLI entailment for “does the context support this claim”. Confident AI walks through the pattern with cross-encoder/nli-deberta-v3-large, taking a softmax over the entailment score and applying a 0.6 pass threshold. Copy the shape. That specific checkpoint is a 2023-era choice, so benchmark a current cross-encoder on your own labelled sample first.

Escalate to LLM-as-a-judge only for reasoning criteria. Tone. Helpfulness. Whether the response addresses what the user meant. Policy nuance where the rule has exceptions. G-Eval, the chain-of-thought judging method introduced by Liu et al. in March 2023 (arXiv:2303.16634), reported a Spearman correlation of 0.514 with human ratings on the SummEval summarisation task, ahead of BLEU, ROUGE, BERTScore and MoverScore on the same data.

Judge design rules worth enforcing in review:

• Low-precision scales. Binary, or 1 to 5. Never 1 to 100.
• Randomise ordering in any pairwise comparison, to counter position bias.
• Use more than one judge model, ideally from different families such as OpenAI, Anthropic and Google.
• Calibrate against a human-labelled gold set. Protocol below.
• Version the judge prompt in the same repo as the code it grades. See auditable prompt versioning.

Human review is not optional. It is the only signal telling you whether the judge measures anything at all. Budget it as a recurring line, a fixed sample every week reviewed by someone who knows the domain.

Thresholds. A threshold that fails every run and one that never fails are equally useless. Set it from the current production baseline plus a tolerance you argued about in advance, then version it with the code so a change shows up in a diff.

---

What the evaluation of LLM applications costs at production volume

No page currently ranking for this topic models this, which is odd, because it is the second question every buyer asks.

Cost per scored trace = (judge prompt tokens + context tokens + output tokens) × per-token price × number of judge metrics.

Multiply by traces scored, which is traffic times sampling rate. Add any platform or span-ingest fee.

Assumption	Typical range	Why it matters
Judge system prompt and rubric	300 to 800 tokens	Fixed per call, multiplied by metric count
Retrieved context passed to judge	0 tokens without RAG, 8,000+ with top-k=10	In RAG this is usually 80% or more of judge input
Application response length	100 to 600 tokens	Modest contributor
Judge output, score plus rationale	50 to 300 tokens	Priced at the higher output rate
Judge metrics per trace	1 to 5	Each is normally a separate call
Sampling rate	1% to 100%	Linear multiplier on everything

Put numbers through it. A faithfulness judge over top-k=10 retrieval sends roughly 8,800 input tokens and returns 200. Run four judge metrics on that trace, each re-sending the same context, and you are at 35,200 input tokens per trace. At 100,000 monthly traces scored in full, that is 3.52 billion input tokens a month for evaluation alone. Sample one trace in ten and it is 352 million.

Three levers dominate the bill, and only one is the model you pick.

1. Sampling rate. Scoring one trace in ten drops judge spend by 90% and costs almost no precision at volume. Use the sizing table: 10,000 scored rows is fifteen times the 683 needed for a ±3 point margin.
2. Metric count per trace. Four judge metrics means four calls, four rubrics, four sets of output tokens. Combining them into one call cuts cost and raises the chance the scores correlate with each other artificially.
3. Whether retrieved context reaches the judge. For faithfulness it must. For tone it must not. Sending 8,000 tokens of retrieved context to a tone judge is the most common way teams accidentally multiply their eval bill tenfold.

Built from each provider’s published per-million-token list prices on the date of writing, with the assumptions table beside it. Modelled from list prices, not measured.

The platform-fee layer. Datadog states that Agent Observability is free for Datadog customers submitting up to 40,000 LLM spans per month (datadoghq.com). Verify on the current pricing page and date-stamp it. The threshold anchors how span-volume tiers interact with token cost. An agent emitting 8 spans per request exhausts 40,000 spans at 5,000 requests a month. Span volume follows trace granularity rather than user count, so instrumenting more finely raises your bill without raising your traffic.

Self-hosted comparison. Running MLflow, Langfuse, Opik, Promptfoo or DeepEval yourself takes the platform fee to zero and leaves judge token cost untouched, because that goes to the model provider either way. What replaces the licence fee is an engineer who owns Postgres migrations, object storage for traces, and the pager when trace ingest backs up. For a team of five that is usually a worse trade. For a platform team running fifty apps under data-residency rules it is usually better.

The shape that almost always wins. Deterministic scorers on 100% of traffic, judge scoring on a stratified sample. Stratify by user segment, so your enterprise tier is over-sampled relative to its volume, and by failing-code-check, so every trace that already tripped a cheap scorer gets judged. Uniform random sampling spends judge calls on the boring middle.

---

Calibrating your judge against humans

Start with the most repeated number in this niche. Confident AI’s guide attributes to Databricks the claim that LLM-as-a-judge agrees with human grading on over 80% of judgments (confident-ai.com). There is no link and no date on that post, and the surrounding article is GPT-3.5 and GPT-4 era. Treat it as folklore until someone produces the original with a date.

The deeper problem is that raw agreement is the wrong statistic whatever its provenance. On a task where 85% of outputs are fine, a judge that returns “pass” unconditionally scores 85% agreement while carrying zero information. Class imbalance inflates agreement, and production data is imbalanced by design, because you already fixed the obvious failures.

Use Cohen’s kappa or balanced accuracy on a class-balanced sample. Kappa corrects for agreement expected by chance. On the Landis and Koch scale from 1977, still the common reference, 0.41 to 0.60 is moderate and 0.61 to 0.80 is substantial. Ship at 0.6 and above, investigate below it.

A 100-row calibration protocol you can run in an afternoon

1. Pull 100 production traces, over-sampling failures so you land near 50/50 rather than the natural 85/15.
2. Have a domain expert label each one against the same written rubric the judge receives. Not a looser one. The same one.
3. Run the judge over the identical 100 rows.
4. Build the 2×2 confusion matrix, then compute Cohen’s kappa and balanced accuracy.
5. Commit the number, the date, the judge model version and the rubric hash as your calibration baseline.
6. Read the disagreements.

Step 6 is where the value sits. A judge that fails only on rows where two humans also disagree is telling you the rubric is ambiguous, not that the model is weak. Track inter-annotator agreement on the human side too, with a second labeller on 20 of the 100 rows, or you have no idea what ceiling the judge is being held to.

Which significance test for which scale

When you compare two systems, or a judge against humans, the rating scale picks the test. The arXiv survey supplies the mapping and nobody else on this topic does.

Rating scale	Test	Why
Binary, pass or fail	McNemar’s test	Compares paired discordant outcomes on the same items
Continuous 0 to 1	Two-tailed paired t-test	Paired differences on an interval scale
Likert 1 to 5	Wilcoxon signed-rank	Ordinal data, equal interval spacing is not guaranteed

The Likert row is the one teams get wrong. Averaging 1 to 5 ratings and running a t-test assumes the gap between 4 and 5 equals the gap between 1 and 2. It does not.

Error bars. Anthropic’s “Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations”, published November 2024 (arXiv:2411.00640), is the reference for reporting confidence intervals on eval scores, including the clustered-standard-error correction for datasets with repeated questions per document. A score reported without an interval is a rumour with a decimal point.

Non-determinism. For regression cases that must reproduce, fix temperature to 0. For inherently variable outputs, run k times, aggregate, and report the interval. Braintrust calls the alternative “flaky evaluations”. A test that fails one run in five is a coin.

Judge bias	What it looks like	Mitigation
Position bias	Whichever response is shown first wins more often	Randomise order, run both orderings and average
Verbosity bias	Longer answers score higher regardless of quality	Length-controlled comparisons, penalise padding in the rubric
Self-model bias	A judge prefers outputs from its own model family	Judge with a different family than the system under test

Databricks adds the point underneath all three: a judge inherits the blind spots of the model it runs on. If your application and your judge share a model, a shared misconception is invisible to your entire measurement system.

---

Step 4. Evaluating RAG pipelines component by component

Score retrieval before generation. Context relevance, context precision and context recall answer the prior question: did the pipeline fetch the evidence at all? A faithfulness score of 1.0 over irrelevant context is a system confidently grounding itself in the wrong document. Our RAG evaluation metrics reference works through each formula.

Needle-in-a-haystack, as a procedure (datadoghq.com):

1. Embed a specific, verifiable statement into the vector store.
2. Prompt the application for that fact, constraining it to answer only from provided context.
3. Check the response for a semantic match against the planted statement.
4. Repeat across insertion depths of 10%, 25%, 50%, 75% and 90%, at several context sizes.

The depth sweep is the point. Retrieval and attention both degrade unevenly across position, and a system that finds the needle at 10% depth and loses it at 60% has a specific, fixable problem.

Generation stage. Faithfulness against the retrieved chunks, plus answer relevance against the original question. Both, because they fail independently. A response can be flawlessly grounded in the retrieved text and still not answer what was asked.

Hyperparameter sweeps. Once the pipeline is scored you can sweep chunk size, top-k, embedding model, reranker on or off, and prompt template. This is the highest-return use of an eval set and also how overfitting starts. Sweep 20 configurations against 50 cases and the winner is partly noise. Hold out a slice you never optimise against and confirm the winner there.

Two open-source faithfulness implementations you can adopt without buying a platform: Ragas and DeepEval. Read both. They decompose claims differently and the scores are not interchangeable.

---

Step 5. Evaluating agents as trajectories

The unit of evaluation changes. For a single-turn app it is an input and output pair. For an agent built on LangChain, LlamaIndex or your own loop it is the trajectory: every tool call, every argument, every retry, ending in a final answer that may be right for entirely the wrong reasons.

Instrument tool-selection accuracy, argument-construction quality, task completion rate, step efficiency against the minimum viable path, error recovery after a failed call, and loop detection on repeated identical calls with no state change.

MLflow enumerates the matching failure modes: infinite loops, wrong tool selection, incomplete goals, inefficient paths (mlflow.org/llm-evaluation). The arXiv survey flags the hardest one, compounding error across multi-turn workflows, where a small mistake at step two becomes a confidently wrong answer at step nine.

Tracing is a prerequisite. Without spans you can see the score dropped and have no idea which step caused it. Tag traces with their eval scores so a failing case links straight to the offending span. We go deeper in agent observability from first principles.

Session-level scoring is the emerging gap. Braintrust’s changelog records adding “Group scope” for online scoring, so related multi-turn traces can be evaluated as one unit keyed on a session identifier (braintrust.dev/changelog). That is a vendor changelog entry, and a dated admission that per-trace scoring was not enough for conversational systems. Scoring turns in isolation misses the agent that answered every turn adequately and never resolved the issue.

Portability. Instrument with the OpenTelemetry GenAI semantic conventions wherever your platform supports them, so traces are not trapped in one vendor’s schema. Support varies by tool and moves quickly. Check it before you commit, because it decides observability vendor choice for a lot of teams and it is absent from every guide currently ranking.

---

Step 6. Wiring evals into CI/CD as quality gates

Treat evals as tests. Pytest-style assertions against the golden set, on every PR touching a prompt, a retriever, a model version or a tool definition. DeepEval’s assert_test and LLMTestCase pattern is the canonical open-source example. Construct a case with input, actual output and retrieval context, attach metrics with thresholds, assert.

What the CI report must show. Which cases improved, which regressed, by how much, against which baseline commit. A single pass/fail bit gets ignored within two sprints, because when it goes red nobody can tell whether one case flipped or forty did.

Tier	What runs	When	Cost
Fast gate	Deterministic scorers on the full golden set	Every PR	Seconds, near zero
Deep gate	Judge scorers on a subset, or the full set	Nightly and pre-release	Minutes, real token cost

Release criteria, written in advance. Minimum scores on named metrics, a maximum tolerated regression against current production, all safety checks passing. Agreed before release day rather than argued at 6pm during it.

Version the eval dataset with the code, in a lineage kept separate from any training data. The moment eval data flows into fine-tuning, every score becomes a measure of memorisation.

---

Step 7. Online evaluation and closing the production loop

Log prompts, responses, retrieved context, tool calls and metadata such as user feedback and session ID. Score asynchronously on a sample, emit scores as metrics, dashboard and alert.

Proxies that need no ground truth, because nobody labels production traffic in real time: topic relevancy as a binary domain-boundary check, negative-sentiment rate in user replies, refusal rate, failure-to-answer rate, retry rate, thumbs-down rate.

Alert on rate of change. A steady 3% negative-sentiment rate is background. A jump from 3% to 7% concentrated in one query cluster is the signal. Thresholds on absolute values either fire constantly or never.

The triage rule that makes the system compound. Reproduce the failure, label it, add it to the golden set, fix it, confirm the new case passes. Every production failure becomes a permanent test the same day. Every vendor page gestures at this; the rule is the part that matters, because without a named owner and a same-day SLA it does not happen.

Drift you can only see online. Prompt drift, where accumulated small edits move behaviour in a direction nobody chose. Model drift, where a provider updates a model behind a stable alias and your baseline shifts underneath you. Pin model versions explicitly and re-run the full golden set on every provider release.

Alert fatigue is a real failure mode. Confident AI’s changelog documents shipping alert priority tiers, critical through info, with per-integration filtering (confident-ai.com/docs/changelog). Route the five-alarm fire to Slack and leave the informational score wobble in a dashboard nobody has to acknowledge.

---

Safety, security and red-teaming

Pre-production safety evaluation means adversarial suites that actively try to elicit biased, toxic or policy-violating output, plus attempts to make an agent take actions it should not be authorised to take. The second category grows with every tool you connect.

Toxicity scoring. Word-list matching is unmaintainable and misses the polite insult, the coded slur, the accurate-but-cruel summary. Use a tuned classifier such as Meta’s Llama Guard, or a judge given an explicit written definition and a 1 to 5 scale. The written definition does most of the work.

Prompt injection and jailbreaking belong in the eval suite, not in a security review that happens twice a year. Map your cases to the OWASP Top 10 for LLM Applications so coverage gaps are visible, and run JailbreakEval and NVIDIA Garak on a schedule, both catalogued in the bibliography at alopatenko.github.io/LLMEvaluation. Treat every new successful attack as a golden-set case.

Evals are not guardrails. Evals are measurement, run offline or on a sample. Guardrails are runtime enforcement on every request: Llama Guard, NVIDIA NeMo Guardrails, Guardrails AI. Teams that conflate the two either block traffic with a judge on the critical path, or ship a measurement system with no enforcement behind it. You need both, wired separately.

Red-teaming has moved into the eval tooling itself. Confident AI’s changelog records red-teaming picking up code vulnerability scanning, and organisations gaining the ability to restrict model providers to an approved set such as Amazon Bedrock only. Compliance controls became table stakes for regulated buyers.

Audit trail. In regulated industries, evaluation results, judge rationales and dataset versions must be retained and queryable. Judge rationales especially. A score without a reason is not defensible to an auditor asking why the system shipped.

---

Governance frameworks that shape your eval requirements

Three documents now set the floor for regulated buyers, and they change what your eval suite has to store rather than what it has to measure.

The EU AI Act entered into force in August 2024, with obligations for general-purpose AI models applying from 2 August 2025 and high-risk system obligations phasing in through 2026 and 2027. It requires documented testing and risk management for high-risk uses. NIST’s AI Risk Management Framework 1.0, published January 2023, organises the same work under Govern, Map, Measure and Manage, and the Measure function is exactly your eval pipeline. ISO/IEC 42001:2023 is the certifiable management-system standard auditors increasingly ask about.

The practical consequence is retention. Dataset version, judge model version, rubric text, rationales, dates. If your eval platform cannot export all five for an arbitrary past release, it will not survive procurement at a bank.

---

The pitfalls that make good scores meaningless

Braintrust’s pitfalls table is the strongest single asset on this topic anywhere. Here is that coverage, extended with four failure modes nobody lists.

Pitfall	Cause	Fix
Overfitting to the eval set	Optimising against the only set you have	Hold out a slice you never tune against
Judge bias	Position, verbosity, self-model preference	Randomise order, multiple judge families, calibrate
Data leakage	Eval items present in training data	Private sets, post-cutoff collection, continuation testing
Metric gaming	Team optimises the number, not the outcome	Tie one metric to a user-observable result
Ignoring edge cases	Set built from convenient examples	Mandate adversarial and off-topic categories
Vibes-based evaluation	No dataset, no scores, “it feels better”	Any written-down score beats none
Underpowered eval sets	40 examples, four-point “improvement” reported	Compute the margin of error, see the sizing table
The metric that never fails	Threshold too loose, or the dataset lost its hard cases	Rotate it out or make it harder
Judge drift	Judge sits behind a moving model alias	Re-calibrate quarterly and after any judge-model change
Evaluating only what is easy	Format and toxicity are cheap, “did it solve the problem” is not	Budget for the expensive metric deliberately
English-only coverage	Test set written by the team, in one language	Tag rows by language, hold a per-language minimum

Two deserve expansion.

Underpowered sets are the most common and the most invisible. A four-point gain on 40 examples sits well inside the ±12.4 point noise band and reaches leadership as progress.

Evaluating only what is easy to score is why dashboards go green while retention falls. Format compliance is cheap to measure. Toxicity is cheap. “Did this solve the user’s problem” needs human review or a well-calibrated judge over a task-completion rubric, and its absence is the gap between your eval suite and your churn number.

Databricks makes an admission worth echoing. Correct answers that differ from the reference get marked wrong routinely. If your correctness metric is exact match against a gold answer, some share of your recorded failures are the model being right in unexpected words.

---

Tools for the evaluation of LLM applications, compared

First, the distinction readers keep conflating and no vendor draws.

Frameworks are scorers plus a harness. You run them anywhere: locally, in CI, in a notebook. They produce numbers and leave storage to you. DeepEval, Ragas, Promptfoo, TruLens, Giskard, OpenAI Evals and EleutherAI’s LM Evaluation Harness sit here.

Platforms add hosted storage, a UI, trace search, online scoring, annotation queues and alerting. Braintrust, Langfuse, Opik, Arize Phoenix, LangSmith, Weights & Biases Weave, Galileo, Patronus AI, Datadog Agent Observability, Databricks Agent Evaluation, Google Vertex AI’s evaluation service and Azure AI Foundry evaluations sit here. MLflow spans both.

Most teams end up with one of each, and several frameworks ship with a platform attached: DeepEval with Confident AI, Ragas with its own app, Opik with Comet.

Rows: MLflow, DeepEval, Ragas, Braintrust, Langfuse, Opik, Arize Phoenix, TruLens, Promptfoo, Helicone, LangSmith, Datadog Agent Observability, Databricks Agent Evaluation, OpenAI Evals, EleutherAI LM Evaluation Harness. Every cell filled from that tool’s own docs, LICENSE file, repo or pricing page, with the retrieval date footnoted.

On vendor self-claims. MLflow states it is “the largest open-source AI engineering platform… with over 30 million monthly downloads” (mlflow.org/llm-evaluation). That is a vendor claim about itself, checkable against public PyPI statistics, and raw download counts include CI runs and mirrors. MLflow’s page also carries an “Open Source vs. Proprietary Evaluation Tools” section, which is a sales argument by an interested party dressed as neutral analysis. The honest version: self-hosting removes per-seat fees and data-residency exposure. It removes neither judge token cost, nor storage cost, nor the engineer who owns the pipeline at 2am.

Choose by situation:

• One team, one RAG app. Ragas or DeepEval for scorers, Langfuse or Phoenix for traces. Optimise for how little the tool asks of you.
• Platform team standardising twenty apps. Dataset management, per-project access control and a common trace schema matter more than clever metrics. MLflow or a hosted platform, with OpenTelemetry GenAI conventions enforced from day one.
• Regulated industry with residency constraints. Self-host, and read the LICENSE file yourself rather than trusting an “open source” badge. Audit trail and retained judge rationales are hard requirements.
• Already running Datadog or Databricks. The integrated option is usually right despite looking weaker on paper, because the trace data is already there and a second observability stack has a real operating cost.

Checking a tool’s real failure modes before you adopt it

Spend an hour in the public trackers of mlflow/mlflow, confident-ai/deepeval, explodinggradients/ragas, comet-ml/opik, langfuse/langfuse and promptfoo/promptfoo. Filter to open bugs from the last twelve months, sort by reaction count, and group what you find under the six categories buyers weigh: judge non-determinism across runs, token blowups from context-heavy judge calls, online-scoring reliability at volume, span-schema breakage after SDK upgrades, self-hosted storage and migration issues, and dataset import and versioning limits. Record each as repo#NNNN, symptom, open since date, N reactions, URL. Then cross-reference the vendor changelog, because a fix that landed last month counts as much as the bug.

Built from a live search of the named issue trackers. Every row carries a working URL and the header carries the search date, because issue states decay within weeks.

---

What practitioners keep rebuilding, and why

Ten vendor pages rank for this query. Zero quote a practitioner. The practitioner record is public.

Count the open-source eval tool launches on Hacker News: Opik (41567192), UpTrain (37222930), Composo (38880862), Helicone (42806254), Pi Co-pilot (44061414), Burr (39917364), Spec27 (47959984). Seven teams looked at the existing tooling and concluded it did not solve their problem well enough to use. That is the strongest available evidence that this category is unsettled, and it belongs in your model of the market when a vendor tells you the space is mature.

The structural difficulty behind all seven launches is that no unique ground truth exists. In traditional ML a label is a label. Here, two correct answers can share no vocabulary, which is why so much eval engineering is really rubric engineering.

Adoption tracks how little the tool asks. A framework that runs inside your existing pytest suite gets used. A platform demanding a new vocabulary of experiments, spans, scorers and projects before the first number appears gets a pilot, then silence. Weigh that when you shortlist.

One practitioner framing worth carrying into metric design: hallucination detection splits into two different problems. Verifying a claim against world knowledge needs search and is unbounded. Verifying groundedness against the context you supplied is bounded and tractable (news.ycombinator.com/item?id=39454961). Only the second is solvable inside your own application.

And treat eval infrastructure as production infrastructure. It holds your prompts, your retrieved documents and your customer conversations in one searchable place, under the same access review as your data warehouse.

---

Benchmarks and leaderboards, when they matter and when they mislead

Legitimate uses, in full: shortlisting a base model, tracking where the capability frontier sits, sanity-checking a fine-tune against its own baseline. That is the list.

Names worth knowing. MMLU (57 subjects) and MMLU-Pro for broad knowledge, GPQA Diamond (198 questions) for graduate-level science, SWE-bench Verified (500 human-validated tasks) for software engineering, ARC-AGI-2 and Humanity’s Last Exam (roughly 2,500 questions) for frontier reasoning, τ2-bench for conversational agents, FACTS Grounding for groundedness, Vectara’s hallucination leaderboard for summarisation faithfulness, MTEB for embeddings, and LMSYS Chatbot Arena, Arena-Hard and AlpacaEval for human-preference ranking. Older names still quoted include HellaSwag, TruthfulQA, HumanEval and BIG-bench. All are catalogued at alopatenko.github.io/LLMEvaluation.

Rank does not transfer to your application, for three reasons. Contamination, where benchmark items leak into training corpora. Distributional assumptions baked into how the benchmark was built. Optimisation pressure applied to any public target. Read “The Leaderboard Illusion” (April 2025) and “When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboards” (February 2024) before citing a leaderboard as evidence.

For the Hugging Face question. Hugging Face publishes an LLM Evaluation Guidebook and the evaluate library, while EleutherAI’s LM Evaluation Harness and OpenAI Evals are the two most-used harnesses. All three evaluate models and need wrapping to evaluate your application, because they expect a model endpoint rather than a retriever plus three tools.

Benchmarks choose your starting model. Your golden set decides whether you ship.

---

A 30-day rollout plan for the evaluation of LLM applications

Week 1, instrument. Add tracing. Log prompts, responses, retrieved context, tool calls and metadata. Nothing else. Every uninstrumented day throws test cases away.

Week 2, 40 cases and two code scorers. Write 40 test cases by hand from real logs across the four categories. Add two deterministic scorers, where schema validity plus a required-terms check is a fine start, and wire them into CI. Note in the README that 40 cases cannot resolve differences smaller than roughly ±12 points.

Week 3, one judge metric, calibrated. Add a single judge metric for your dominant failure mode. Hand-label 100 outputs with the protocol above, compute Cohen’s kappa, commit the number with its date and the judge model version.

Week 4, online scoring and the triage rule. Turn on scoring for a sampled slice of production. Set one alert, on rate of change. Establish the rule that turns every production failure into a golden-set case the same day, with a named owner.

Ongoing. Grow the set toward the n your target margin requires, refresh monthly, hold out a slice you never optimise against, and re-calibrate quarterly and after any judge-model change.

Three artefacts by day 30: a versioned dataset, a dated calibration record, a written release criterion.

---

Honest limits of this guide

No first-hand benchmarking was performed. The cost model comes from published list prices and stated assumptions rather than metered spend. The sizing table is computed from a formula rather than observed. Every vendor figure is labelled as a vendor claim where it appears.

One experiment would settle the biggest open question in this niche, and it is not here because it needs hands-on measurement. Take 200 RAG traces, label each response’s groundedness by hand against its retrieved context, run the Ragas, DeepEval and autoevals faithfulness implementations over the identical rows, then report Cohen’s kappa per implementation against the human labels with confidence intervals. About a week of one engineer’s time.

---

Frequently asked questions

What is evaluation of LLM applications?

Evaluation of LLM applications means measuring whether your system retrieves the right evidence, calls the right tools, answers correctly and completes the task, scored repeatably on a dataset you own. Model benchmarking measures a base model’s general capability instead. Four scoring methods apply: code checks, model-based scorers, LLM judges, human review.

How many test cases do I need?

About 246 cases to measure an 80% score within ±5 points at 95% confidence, using n = z²m̂(1−m̂)/ε² (arXiv 2506.13023). Drop to 90% confidence and it is 173. A 25 to 50 case set is a smoke test for catastrophic regressions only, and ±1 point at the same score needs about 6,147 cases.

Is LLM-as-a-judge accurate enough to trust?

Only after you calibrate it on your own data. The widely quoted “over 80% agreement with humans” figure is second-hand, undated and measured on someone else’s task. Class imbalance inflates raw agreement, so use Cohen’s kappa or balanced accuracy on a class-balanced sample, aim for 0.61 or better on the Landis and Koch scale, and mitigate position, verbosity and self-model bias.

What is the difference between offline and online evaluation?

Offline evaluation runs on a fixed curated dataset before release and behaves like unit and integration tests. Online evaluation scores sampled live traffic asynchronously, catching novel queries, distribution shift and model drift that no fixed dataset anticipates. Mature teams run both, and every production failure feeds back into the offline set.

How is agent evaluation different?

The unit shifts from one input and output pair to a multi-step trajectory. Add tool-call accuracy, task completion rate, step efficiency and error recovery. Failure modes include infinite loops, wrong tool selection, incomplete goals and inefficient paths (MLflow). Tracing is a prerequisite.

What tools are used for the evaluation of LLM applications?

Frameworks you run yourself: DeepEval, Ragas, Promptfoo, TruLens, OpenAI Evals, EleutherAI LM Evaluation Harness. Platforms adding storage, UI, online scoring and alerting: MLflow, Braintrust, Langfuse, Opik, Arize Phoenix, LangSmith, Datadog Agent Observability, Databricks Agent Evaluation. Most teams run one of each.

How much does it cost to run in production?

Judge tokens per scored trace, times sampling rate, times number of judge metrics, plus platform or span fees. Four judge metrics over top-k=10 retrieval runs about 35,200 input tokens per trace, so 100,000 traces scored in full is 3.52 billion tokens a month. Sample one in ten, run deterministic scorers on everything, and keep retrieved context out of judges that do not need it.

What is a golden dataset?

A versioned, human-reviewed set of test cases with input, optional expected output and scoring criteria, covering critical use cases, known past failures and edge cases. Golden sets are human-annotated. Silver sets are synthetic and get promoted after review. Keep it private and decontaminated, or your scores measure memorisation.

Done properly, the evaluation of LLM applications stops being a quarterly argument about vibes and becomes something you can point at: a versioned dataset, a dated calibration record, a release criterion someone signed. Start with 40 cases this week and grow toward 246.