LLM Evaluation Guide: Metrics, Methods and Workflow

LLM evaluation means running your system against a fixed set of inputs, scoring each output against explicit criteria, and tracking those scores across versions. That is how you separate an improvement from a regression. This LLM evaluation guide covers the three parts that make it work: a dataset, a scorer, and a methodology for deciding whether a score change is real.

Pick metrics by architecture. RAG systems gate on faithfulness, answer relevancy, contextual precision and contextual recall. Agents gate on task completion, tool correctness, argument correctness and step efficiency. Every system gates on one safety metric plus cost and latency per resolved task.

Choosing metrics is the easy half. Most of this LLM evaluation guide is about the hard half. You have to prove that your metric, your judge and your dataset are themselves correct.

Skip ahead: metric reference · dataset construction · eval set sizing · benchmark defect catalogue · 30-day build plan

Facts checked: every repository count, issue status, changelog entry and API price here is volatile. Items marked [VERIFY] need a fresh read of the linked primary source before you cite them. Nothing was measured first-hand. Where a claim needs hands-on work, this LLM evaluation guide says so and tells you what to run.

---

Why most LLM evals measure the wrong thing

The failure pattern is small and boring. Someone edits a prompt to fix tone. Tone improves. Factual accuracy drops on the 8% of queries involving a date range, and nobody notices for three weeks because the aggregate moved by a point and a half.

The second pattern needs no code change at all. A provider ships a new model snapshot behind the same alias, and your system behaves differently on Tuesday than it did on Monday.

Both are detection problems. Both need an eval suite sensitive enough to see them. Neither is solved by picking a better metric off a list.

Three artefacts here are not in the pages currently ranking for this query. The first is a defect catalogue for HumanEval, built from publicly filed issues on openai/human-eval. It shows a canonical benchmark shipping with a false example in a prompt, a prompt/test name mismatch, and a scoring estimator that can return 1.0 when zero samples passed. The second is an eval-set sizing table computed from the sample-size formula. The third is a cost model for a judge-based suite, priced per run and per month at CI frequency.

---

What LLM evaluation actually is

An LLM evaluation is a repeatable measurement procedure with three components.

The dataset is a fixed collection of inputs. It may pair each input with a reference output, grounding context, or a list of facts the answer must contain. Fixed is the operative word. If the inputs change between runs, you are not comparing versions. You are comparing noise.

The scorer maps an output to a number or a label. A regex, an n-gram statistic, a fine-tuned classifier, an LLM given a rubric, or a human with a form.

The methodology turns scores into decisions. How many samples, how many repeats per sample, which aggregate, which significance test, and what threshold gates a merge.

Deterministic software testing assumes actual == expected. That assumption fails here. The set of acceptable answers to “summarise this support thread” is effectively infinite, and two outputs that share no vocabulary can both be correct. Exact match is not a weak metric for open-ended tasks. It is an incorrect one.

Four axes organise the field, and this LLM evaluation guide uses them throughout.

Offline vs online. Offline evaluation runs against a curated dataset before deployment and gates changes. You control the inputs, you can use reference-based metrics, and you can rerun the same suite a year later. Online evaluation scores sampled production traffic after the response has been served. It has no ground truth, so it uses reference-free metrics only. Its job is catching what your dataset does not contain: novel query shapes, seasonal drift, and provider-side model changes.

Component-level vs end-to-end. Component scoring measures one stage in isolation, such as retriever hit rate or tool-selection accuracy. End-to-end scoring asks whether the user’s task got done. You need both. End-to-end scores tell you something broke. Component scores tell you where.

Reference-based vs reference-free. Reference-based metrics compare the output to a known-good answer. They need expensive labelling and cannot run in production. Reference-free metrics score a property of the output given only the input and the retrieved context. Is this JSON valid. Is every claim grounded. Reference-free scorers are what make online evaluation possible.

Model vs product. Covered next, because it is the error that costs beginners the most time.

---

Model evaluation vs product evaluation

Model evaluation measures raw capability on standardised tasks. MMLU probes multiple-choice knowledge across 57 subjects. HumanEval probes function-level Python synthesis. These exist so model builders can compare training runs and buyers can build a shortlist.

Product evaluation measures whether your prompts, retrieval, tools and guardrails solve a user’s problem on your data. The two relate the way a car’s dyno figure relates to whether it fits in your garage.

	Model evaluation	Product evaluation
Focus	Raw capability of the base or instruct model	Behaviour of the whole assembled system
Scope	Standardised, public, general-purpose tasks	Your task distribution, your domain, your policies
Data source	Published benchmarks (MMLU, HumanEval, GPQA)	Golden set built from your traffic and incidents
Cadence	On model release or provider version bump	Every pull request, plus nightly baselines
Owner	Whoever selects models, often ML or platform	The team that ships the feature
Decision it drives	Which two models to trial	Whether this change merges and ships
Reference data	Usually reference-based	Mixed; production scoring is reference-free
Contamination risk	High and rising, since the sets are public	Low if the set is private and post-cutoff

The transfer problem deserves plain language. A two-point difference between two models on a general benchmark predicts almost nothing about your domain task. The benchmark’s task distribution is not your traffic distribution, and that two-point gap may sit inside the benchmark’s own label-error rate. The HumanEval catalogue below is the strong form of this argument, with citations you can open.

Model evaluation still matters in three situations. Building a shortlist, where benchmarks are a coarse filter that eliminates unsuitable models cheaply. Changing cost tier, where public benchmarks tell you roughly how much headroom you give up before you spend on a product eval. Verifying a provider version bump, where a stable benchmark run is one signal among several that something moved.

---

Why LLM evaluation is harder than it looks

Non-determinism. Identical inputs produce different outputs. Temperature 0 reduces variance without eliminating it. Floating-point non-associativity under different batch sizes, kernel selection on different GPUs, and provider-side routing between replicas all introduce drift. A single run per test case conflates that variance with a real regression.

No single correct answer. Open-ended generation admits many valid outputs that share little surface form. Fuzzy scoring becomes mandatory rather than convenient.

Unbounded input space. Your test set samples a distribution you do not control and cannot fully observe. Users type things nobody on the team imagined, and the tail is where incidents live.

Compounding error. Do the arithmetic, because it changes how people design agents. An agent that is 95% correct at each step lands at 0.95¹⁰ = 0.599 across a ten-step task. At 99% per step, ten steps land just above 0.90. At 90 per 100 steps correct, you finish at 0.35. Per-step accuracy that sounds excellent produces a product that fails more often than it works. That is why step-level metrics and end-to-end completion have to be tracked together.

Novel risk classes. Hallucination, jailbreaks, prompt injection through retrieved content, and leakage of system prompts have no clean analogue in traditional QA. You cannot unit-test “the model was talked out of its policy by a role-play framing” without building an adversarial dataset first.

Then there is the problem underneath all of them. The most popular scorer in production today is an LLM given a rubric. That scorer is the same class of system being measured, with the same failure modes, evaluated by nobody. Calibration is the answer, and it gets its own section.

---

The map: every way to score an LLM output

Six families cover everything you will meet.

Cost column derived from published per-token prices. Cite the pricing page and the date read.

Scorer family	Cost per 1,000 evals	Latency	Deterministic	Needs reference	Detects	Misses
Rule-based / exact	~$0 (compute only)	microseconds	Yes	Sometimes	Schema violations, forbidden strings, missing citations, length breaches	Anything about meaning
Statistical n-gram (BLEU, ROUGE, METEOR)	~$0	milliseconds	Yes	Yes	Surface divergence from a reference	Paraphrase, factual inversion
Embedding similarity (BERTScore)	cents (embedding calls or local GPU)	~10–100 ms	Yes, given a pinned model	Yes	Paraphrase, semantic drift	Factual error where wording is close
Trained classifier (NLI, toxicity, bias)	cents	~10–100 ms	Yes, given a pinned model	Context, not a reference	Contradiction with context, toxic content	Nuance outside training distribution
LLM judge	dollars to tens of dollars	1–10 s	No	No	Subjective quality, groundedness, task success, tone	Its own biases; anything the rubric omits
Human review	hundreds of dollars	minutes to days	No	No	Everything, including novel failures	Scale; consistency without training

The rule that follows: use the cheapest scorer that can detect the failure you care about. Format compliance never needs a judge. A regex will never tell you whether the tone is condescending. Teams burn budget sending every case to a frontier judge when 40% of their failure modes are caught by json.loads and a length check.

Original decision tree. Terminal nodes give a gating set plus a diagnostic set per architecture.

---

Statistical and lexical metrics (BLEU, ROUGE, METEOR)

These predate LLMs by two decades. They still have a place, but a narrow one.

BLEU computes modified n-gram precision against one or more references, typically for n = 1 to 4, combined as a geometric mean and multiplied by a brevity penalty. One reporting error is worth naming. Raw BLEU scores are not comparable across papers, or across your own experiments, unless tokenisation is identical. SacreBLEU fixes this by standardising tokenisation and emitting a version signature with every score. Report BLEU without that signature and the number means something only inside your own repo. NIST weights n-grams by information gain, so rare matches count for more.

ROUGE is recall-oriented and built for summarisation. ROUGE-N counts n-gram overlap. ROUGE-L uses the longest common subsequence, rewarding word order without requiring contiguity.

METEOR combines unigram precision and recall with a heavier weight on recall. It adds stemming and WordNet synonym matching, so “purchase” can match “buy”, and applies a fragmentation penalty when matched words appear scrambled.

Levenshtein distance measures character-level insertions, deletions and substitutions. It is the correct metric for spell correction, OCR post-processing, and structured-ID extraction, where one wrong character is the whole failure.

Here is the limitation, concretely. Source sentence: “The 2024 audit found no material weaknesses in the vendor’s controls.” Summary: “The 2024 audit found material weaknesses in the vendor’s controls.” One deleted token. The claim is inverted. ROUGE-L scores that summary in the high 0.9s, because every remaining n-gram matches. A metric that assigns near-perfect scores to a reversal of meaning cannot be your primary quality signal.

They still earn their place as free, sub-millisecond pre-filters that run before you spend judge tokens. They work as regression tripwires on tightly constrained outputs such as machine translation, structured extraction and code-diff similarity. And a BLEU score that collapses from 0.42 to 0.03 overnight means something went badly wrong, whatever BLEU cannot see.

---

Embedding, entailment and perplexity scorers

BERTScore encodes candidate and reference with a contextual embedding model, greedily matches each candidate token to its nearest reference token by cosine similarity, and aggregates into precision, recall and F1, usually with IDF weighting. It catches paraphrase that BLEU misses. It still requires a reference, and scores compare only when the encoder is pinned to a specific checkpoint.

BLEURT is a transformer fine-tuned to predict human ratings directly. That is why it correlates better with human judgement than overlap metrics on the domains it was trained for. The training set is also the ceiling. Move outside the text types it saw, and correlation degrades with no warning in the score itself.

NLI scorers classify a claim as entailed, neutral or contradicted against a premise. This is the cheap backbone of groundedness checking. Pass the retrieved context as premise and the generated claim as hypothesis. A contradiction label is a hallucination flag that cost a millisecond instead of a judge call. Performance degrades on long premises, so decompose the answer into atomic claims and chunk the context first. Do not feed a 4,000-token passage to a model trained on sentence pairs.

Perplexity measures how well a model predicts a token sequence. Lower means more predictable. It says nothing about whether the text is useful, correct or on-policy. A fluent, confident, entirely fabricated answer has low perplexity. It is genuinely useful for two things. Comparing base models on held-out corpora during pretraining. And contamination detection, where anomalously low perplexity on public benchmark items relative to matched fresh items is evidence the model has seen them. Stop reporting it as a product quality metric.

---

LLM-as-a-judge: the three shapes and when each fails

An LLM judge takes an output, a rubric, and optionally an input and context, then returns a verdict. It comes in three shapes. Choosing the wrong one is the most common way judge programmes go sideways.

Pairwise comparison. Show the judge output A and output B and ask which is better, with an explicit tie option. Pairwise is the most reliable shape, because relative judgement is easier than absolute judgement for models as it is for people. It answers exactly one question: did this change help? It gives you no absolute score to gate on, and comparing k versions properly means k(k−1)/2 comparisons.

Direct scoring against a rubric. Likert 1–5, or binary pass/fail. Binary is better than it sounds. A 1–5 scale invites the model to park on 3 and 4, compressing the range into two values and hiding small deltas. Binary property checks force a decision, produce a proportion you can put a confidence interval around, and let you run McNemar’s test on paired runs.

Reference-free property checks. Is this valid JSON against the schema. Is every claim supported by the retrieved context. Does this response decline the out-of-scope request. These run in production, because they need no ground truth.

Adopt this default. Pairwise for “did this change help”. Binary property checks for CI gating. Likert reserved for the dashboard number somebody wants.

G-Eval is the most widely implemented judge construction. Give the model a criterion definition, have it generate the evaluation steps itself, then score via a form-filling prompt that follows those steps. The original method weights the score by the output token probabilities of each score value, which breaks the clustering problem by producing a continuous score. That step requires logprob access. Not every endpoint exposes it. If yours does not, you get integer scores with the clustering intact. Read the G-Eval paper rather than a vendor’s summary of it.

DAG-style judging is the highest-value technique almost nobody covers. Most criteria that look subjective decompose into a graph of verifiable questions. “Is this support reply acceptable?” becomes four checks. Does it address the stated issue, checkable by keyword and entailment. Does it cite a policy, checkable by regex. Is the cited policy the right one, checkable against the policy index. Is the tone within brand guidelines, genuinely subjective, so send it to the judge. Three of the four are deterministic and free. Encode the decomposition as a decision graph, fail fast on the deterministic nodes, and invoke the judge only on the leaf that requires taste. Cost drops by most of an order of magnitude, and the failing node tells you what to fix.

Judge prompt details that measurably change verdicts:

• Per-level rubric anchors. “Rate helpfulness 1–5” produces noise. “5 = answers the question and cites the specific policy clause; 3 = answers the question but cites no source; 1 = does not address the question” produces a usable distribution.
• Few-shot examples of the extremes. Show one 1 and one 5 from your own data.
• Forced rationale before score. Make the model write the justification first. Score-first prompts produce post-hoc rationalisation.
• Structured JSON output with a fixed schema, so parsing failures are parsing failures and not silent zeros.
• Randomised option ordering in pairwise, recorded per case so you can audit for position effects.

Test for these biases by name. Position bias: the judge prefers the first or second option. Measure it by running every pair twice with the order flipped and reporting the disagreement rate. Verbosity bias: longer answers score higher independent of correctness. Measure it by regressing score on output length. Self-preference: judges favour outputs from their own model family. Use a judge from a different family than the system under test, and keep a human-labelled holdout. Formatting sensitivity: bullets and headers raise scores relative to identical content in prose.

There is a systemic version of self-preference. If most of the industry optimises against judges drawn from two model families, products converge on the verbose, hedged, confidently structured register those judges reward, whether or not users like it. The defence is a human-labelled holdout that never gets optimised against, reviewed by people who are allowed to say the output is annoying.

Further reading: calibrating an LLM judge

---

Calibrating your judge: the step almost everyone skips

A judge you have not calibrated is a random number generator with good grammar. The procedure takes about a day.

1. Sample 100–200 examples from your golden set, stratified across categories and deliberately including cases near the decision boundary.
2. Label them by hand against the rubric the judge will see. Use a domain expert where correctness requires domain knowledge. Have a second person label a 30-case overlap, so you can measure inter-annotator agreement before treating your own labels as truth.
3. Run the judge on the identical set with the identical rubric.
4. Build a confusion matrix. Do not skip to the summary statistic.
5. Read the disagreement pattern. Judges usually fail in one direction and for one reason, and the reason is usually a rubric ambiguity you can fix in a sentence.
6. Revise, rerun, repeat until the residual disagreements are ones you understand and accept.

Triggers to show: judge model version change, judge prompt change, input distribution shift. Include the kappa formula and the worked matrix below.

Why percent agreement is the wrong number

Take a task where 92% of outputs are genuinely acceptable. A judge that returns “pass” unconditionally, ignoring its input, scores 92% agreement with your human labels. It carries exactly zero information about quality. Any threshold expressed as raw percent agreement is satisfied by that judge.

Cohen’s kappa corrects for agreement expected by chance. κ = (p₀ − pₑ) / (1 − pₑ), where p₀ is observed agreement and pₑ is the agreement you would expect from the marginal distributions alone.

Worked example on 200 calibration cases, human pass rate 92%:

	Judge: pass	Judge: fail	Human total
Human: pass	179	5	184
Human: fail	9	7	16
Judge total	188	12	200

Observed agreement p₀ = (179 + 7) / 200 = 0.930. Expected agreement pₑ = (184/200 × 188/200) + (16/200 × 12/200) = 0.8648 + 0.0048 = 0.8696. Kappa = (0.930 − 0.8696) / (1 − 0.8696) = 0.0604 / 0.1304 = 0.46.

That judge agrees 93% of the time and has moderate agreement at best once chance is removed. It misses 9 of the 16 genuine failures, which is the only column anyone cares about. Report kappa and the base rate together, always. Kappa alone misleads in the other direction on extremely skewed sets.

Use Krippendorff’s alpha instead when you have more than two raters, missing labels, or ordinal categories. It handles all three cases where kappa does not.

The threshold claim you should not adopt

Caylent’s LLM evaluation guide tells readers to keep refining “until the agreement between yourself and the judge is above 85-90%” and states that LLM-as-a-judge is “good enough for approximately 90% of your use cases” (caylent.com). Both are presented as facts. Neither carries a statistic, a base rate, a dataset size, or a source. Treat them as one consultancy’s practitioner intuition, which is what they are.

A defensible threshold needs three things that figure omits. An agreement statistic that corrects for chance. The base rate of the positive class, so a reader can tell whether the number is impressive. And an explicit statement of the cost asymmetry between a false pass and a false fail on your task. A judge gating medical-claim accuracy and a judge gating marketing tone need different operating points. No single percentage covers both.

Direction of error

For gating metrics, tune the judge conservative. A false fail costs an engineer twenty minutes. A false pass costs you the incident. Move the threshold until false passes are rare, accept the noisier failure list, and route borderline cases to a stricter second judge.

For monitoring metrics, tune for stability rather than peak accuracy. An online score that is 4 points pessimistic but never moves without cause beats an unbiased score that wobbles 6 points week to week. You are watching the derivative, not the level.

Recalibration triggers

Recalibrate when the judge model version changes, when the judge prompt template changes, and when the input distribution shifts enough that your calibration set no longer resembles traffic. Pin the judge to a dated snapshot and record that version in every stored result row. An unpinned judge means a provider update silently redefines your metric. Every historical score becomes incomparable, with no visible event in your logs to explain the step change.

One result circulating in practitioner discussion is worth citing with its caveat attached. A Hacker News thread reports that automatically generated evaluation checklists raised exact agreement between LLM judgements and human preferences from 46.4% to 52.2% (news.ycombinator.com/item?id=41776194) [VERIFY]. That is a secondhand summary. Trace it to the underlying paper before quoting it. Taken at face value it cuts both ways: the technique helps, and the absolute ceiling sits barely above coin-flip territory on whatever task was measured. Anyone quoting a blanket agreement rate as a general property of judges should be asked which task, which base rate, and which statistic.

---

Human evaluation: what it’s for and what it costs

Three formats do almost all the work. Rubric rating, where an annotator scores an output against defined anchors. Pairwise preference, where they pick between two outputs. Blind review, where model identities are hidden so brand expectations do not leak into the labels.

Human evaluation has its own failure modes, and teams routinely treat human labels as ground truth without checking any of them. Vague instructions produce low inter-annotator agreement. If two annotators only agree at κ = 0.3, no judge can be calibrated against their labels. Fatigue drift is real within a session, so labels drawn late in a 200-case batch differ systematically from labels drawn early. Randomise case order per annotator. Expertise and cultural variation matter for domain correctness, register and politeness norms. Measure inter-annotator agreement and publish it with the labels, or you are anchoring your judge to noise.

The budgeting rule that keeps this affordable: you are not buying human labels to score production. You are buying a few hundred labels to anchor a judge that scores production. Size the spend for 150–300 chosen cases refreshed quarterly.

Humans stay non-substitutable in four places. Safety-critical domains where a false pass has legal or physical consequences. Brand voice, defined by people who cannot articulate it in a rubric until they see a violation. Creative quality, where the interesting output is by definition outside the distribution the judge was calibrated on. And adjudicating novel failures the judge has never been shown, which is most of what an incident review consists of.

---

RAG evaluation metrics

Split every RAG metric by the component it indicts. That one habit turns a dashboard into a debugging tool.

Retriever-side metrics

• Contextual precision: are the relevant chunks ranked above the irrelevant ones? A low score means your reranker or embedding model is misordering, and the right context may be buried below the generator’s attention budget.
• Contextual recall: does the retrieved context contain everything needed for the expected answer? Requires a reference answer or a list of required facts. This is the hard ceiling on the whole system.
• Contextual relevancy: what proportion of retrieved sentences are actually relevant? Low relevancy with high recall means you are paying for tokens that dilute the signal.
• Classic IR metrics still apply. Hit rate @k, NDCG, precision@k, MRR. Use them for retriever development because they are free, deterministic and fast.

Generator-side metrics

• Faithfulness: is every claim traceable to the provided context? Score against the retrieved context alone, never against the ground truth. That is the only way to separate “the model made it up” from “retrieval did not supply it”.
• Answer relevancy: does the response address the question asked, without padding or topic drift?

Symptom	Likely component	Confirming metric	Typical fix
Answer is fluent and factually wrong	Retriever missed the evidence, or generator ignored it	Contextual recall first; if recall is high, score faithfulness against retrieved context alone	Low recall → chunking, embedding model, k. High recall plus low faithfulness → prompt constraint plus a claim-level grounding check
Answer is correct but misses half the question	Contextual recall on multi-part queries	Contextual recall, split by sub-question	Query decomposition, multi-query retrieval, raise k
Answer cites the wrong document	Contextual precision	Contextual precision and rank of the gold chunk	Reranking, metadata filters, hybrid keyword plus vector search
Answer is vague and hedged	Contextual relevancy is low; noise drowns signal	Contextual relevancy plus retrieved token count	Tighter chunking, reranker cutoff, drop k
System says “I don’t know” too often	Retrieval returns nothing above threshold	Hit rate @k, similarity score distribution	Lower the similarity threshold, add query rewriting, expand the index
Answer contradicts the cited chunk	Generator	Faithfulness, claim-level	Constrain the prompt, add an NLI contradiction gate before returning
Scores fine offline, bad in production	Dataset representativeness	Compare offline query embeddings against production query clusters	Promote real production traces into the golden set

State the ceiling explicitly, because it saves teams weeks. Generator quality is bounded by retrieval. If contextual recall is 0.4, the necessary evidence is absent from 60% of your contexts, and no prompt engineering recovers information that was never supplied. Fix retrieval first, every time.

Claim-level scoring is worth the cost wherever correctness matters. Decompose the answer into atomic claims, then check each against the context. A single 0.72 faithfulness score tells you nothing you can act on. A list saying claim 3 of 5 is unsupported, with the claim text, is a bug report.

Further reading: RAG evaluation metrics

---

Agent evaluation metrics

Agents fail differently, and the aggregate success rate hides almost everything useful.

End-to-end: task completion. Judge completion from the full execution trace, not the final message. Agents produce confident closing summaries describing work they did not do, and a judge shown only the last turn will happily mark it complete. Give the judge the tool calls, the returned values and the final state.

Component-level metrics

• Tool correctness: were the expected tools called at all? Compare the called-tool set against an expected set per case.
• Argument correctness: were the arguments right? This is where agents fail most and where nobody looks. The right tool with a malformed date range fails silently and returns an empty result the agent then reports as “no records found”.
• Plan quality: was the plan complete and logically ordered before execution started? Score the plan artefact separately if your agent emits one.
• Plan adherence: did execution follow the plan, or did the agent improvise after step 3?
• Step efficiency: how many steps, redundant calls and loops relative to a reference trajectory? Count repeated identical calls explicitly. They are the signature of a stuck agent.

Trace-level and span-level scoring are different things, and you need both. A trace is the whole request. A span is one unit of work inside it: a retrieval, a tool invocation, a model call, a nested sub-agent. Agent evaluation without span-level tracing is not possible in any meaningful sense. You cannot attribute a failure to step 4 if the only artefact you stored is the final answer. Instrument spans before you write a single agent metric.

Cost and latency are first-class quality metrics for agents, not operational footnotes. An agent that reaches the same completion rate in 40 tool calls instead of 6 is a product failure. It is eight times slower, it costs more per resolved task than the human it replaced, and the user watched a spinner for ninety seconds. Gate on cost per resolved task, not cost per call.

The dimension absent from nearly every guide is recovery behaviour. What does the agent do when a tool returns a 500, an empty result set, a timeout, or malformed JSON? The realistic options are retrying forever with the same arguments, hallucinating a plausible result, abandoning the task silently, or reporting the failure and asking. Only one is acceptable. You will not find out which one you have unless you build test cases that force it. Stub your tools to return errors deliberately and score the trace that follows.

Further reading: AI agent evaluation metrics

---

Multi-turn and session-level evaluation

Per-turn scoring misses the failures that generate support tickets. The model states a policy at turn 3 and contradicts it at turn 9. The user says “nothing containing dairy” at turn 2 and the assistant recommends a cheese dish at turn 14. Every individual turn is faithful, relevant and well-formed. The session is broken.

Turn-level variants of the RAG metrics still apply and should be computed per turn. Turn faithfulness against that turn’s retrieved context, turn relevancy against that turn’s user message, plus turn contextual precision, recall and relevancy. Aggregate them across the session as a minimum rather than a mean. One badly grounded turn poisons a conversation, and an average hides it.

Session-level criteria are mostly custom. These four cover most of what goes wrong:

• Constraint retention: were stated constraints honoured for the rest of the session? Encode the constraint and the turn it was stated at in the test case.
• Contradiction across turns: does any later assertion contradict an earlier one? An NLI scorer run pairwise over assistant assertions catches a useful fraction cheaply.
• Escalation appropriateness: did the assistant hand off to a human at the right point, neither too early nor three turns after the user asked?
• Goal completion within N turns: completion alone is not enough if it took eleven turns.

Vendor tooling is catching up. Braintrust’s changelog documents a Group scope for online scoring that evaluates related multi-turn traces as one unit, keyed on a session key (braintrust.dev/changelog) [VERIFY]. That is a release note describing a shipped feature, not independent validation that the feature works well. Read it as evidence that session-level scoring is now a product category.

---

Safety, red teaming and guardrails

Three things get conflated constantly. They have different owners and different cadences.

Safety evaluation is offline scoring against a curated adversarial dataset. It runs in CI, it has an absolute floor rather than a relative tolerance, and it answers whether this version fails the attacks you already know about. RealToxicityPrompts and similar public sets are a starting point. Your own recorded attacks matter more.

Red teaming is active adversarial generation to find failures you have not seen. It is exploratory, it produces new test cases, and its output feeds the safety dataset. Automated red teaming uses an attacker model to mutate seed attacks. Human red teaming finds the categories the attacker model does not think of.

Guardrails are runtime filters that block or rewrite requests and responses in the live path. They are a control, not a measurement. A guardrail that blocks four in five jailbreak attempts tells you nothing about how often your model would have complied. Reporting block rates as a safety metric confuses the seatbelt with the crash test.

Attack categories to build test cases for:

• Direct jailbreak: role-play framings, hypothetical framings, encoding tricks, instruction-hierarchy confusion.
• Prompt injection via retrieved content: instructions embedded in a document, a web page or a support ticket that your retriever pulls into context. This is the attack surface RAG creates and the one teams underestimate most.
• Data exfiltration: eliciting the system prompt, tool schemas, other users’ records, or credentials reachable through connected tools.
• Policy circumvention: getting a refusal-worthy answer by decomposing the request across turns, or by asking in a format the policy language did not anticipate.

Vendor tooling here is expanding. Confident AI’s changelog records red teaming picking up code vulnerability scanning (confident-ai.com/docs/changelog) [VERIFY], which is a release note and should be read as one.

The compliance argument unlocks budget for this work. The EU AI Act and NIST’s AI Risk Management Framework both push toward documented, repeatable evaluation with retained records rather than ad hoc review. Read the primary texts rather than a vendor summary: the EU AI Act and the NIST AI RMF. If you will need an audit trail later, the cheapest time to start versioning eval datasets and storing result rows is before you have any.

Further reading: red teaming LLM applications

---

Operational metrics and what your eval suite actually costs

Track these alongside quality, in the same result row, from day one. P50 and p95 latency end to end and per span. Input and output tokens per request. Cost per resolved task. Throughput at your concurrency limit. Error and timeout rate by cause.

Cost per resolved task is the one that changes decisions. Cost per API call flatters an agent that makes 40 cheap calls to do a job another architecture does in 3.

The cost model

Cost per eval run = (test cases) × (judge metrics per case) × [(judge input tokens × input price) + (judge output tokens × output price)] × (repeats per case).

Worked example. 1,500 input tokens and 300 output tokens per judge call, 4 judge metrics per case, 1 repeat.

Per judge call at a small-model tier priced $0.25 per 1M input and $1.25 per 1M output: (1,500 × $0.25 ÷ 1M) + (300 × $1.25 ÷ 1M) = $0.000375 + $0.000375 = $0.00075.

Per judge call at a frontier tier priced $3.00 per 1M input and $15.00 per 1M output: $0.0045 + $0.0045 = $0.009, twelve times the small-tier cost.

Suite size	Judge calls per run	Cost/run (small tier)	Cost/run (frontier tier)	Monthly, 10 CI runs/day (small)	Monthly, 10 CI runs/day (frontier)	Monthly with 60% pre-filter (frontier)
50 cases	200	$0.15	$1.80	$45	$540	$216
250 cases	1,000	$0.75	$9.00	$225	$2,700	$1,080
1,000 cases	4,000	$3.00	$36.00	$900	$10,800	$4,320

Assumptions: 4 judge metrics per case, 1,500 input / 300 output tokens per judge call, 1 repeat per case, 30 days at 10 CI runs per day. The two price tiers are illustrative placeholders, not quoted prices [NEEDS SOURCE — replace both rows with the per-1M input and output prices from your provider's public pricing page, and date-stamp the table with the day you read them]. Rebuild this table with your own token counts. Judge prompts with long few-shot blocks routinely run 4,000+ input tokens and quadruple the input side.

Caption must state the token assumptions and the pricing date, so readers can recompute against their own rates.

Two multipliers surprise teams. Running k = 3 repeats per case triples every figure above. And the monthly column, not the per-run column, is the number that matters. Nobody blinks at $9 a run. A $2,700 monthly bill for a 250-case frontier-judge suite on every pull request gets escalated.

Three levers cut cost without gutting signal:

1. Deterministic pre-filters. Run schema validation, length checks, forbidden-string checks and NLI grounding before any judge call. Cases that fail a deterministic check never reach the judge. A 60% reduction is realistic on suites with strict output formats. Measure your own hit rate rather than assuming it.
2. Tiered judging. A small model screens every case. The frontier judge adjudicates only cases scored near the threshold, plus a random audit sample. You pay frontier prices on 10–20% of cases and keep frontier accuracy where it matters.
3. Sampling online evals. Score 1–5% of production traffic. You are watching a distribution, and a 3% sample of 100,000 daily requests is 3,000 scored traces a day, which is far more statistical power than your offline suite has.

---

Building the evaluation dataset

The dataset is the highest-leverage part of the whole system and the part teams skip. A mediocre metric on a representative dataset beats a sophisticated metric on 30 cherry-picked happy-path examples.

Three sources

Public benchmarks are fast and free, generic and contaminated. Use them for model shortlisting, never for product gating.

Human-annotated golden sets are authoritative and expensive. This is your ground truth and your calibration anchor.

Synthetic silver sets scale cheaply and inherit the generator’s blind spots. Bootstrap: hand-write a golden seed of 30–50 cases, generate silver variants, spot-check a sample by hand, and promote reviewed silver cases into the golden set as they earn it.

The 5 D’s, with a measurement for each

The arXiv practical guide (2506.13023) frames dataset quality as five properties. Each one is checkable.

Property	What it means	How to measure it
Defined scope	Each sub-dataset targets one component or capability	Every case tagged with the component it exercises; no untagged cases
Demonstrative of production usage	The set reflects real traffic, not imagined traffic	Classify a sample of real prompts into categories, compare proportions against the eval set, correlate offline scores with a satisfaction signal
Diverse	Coverage across topic, difficulty, length and input style	Cluster case embeddings and check for empty regions; tag by difficulty and confirm every bucket is populated
Decontaminated	Not present in the model’s training data	See the contamination section below
Dynamic	Versioned, audited, refreshed on a schedule	Dataset version recorded in every result row; documented refresh cadence with an owner

Composition target for a golden set

Five categories, each non-empty before you quote a number:

• Happy path: the queries the feature was built for.
• Edge cases: long inputs, empty inputs, ambiguous phrasing, multiple entities, unusual formats, non-English text if you serve it.
• Adversarial inputs: the attack categories above, at minimum one per category.
• Out-of-scope requests the system must decline. A system with no test cases for correct refusal will not refuse correctly.
• Every past production incident, converted into a permanent case. Highest value, and free. Every postmortem ends with a test case added to the golden set, linked to the ticket.

Synthetic generation techniques

Distillation from a frontier model produces fluent cases skewed toward that model’s phrasing. Persona prompting (“write this query as a frustrated first-time user on a phone”) produces diversity that temperature alone does not. Temperature and top-p variation broadens surface form without changing intent. Evol-Instruct-style escalation takes a simple seed instruction and iteratively deepens it, adding constraints and reasoning steps, which is how you get hard cases without hand-writing them. Constitutional AI-style critique-and-revise loops generate policy-boundary cases from a written policy.

The caveat applies to all of them. Synthetic data inherits the generator’s blind spots. If your generator does not think of an attack, your dataset does not contain it, and your eval will report that you are safe from it.

Metadata per case

Attach tags for component, category and difficulty, plus grounding documents, the expected key facts, an optional reference output, the provenance ticket, the date added, and the owner.

One leak vector gets missed constantly. Source the grounding documents independently of the system under test. If you build expected answers by running your own RAG pipeline and having a human approve the output, your eval measures whether the system still does what it did in March.

Ownership and maintenance

Name a person. Define what triggers a refresh: a new feature, a shifted traffic mix, a quarterly review, or any incident. And carve out a holdout set that is never optimised against, never looked at during development, used only to check whether improvements on your main set are real. If a change gains 6 points on the working set and 0 on the holdout, you tuned to the test.

Further reading: building a golden dataset

---

How many test cases do you actually need?

This is the question every guide dodges, and it has an answer.

For a proportion metric, the required sample size is:

n = z² × m̂(1 − m̂) / ε²

Here z is the normal quantile for your confidence level (1.96 for 95%), m̂ is the expected value of the metric, and ε is the margin of error you accept. The arXiv practical guide gives this formula and works the canonical case. At 95% confidence, ±5% margin, expected score 0.8: 1.96² × 0.8 × 0.2 / 0.05² = 3.8416 × 0.16 / 0.0025 ≈ 246 cases.

Expected score	±10%	±5%	±3%	±1%
0.50	97	385	1,068	9,604
0.70	81	323	897	8,068
0.80	62	246	683	6,147
0.90	35	139	385	3,458
0.95	19	73	203	1,825

All values computed from n = z² × m̂(1 − m̂) / ε² at z = 1.96, rounded up. Reproduce any cell in fifteen seconds with a calculator.

X-axis from ±10% down to ±1%. Caption carries the formula and z = 1.96 so every point is reproducible. Source: arxiv.org/html/2506.13023v1.

Read the table for what it kills. On a 50-case suite with an expected score near 0.8, your margin of error is roughly ±11%. A three-point improvement sits inside the noise band by a factor of nearly four. Every “we improved accuracy from 78% to 81%” claim made on a 50-case suite is a coin flip described as a result.

Small suites are not useless. They are for iteration, not for claims.

Choosing the significance test

Comparing two versions is a hypothesis test, and the right test depends on what your scorer emits. The arXiv guide maps them:

Score type	Test	Why
Binary pass/fail, paired across versions	McNemar’s test	Uses only the discordant pairs, which is exactly the information a paired binary comparison carries
Continuous 0.0–1.0 scores, paired	Two-tailed paired t-test	Interval-scaled and paired; report the confidence interval on the difference, not just p
Likert 1–5	Wilcoxon signed-rank	Ordinal, because the gap between 3 and 4 is not guaranteed equal to the gap between 4 and 5

Paired designs matter. Run both versions on identical cases and compare per case. Comparing two independent samples throws away most of your statistical power.

Handling non-determinism

Run each case k times, aggregate per case, then compare aggregates. k = 3 is a reasonable default for judge-scored metrics, k = 5 for anything high variance. Report a confidence interval on the aggregate. A single run per case conflates model variance with a real regression, and the resulting flaky gate trains your team to re-run until green.

How much variance your model and judge actually show is not something a guide can tell you. Measure it. Run the same unchanged suite ten times, compute the standard deviation of the aggregate, and set your regression tolerance above it. That experiment costs one afternoon and prevents a year of arguing about flaky results.

The practical staging

25–50 cases to start iterating and find obvious breakage. Around 250 before you quote a number outside the team. Stratify by category and report per-segment scores. An aggregate of 0.84 across five categories can hide a category sitting at 0.51, and that category is where your users are.

---

Your benchmark is buggy: a defect catalogue from HumanEval

Provenance note, stated up front. Nothing in this section was run, reproduced or measured. Every claim is a publicly filed issue on the openai/human-eval repository, linked so you can read the thread yourself. Issue states change. Check the current status before citing, and record the date you checked.

HumanEval is one of the most-cited code generation benchmarks in the field, and its pass@k estimator is the reference implementation downstream harnesses copy. The repository is MIT licensed, and as captured in the source brief stood at 3,331 stars, 44 open issues, last pushed 2025-01-17 (github.com/openai/human-eval) [VERIFY — counts and push date are volatile; re-read the repo page and date-stamp before publishing].

Issue	What it breaks	Category	Link
#6	HumanEval/47 prompt docstring asserts a false example answer	Prompt label error	issues/6
#60	HumanEval/161 prompt defines solve; test expects a different name	Prompt/test mismatch	issues/60
#35	estimate_pass_at_k returns 1 when c=0 and n<k	Scoring estimator	issues/35
#52	IndentationError running human_eval/execution.py	Harness fragility	issues/52
#36	ThreadPoolExecutor wrapping per-thread multiprocessing questioned	Harness design	issues/36
#8	Reproduced GPT-Neo 125M/1.3B scores far below the Codex paper figures	Reproducibility	issues/8

Extend the table with each issue’s reaction count and status read from GitHub on publication day, plus star count, open-issue count and last-push date. Caption must state the check date.

Defect 1: a wrong answer inside the prompt

Issue #6 reports that the docstring for HumanEval/47, which is the prompt shown to the model, gives median([-10, 4, 6, 1000, 10, 20]) as 15.0.

Sort the list yourself: −10, 4, 6, 10, 20, 1000. Six elements, so the median is the mean of the third and fourth: (6 + 10) / 2 = 8.

The example in the prompt is false. A model reasoning carefully from the docstring is pulled toward a wrong answer by the specification it was given. The brief records 22 reactions on the issue [VERIFY]. Ten seconds of arithmetic verifies the defect without trusting anyone.

Defect 2: the prompt and the test disagree

Issue #60 reports that HumanEval/161 declares the function as solve in the prompt while the test code expects a different name. A model that implements the specified function correctly, with the name the prompt asked for, fails the test. That is not a measurement of code generation ability. It is a naming collision scored as incompetence.

Defect 3: the scoring estimator itself

pass@k estimates the probability that at least one of k sampled solutions passes. Naively sampling k solutions gives a high-variance estimate, so the Codex paper’s unbiased estimator generates n ≥ k samples, counts c passing, and computes:

pass@k = 1 − C(n − c, k) / C(n, k)

The logic: C(n − c, k) / C(n, k) is the probability that a random size-k subset of the n samples contains no passing sample, so one minus that is the probability it contains at least one.

Issue #35 reports that when c = 0 and n < k, the implementation returns 1 where 0 is expected. Read the formula at that boundary. With c = 0 the numerator is C(n, k), and when n < k the standard convention gives C(n, k) = 0, so the expression evaluates according to the implementation’s guard, and the reported result is 1.0. A task where nothing passed is credited as fully passing.

Whether that boundary is reachable in a normal run depends on the n and k a given harness uses. That is exactly the point. The correctness of a widely-copied scoring function depends on configuration details that published scores never state.

Defect 4: two teams running “HumanEval” may not be running the same thing

Issue #52 reports an IndentationError running human_eval/execution.py. Issue #36 questions the harness structure, asking why a ThreadPoolExecutor wraps per-thread multiprocessing rather than using a ProcessPoolExecutor at the top level. Neither is exotic. Both mean the artefact called “HumanEval” in a paper is a specific fork at a specific commit with specific local patches, and that two reported scores are not necessarily comparable measurements.

Defect 5: reproduction

Issue #8 reports an attempt to reproduce raw GPT-Neo 125M and 1.3B HumanEval performance that landed far below the Codex paper figures. Who was right is not the interesting question, and this LLM evaluation guide takes no position on it. The interesting fact is that a public, filed, unresolved reproduction discrepancy exists on the canonical implementation of a benchmark that appears in thousands of model cards.

The three conclusions the rest of this guide depends on

1. A benchmark number without a harness version and a decontamination statement is not a measurement. It is a claim about an unspecified procedure.
2. The gap between two models’ benchmark scores is frequently smaller than the benchmark’s own label-error rate. When the top of a leaderboard is separated by 1.5 points and the test set contains prompts with false examples, the ranking is not information.
3. This is the strongest available argument for building your own product eval set. Your set is small, private, post-cutoff, and about your actual task. Those four properties beat 164 public problems with an open issue tracker.

Further reading: LLM benchmarks explained

---

Contamination: when the test set is in the training set

Contamination means evaluation data appearing in training data, and it happens at every stage. Continued pretraining on scraped corpora, supervised fine-tuning on instruction sets that quietly include benchmark items, and preference alignment on comparisons built from benchmark prompts all introduce it. A model can be contaminated by a dataset its base pretraining never touched.

Detection without training-data access. You will not have the training corpus for any commercial model, so use behavioural probes. Continuation testing gives the model the first half of an eval item and checks whether it completes the rest verbatim, including incidental formatting and unusual variable names. Reproduction of arbitrary detail is not generalisation. Log-probability inspection compares perplexity on benchmark items against matched fresh items of the same domain, length and difficulty. Anomalously low perplexity on the benchmark side is evidence of memorisation. Both techniques come from the arXiv practical guide. Neither is conclusive alone. Together they are a reasonable screen.

Detection with training-data access. If you control the corpus, do exact string, substring and hash comparison between eval items and training documents. At scale, suffix arrays or Bloom filters make n-gram overlap checks tractable across terabytes. Decontaminate by removing matched documents and re-running, and record what you removed.

The only durable defence is a private evaluation set collected after the model’s training cutoff. Never published, never pasted into a prompt-sharing tool, never committed to a public repo, never sent to a service whose terms permit training on submitted data. Read those terms before you send your golden set through an API.

The tell to watch for in your own numbers. Benchmark scores that climb fast while user satisfaction, task completion and ticket volume stay flat. If a model gains eight points on a public benchmark and nothing your users experience changes, you have measured memorisation.

---

Regression testing and CI/CD for LLM applications

This is where the LLM evaluation guide turns into a gate that runs without anyone remembering to run it.

Golden set mechanics. Each case stores input, optional reference output, grounding context, per-case scoring criteria, category tag, and provenance. Version the file in the repository next to the code, so a pull request diff shows which cases changed and reviewers can object. A dataset stored in a SaaS tool with no export is a dataset you cannot review, cannot bisect and cannot take with you.

Threshold design. Three kinds, and they behave differently.

• Absolute floors on safety metrics. Never relative. Zero tolerance for a jailbreak that previously failed now passing.
• Relative tolerance on quality metrics versus the current production baseline, sized above your measured run-to-run variance. If repeated identical runs vary by ±2 points, a 1-point tolerance is a random number generator attached to your merge button.
• Hard fail on any previously-passing case that now fails. This is the rule that catches what aggregates hide. A change that fixes six cases and breaks six holds the average flat while silently swapping which users are broken. Per-case regression detection catches it. The mean never will.

Handling flakiness. Pin temperature and seed where the provider supports it. Pin the judge model to a dated snapshot. Run k samples on high-variance cases and aggregate. Quarantine persistently flaky cases rather than deleting them, with a named owner and a date. A deleted flaky case is a failure mode you decided to stop observing.

What the CI report must contain. The list of newly improved cases. The list of newly regressed cases with diffs. Per-category score deltas. Cost and latency deltas. A link to the full trace for every regressed case. A report that says “score: 0.83 (was 0.85)” gets ignored by the third week.

Drift you cannot catch in code review. Prompt drift accumulates from ten small edits, none of which regressed anything visibly, which together moved behaviour somewhere nobody chose. Provider-side drift happens with no change on your side at all. The defence against the second is a scheduled nightly run of a pinned dataset against a pinned prompt, with the provider as the only variable, plotted over time. When the line steps, you have your answer and a date.

Release criteria, written down before release day. Minimum score per gating metric. Maximum tolerated regression count. All safety checks green with no exceptions. A named person who can override, and a recorded reason when they do. Criteria negotiated during a release are not criteria.

Further reading: prompt regression testing

---

From offline evals to production observability

Offline evaluation tells you whether a change is safe to ship. Production observability tells you what your dataset was missing.

What to log. Full traces with span-level detail. Inputs, retrieved context with document IDs and scores, every tool call with arguments and return value, model identifier and version, prompt template version, dataset version, token counts, latency per span, and any user feedback signal you can collect. Thumbs, resolution, escalation, abandonment, retry. Redact PII at the logging boundary, not later.

Online scoring design. Sample rather than scoring everything. Use reference-free metrics only, because production has no ground truth. Alert on distribution shift in the score, not on individual low-scoring traces. One bad trace is Tuesday. A groundedness distribution whose mean drops four points across a day is an incident.

The feedback loop that keeps the suite honest. Cluster low-scoring production traces by embedding, triage clusters into failure categories, pick two or three representative traces per category, and promote them into the golden set with the trace ID as provenance. Do this weekly. This single loop is the difference between a suite that reflects your product and one that reflects the product you had at launch.

Alert hygiene. Vendors are converging on severity tiers. Confident AI’s changelog documents alert priorities (critical, warning, error, info) with per-integration filtering and JSONL trace export (confident-ai.com/docs/changelog) [VERIFY]. That is a vendor release note. The general point stands independently. An alerting setup with one severity level trains people to ignore it.

The practical minimum for a small team. Trace everything. Score 1–5% of traffic on two or three reference-free metrics. Read the worst 20 traces by hand every week. That last item is not a fallback for teams without tooling. It is where you find the failure categories no metric was watching for.

Further reading: LLM observability and tracing

---

Nine ways LLM evaluations go wrong

Pitfall	What it looks like	Why it happens	The fix
Overfitting to the eval set	Scores climb steadily; users report nothing changed	The set is small and every change is tuned against it	A holdout never optimised against, plus scheduled refresh from production traces
Judge bias	Verbose answers win; option A wins; same-family outputs win	Judges inherit the preferences of their training	Randomise ordering, judge from a different model family, calibrate against human labels with kappa
Data leakage	Strong public benchmark scores, flat user satisfaction	Benchmarks are public and scraped	Private post-cutoff eval data; continuation and perplexity screens
Metric gaming	The system learns the scorer’s shortcuts, not the task	Any single metric is optimisable	Two or more diverse metrics, review trade-offs explicitly, rotate judge prompts periodically
Happy-path-only datasets	Perfect scores, incidents anyway	Test cases written by the person who built the feature	Adversarial and out-of-scope cases as required, non-empty dataset categories
Too many metrics	A dashboard with 20 scores that drives no decision	Every metric seemed worth adding	Cap gating metrics at about five; everything else is diagnostic and unpinned from the gate
Unpinned judge model	Historical scores become incomparable with no visible event	A model alias silently moved to a new snapshot	Pin the judge to a dated version, record it in every result row, recalibrate on change
Undersized suites	A 3-point gain on 50 cases announced as an improvement	Nobody computed a margin of error	The sizing table above; ~250 cases before quoting a number; a paired significance test
Trusting benchmark harnesses uncritically	A leaderboard delta drives a model decision	Benchmark scores look like measurements	Record the harness commit, read the issue tracker, treat benchmarks as a shortlist filter only

---

LLM evaluation tools and frameworks

Three categories solve three different problems. Conflating them is why tool comparisons go in circles.

Open-source metric libraries give you scorer implementations you call from your own code: DeepEval, RAGAS, Evidently, OpenAI Evals, promptfoo, lighteval. They cost nothing, run in your CI, and leave you responsible for storage, dashboards and trace capture.

Managed platforms add tracing, storage, dashboards, dataset management, online scoring and alerting: Braintrust, Comet Opik, LangSmith, Arize Phoenix, Confident AI, Databricks Agent Evaluation. You are buying the plumbing around the metrics, which is most of the work.

Benchmark harnesses run standardised academic benchmarks: openai/human-eval, EleutherAI’s lm-evaluation-harness, and lighteval, which spans both categories.

Tool	Category	Licence	Self-host	Tracing included	CI integration documented	Public pricing
DeepEval	Metric library	Apache 2.0 [VERIFY]	Yes (library)	Via Confident AI	Yes	Library free; platform tiers published
RAGAS	Metric library	Apache 2.0 [VERIFY]	Yes (library)	No	Yes	N/A
Evidently	Metric library + monitoring	Apache 2.0 [VERIFY]	Yes	Partial	Yes	Cloud tiers published
OpenAI Evals	Metric library	MIT [VERIFY]	Yes	No	Manual	N/A
promptfoo	Metric library + red teaming	MIT [VERIFY]	Yes	Partial	Yes, first-class	Enterprise pricing on request
lighteval	Benchmark harness	MIT [VERIFY]	Yes	No	Manual	N/A
Opik	Platform	Apache 2.0 [VERIFY]	Yes	Yes	Yes	Cloud tiers published
Arize Phoenix	Platform	Check repo [VERIFY]	Yes	Yes	Yes	Cloud tiers published
Braintrust	Platform	Proprietary	Hybrid deployment offered [VERIFY]	Yes	Yes	Published
LangSmith	Platform	Proprietary	Enterprise self-host [VERIFY]	Yes	Yes	Published
Confident AI	Platform	Proprietary	No [VERIFY]	Yes	Yes, via DeepEval	Published
Databricks Agent Evaluation	Platform	Proprietary	Within Databricks	Yes, via MLflow	Yes	Within Databricks pricing
lm-evaluation-harness	Benchmark harness	MIT [VERIFY]	Yes	No	Manual	N/A

Date-stamp the caption and give one source link per cell. Do not source any row from a competitor vendor’s comparison page.

Two columns are deliberately absent above: last commit date and open issue count. Both go stale within days, and a stale maintenance figure is worse than none. Open each repository link and read them yourself. The GitHub insights page gives commit frequency over the last year in about five seconds. Confirm every licence against the repository’s LICENSE file, since projects relicense.

Selection criteria, in priority order:

1. Does it capture traces at span level? Without this you cannot evaluate agents at all.
2. Can you define custom scorers in code? Every serious eval eventually needs a scorer nobody ships.
3. Does it integrate with your CI runner, with a non-zero exit code on threshold breach?
4. Can you export raw results and datasets? Avoid any platform you cannot leave with your data intact.
5. Does self-hosting exist if you have data-residency constraints? Decide this before procurement.

On the Hugging Face evaluation guidebook. It ranks for this query and it is worth reading, but the GitHub repository opens by stating that the guidebook is no longer maintained and directs readers to a Hugging Face Space, as of Dec 2025 [VERIFY]. Read the Space, not the repo. A top-ranking result for this keyword being an explicitly abandoned resource is the clearest available signal that the field lacks a maintained, vendor-neutral reference.

On vendor numbers. Evidently’s LLM guide states it has “over 25 million downloads” (evidentlyai.com) [VERIFY]. That is a self-reported figure on the vendor’s own marketing page, and PyPI counts include CI runners and mirrors. Present it as a vendor claim if you present it at all.

Further reading: LLM evaluation tools compared

---

Your first 30 days of LLM evaluation (with a template)

An ordered build sequence. Each week produces one artefact.

Week 1: one failure, 25–50 cases, labelled by hand

Pick one failure mode already costing you something. A support escalation category, a known hallucination pattern, a format break that trips a downstream service. Not “quality” in general. One failure.

Write 25–50 cases covering it across the five required categories: happy path, edge cases, adversarial inputs, out-of-scope requests that must be declined, and any past incident that fits. Score all of them by hand and write down what made each one pass or fail while you do it. That written record becomes your rubric, and it is the only way to get anchors that are real instead of invented.

Output: a labelled golden set and a written definition of “good” for one failure mode.

Week 2: tracing, then three scorers

Instrument tracing across every span before writing scorers. Retrieval, tool calls with arguments and returns, model calls with token counts, latency per span.

Then implement three scorers, deliberately different in kind. One deterministic format check that costs nothing. One judge for groundedness or task success. One operational metric, where cost per resolved task is the most useful.

Output: a script that runs the suite end to end and prints per-case results with pass/fail, score, judge rationale, and a trace link.

Week 3: calibrate the judge

Run the judge on the week-1 hand-labelled cases. Build the confusion matrix. Compute Cohen’s kappa alongside the base rate. Read the disagreements case by case and fix the rubric ambiguities they expose. Repeat until the residual disagreements are ones you can explain.

Output: a pinned judge model version and a versioned rubric, both recorded in every result row.

Week 4: wire it into CI

Add the suite to your pull request pipeline with explicit thresholds. Absolute floors on safety. Relative tolerance on quality, sized above your measured variance. Hard fail on any previously-passing case that now fails. Add a nightly baseline run against a pinned dataset to catch provider drift. Set up the weekly loop that promotes failing production traces into the golden set.

Output: a gate that blocks a bad merge, and a queue of real production failures becoming test cases.

The template

Build the Sheet, CSV and Markdown files first, then screenshot the real artefact.

The downloadable template ships as a Google Sheet, a CSV and a Markdown table with these columns:

case_id · category · difficulty · input · retrieved_context · expected_key_facts · reference_output · metric_scores (one column per gating metric) · judge_rationale · judge_model_version · human_label · provenance_ticket · date_added · owner

It comes with a one-page pre-release checklist covering dataset version, judge version, threshold values, safety check status, and the named override authority.

Judge prompt skeleton

You are evaluating a response for GROUNDEDNESS against a provided context.

RUBRIC
PASS: Every factual claim in the response is directly supported by the
      CONTEXT. Claims that restate the question, or that hedge without
      asserting a fact, do not require support.
FAIL: The response contains at least one factual claim that is absent
      from, or contradicted by, the CONTEXT.

EXAMPLES
[one worked PASS example from your own data]
[one worked FAIL example from your own data]

PROCEDURE
1. List every factual claim in the RESPONSE as a numbered list.
2. For each claim, quote the supporting span from CONTEXT, or write
   "UNSUPPORTED".
3. Only then, assign the verdict.

CONTEXT:
{{context}}

QUESTION:
{{input}}

RESPONSE:
{{output}}

Return JSON only:
{"claims": [{"claim": "...", "support": "...|UNSUPPORTED"}],
 "rationale": "...",
 "verdict": "PASS|FAIL"}

Rationale before verdict, claims enumerated before judgement, binary output, JSON schema. Swap the criterion and the examples. Keep the structure.

---

How to learn LLM evaluation

An ordered path, not a link dump.

1. Build one 30-case eval for something you have already shipped. Nothing teaches this as fast. You will hit every real problem in a week. Your cases are all happy path. Your judge disagrees with you and you cannot say why. Your scores move 4 points between identical runs. Reading about those problems does not produce the understanding that debugging them does.

2. Read the Hugging Face evaluation guidebook at its current maintained location. The GitHub repo states it is deprecated and points to a Hugging Face Space. Read the Space.

3. Read the arXiv practical guide, 2506.13023. Take from it the 5 D’s dataset framework, the sample-size formula, and the score-type-to-significance-test mapping. It is the only source in this space that treats eval as a statistics problem.

4. Read the G-Eval paper to understand how judges are constructed, including the token-probability weighting step most implementations quietly drop.

5. Subscribe to two eval-tool changelogs. Vendor release notes are marketing, and they are also the fastest signal about what practitioners hit this quarter. Session-level scoring and code vulnerability scanning in red teaming both showed up in changelogs before they showed up in guides.

Skills to acquire, so you can self-assess: writing a rubric with per-level anchors that two people apply identically; computing an agreement statistic and knowing when kappa is the wrong one; sizing a sample and stating a margin of error; reading a span-level trace and attributing a failure to a step; choosing a significance test from the score type.

A practitioner on Hacker News names the gap this LLM evaluation guide fills. Most “LLM for practitioners” guides skip evaluation entirely. Getting a prompt working on five examples is easy, while knowing whether it generalises across your domain is the hard part, and “the vibes-based evaluation most LLM tutorials teach” fails people used to statistical rigour (news.ycombinator.com/item?id=47075257) [VERIFY]. That is one individual’s comment, quoted as such, and it states the problem more precisely than most vendor documentation.

Independent, non-vendor material exists and is worth finding. The “Forest Friends” zine on system evals for LLM-driven applications has been discussed by operators on Hacker News (news.ycombinator.com/item?id=41905357) [VERIFY]. Material written by people with no product to sell reads differently from material written by people with one.

---

What this LLM evaluation guide does not cover

Stated plainly, so you know what you still have to find out yourself.

No measured variance figures. How much a specific judge model’s scores wobble on your task is knowable only by running your suite ten times unchanged and computing the standard deviation. Do that before setting a regression tolerance.

No independent tool benchmarks. The tooling table reports licences, self-hosting, tracing and CI integration from public documentation. It does not rank tools on scorer accuracy, because doing that credibly requires running each one against a shared human-labelled set and publishing the set.

No confirmation that the HumanEval issues cited are unfixed at HEAD. Every one is a publicly filed issue with a link. Open them and check the current status before you cite them anywhere that matters.

No prices. The cost model uses placeholder rates so the arithmetic stays visible. Substitute your provider’s published per-token prices on the day you build the budget.

Everything else in this LLM evaluation guide is a procedure you can run this week: size the set, calibrate the judge with kappa, gate the merge, and promote production failures back into the golden set until the suite describes the product you actually shipped.

---

Frequently Asked Questions

What are the key metrics for LLM evaluation?

The metrics depend on your architecture. Cap the gating set at about five and keep everything else as diagnostic signal, because a dashboard carrying twenty scores drives no decisions. For RAG: faithfulness, answer relevancy, contextual precision, contextual recall. For agents: task completion, tool correctness, argument correctness, step efficiency. Every system adds one safety metric plus latency and cost per resolved task. BLEU and ROUGE apply only to translation and summarisation where you hold reference outputs.

How do I learn LLM evaluation?

Build one 30-case evaluation for a feature you have already shipped, because hands-on work teaches this faster than reading. Then read the maintained Hugging Face evaluation guidebook, noting that its GitHub repository is deprecated and points to a Space. Follow it with arXiv 2506.13023 for dataset and statistics methodology, and the G-Eval paper for judge construction. Five skills to target: writing rubrics with anchors, computing agreement statistics, sizing a sample, reading a trace, choosing a significance test.

What is LLM evaluation assessment?

LLM evaluation assessment means measuring an LLM or an LLM-powered system against explicit quality criteria. You run it on a fixed dataset, score the outputs, and track those scores across versions, so a real improvement can be distinguished from run-to-run noise. Assessing the model uses public benchmarks and measures general capability. Assessing the product measures your prompts, retrieval, tools and guardrails on your own data. Three components make it work: a dataset, a scorer, and a methodology.

How many test cases do I need for an LLM evaluation?

Roughly 246 cases gives you 95% confidence at a ±5% margin of error when the expected score is 0.8. That comes from n = z² × m̂(1 − m̂) / ε² with z = 1.96, a formula published in arXiv 2506.13023 that you can reproduce with a calculator. Practically: 25–50 cases to start iterating, around 250 before you quote a number externally. A 3-point improvement on a 50-case suite is indistinguishable from noise.

Can I trust LLM benchmark scores like HumanEval or MMLU?

Not as ground truth. The canonical implementations carry uncorrected defects that are publicly filed and still readable on the issue tracker, including a false example inside a prompt and a scoring estimator that can credit a full pass when nothing passed. See HumanEval/47’s wrong docstring answer, the HumanEval/161 prompt/test name mismatch, and the pass@k boundary bug. Benchmarks are useful for coarse model shortlisting and useless as a substitute for a product eval on your data.

Is LLM-as-a-judge reliable enough to replace human review?

Only after you have measured its agreement with human labels on your specific task, and never completely. The judge shares its failure modes with the system it is scoring and cannot detect anything your rubric failed to describe. Measure it properly: 100–200 human-labelled cases, a confusion matrix, and Cohen’s kappa rather than raw percent agreement. On a task where 92 of every 100 answers already pass, a judge that always says “pass” scores 92% agreement with zero information. The agreement targets circulating in vendor guides are unsourced.

What is the difference between offline and online LLM evaluation?

Offline evaluation runs before deployment against a curated dataset with known inputs, can use reference-based metrics because ground truth exists, and gates merges in CI. Online evaluation scores sampled live traffic asynchronously using reference-free metrics only, and catches novel queries, distribution shift and provider-side model drift. You need both. Production has no ground truth, which is why reference-based metrics cannot run there and why online scoring watches score distributions rather than individual traces.

How much does it cost to run LLM evaluations?

Cost per run equals test cases × judge metrics per case × (judge input tokens × input price + judge output tokens × output price) × repeats per case. The monthly figure at CI frequency is the number that decides which judge you use. At 250 cases, 4 metrics, 1,500 in / 300 out tokens and 10 CI runs a day, a frontier-tier judge lands near $2,700 a month against roughly $225 for a small-model tier, using illustrative placeholder rates. Three levers cut it: deterministic pre-filters, tiered judging, and sampling in production.

Is there a free LLM evaluation template or checklist?

Yes. This LLM evaluation guide ships a downloadable evaluation-case template as a Google Sheet, CSV and Markdown table, with columns for case ID, category, difficulty, input, retrieved context, expected key facts, reference output, per-metric scores, judge rationale, judge model version, human label, provenance ticket, date added and owner, plus a one-page pre-release checklist. Free scorer implementations also ship with the open-source frameworks: DeepEval, RAGAS, Evidently, promptfoo and OpenAI Evals.

---

Changelog

Date	Change
2026-08-11	Initial publication. HumanEval issue statuses, repository counts, tool licences and API prices marked [VERIFY] pending same-day source checks.