Using LLM-as-a-Judge

Why we need a judge at all

In the last chapter you saw that most useful LLM outputs have no single correct answer. A summary can be faithful and well written in a hundred different phrasings. String matching and BLEU scores fall apart the moment wording varies, which is almost always. This is the core problem that LLM-as-a-judge solves - you hand the output to a capable model and ask it to grade the quality directly, the same way a human reviewer would, but at machine speed and machine cost.

The idea is deceptively simple. You write a rubric in plain language, feed the judge the input, the model’s output, and optionally a reference answer, and the judge returns a score plus an explanation. Done well, this correlates with human judgment far better than any lexical metric. Done carelessly, it produces confident nonsense that quietly corrupts every decision you make downstream. Most of this chapter is about the difference.

The three shapes of a judge

There are three common patterns, and choosing the right one matters more than the prompt wording.

Reference-free scoring asks the judge to rate an output against a rubric with no gold answer. “On a scale of 1 to 5, how faithful is this summary to the source document?” This is what you use in production where you have no ground truth, and it is the most powerful and the most dangerous mode.

Reference-based scoring gives the judge a known-good answer to compare against. “Does this response convey the same information as the reference?” This is stricter and less prone to drift because the judge has an anchor, but it requires you to have written references.

Pairwise comparison shows the judge two outputs and asks which is better. “Response A or Response B - which more directly answers the question?” Humans and models are both far more consistent at relative judgments than absolute ones, so pairwise is the gold standard when you are comparing two prompt versions or two models. The catch is that it does not give you an absolute quality number, only a preference.

A practical rule - use pairwise when you are comparing candidates, and reference-free scoring when you need a standalone number for a dashboard or a CI gate.

Writing a judge prompt that holds up

A weak judge prompt says “rate this response from 1 to 10.” A strong one does four things.

First, it defines each score point concretely. A 5 means every claim is supported by the source. A 3 means one minor unsupported claim. A 1 means the response contradicts the source. Vague scales collapse toward the middle because the model has no reason to commit.

Second, it asks for reasoning before the score, not after. When the judge writes its justification first, the score it lands on is anchored to actual analysis rather than a gut number it then rationalizes. This is chain-of-thought applied to grading, and it measurably improves agreement with humans.

Third, it constrains the output format so you can parse it. Ask for JSON with a reasoning field and a score field. Frameworks depend on this structure.

Fourth, it isolates one dimension per judge. Do not ask a single call to rate faithfulness, tone, and completeness at once - the scores bleed into each other. Run three narrow judges instead.

Here is the skeleton most production judges follow:

You are grading a summary for FAITHFULNESS to a source document.

Source: {input}
Summary: {output}

A claim is faithful only if it is directly supported by the source.
Score 5 = every claim supported.
Score 3 = one minor unsupported claim.
Score 1 = a claim contradicts the source.

First list each claim and whether the source supports it.
Then return JSON: {"reasoning": "...", "score": <1-5>}

The biases you have to design around

LLM judges have well-documented failure modes, and pretending they do not exist is how people ship broken evals.

Position bias - in pairwise comparison, judges favor whichever answer came first (or sometimes second). The fix is to run each comparison twice with the order swapped and only count a win if the judge agrees both times.

Verbosity bias - judges reward longer answers even when the extra length adds nothing. If your rubric does not explicitly value conciseness, a rambling response often beats a tight one.

Self-preference bias - a model tends to rate outputs written in its own style more highly, which is why judging a model with itself is a trap. Use a different model family as the judge when you can.

Sensitivity to formatting - markdown, confident tone, and citations can all inflate scores independent of substance. Frontier judges are less susceptible than smaller ones, which is one reason judge model quality matters.

Calibrate before you trust

Here is the step almost everyone skips. Before you let a judge gate your releases, measure how well it agrees with humans. Take fifty to a hundred cases, have a person label them, then run your judge on the same set and compute agreement - Cohen’s kappa or plain percent agreement. If the judge agrees with humans 90 percent of the time, you have an instrument you can rely on. If it agrees 60 percent, your judge is barely better than a coin flip and every downstream number is noise. Recalibrate whenever you change the judge model, the prompt, or the task.

How the tools implement this

You rarely hand-roll all of this. DeepEval ships GEval, an implementation of the G-Eval paper that turns a plain-language criterion into a calibrated judge with chain-of-thought scoring, plus prebuilt metrics for faithfulness and answer relevancy. It runs as pytest cases so a failing judge fails your CI. Promptfoo exposes llm-rubric and pairwise select-best assertions driven from its YAML config, which makes A/B comparing prompt variants a few lines. Braintrust provides autoevals, a library of scoring functions you can run in experiments and wire into quality gates. Langfuse lets you attach LLM-as-a-judge scores to production traces so you can grade live traffic, not just test sets.

For the deeper mechanics, what is LLM-as-a-judge covers the concept, and the LLM-as-a-judge guide walks through building one end to end. If you are choosing a framework, best LLM eval frameworks compares the options.

Key takeaways

• Use LLM judges because real outputs have no single correct string to match against.
• Pick the shape - pairwise for comparing candidates, reference-free scoring for standalone numbers.
• Write concrete score definitions, ask for reasoning before the score, and grade one dimension per judge.
• Design around position, verbosity, and self-preference bias with order-swapping and a cross-family judge.
• Calibrate against human labels before trusting a judge to gate anything.

Next, in Core Evaluation Metrics, we turn these judged scores into the specific metrics teams actually track - faithfulness, answer relevancy, hallucination rate, and more.