LLM-as-a-Judge

What it means

LLM-as-a-judge uses a capable model to evaluate generated text on qualities that are hard to check with string matching - helpfulness, tone, factual consistency, or whether an answer followed instructions. You give the judge model the output, often the input and a reference answer too, and a prompt describing the scoring criteria. It returns a score, a label, or a short critique. The judge can grade a single response against a rubric or compare two responses and pick the better one, which is known as pairwise judging.

Why it matters

Most useful qualities of an LLM output are subjective and open-ended, so exact-match or regex checks fail. Human review captures nuance but does not scale to thousands of outputs per release. LLM-as-a-judge sits between the two - cheaper and faster than humans, far more flexible than rule-based metrics. It makes it feasible to score every item in a large eval dataset on each change, which is what turns evaluation into a repeatable part of the development loop rather than a one-off audit.

In practice

Judges are only as good as their prompts, and they carry known biases - toward longer answers, toward the first option in a pair, and toward their own style. Good practice is to calibrate the judge against a set of human-labeled examples before trusting its scores, and to keep the rubric narrow and concrete. Platforms like Braintrust and DeepEval ship built-in judge-based scorers for common criteria and let you define custom ones, so you can attach a judge score to outputs in your eval runs. See the dedicated guide for prompt patterns and calibration steps that reduce judge bias.