BLEU vs ROUGE vs BERTScore - Which to Use and Why All Three Fail on Chat

Three metric names come up constantly in LLM evaluation - BLEU, ROUGE, and BERTScore - and they get treated as interchangeable “text similarity” scores. They are not. Each one measures a different thing, was built for a different task, and fails in a different way. Picking the wrong one gives you a number that looks rigorous and means nothing.

This guide explains how each actually computes a score, runs all three on the same sentence so you can see them diverge, and shows why none of them can grade an open-ended chat answer.

The three metrics at a glance

Metric	What it measures	Built for	Handles paraphrase?
BLEU	N-gram precision vs a reference	Machine translation	No
ROUGE	N-gram recall vs a reference	Summarization	No
BERTScore	Embedding similarity vs a reference	Semantic overlap	Yes

The one-line summary - BLEU asks did you avoid wrong words, ROUGE asks did you cover the right words, BERTScore asks did you capture the right meaning. All three still need a reference answer to compare against, and that shared assumption is where they all break on open-ended text.

BLEU - precision, built for translation

BLEU (Bilingual Evaluation Understudy) counts how many n-grams in the candidate output also appear in the reference. It computes this for unigrams, bigrams, trigrams and 4-grams, then combines them.

The two details that matter:

• It is precision-based. It asks what fraction of the candidate’s n-grams are “correct” - present in the reference. A candidate that only outputs words it is sure about scores well on precision.
• It has a brevity penalty. Pure precision would let a one-word output score perfectly, so BLEU multiplies by a penalty when the candidate is shorter than the reference. This stops the model gaming the score by saying almost nothing.

BLEU was designed for machine translation, where there genuinely is a “right” translation and exact wording carries meaning. Outside translation, its precision-and-brevity design is a poor fit.

ROUGE - recall, built for summarization

ROUGE (Recall-Oriented Understudy for Gisting Evaluation) flips the question. Instead of “how much of the candidate is correct,” it asks “how much of the reference did the candidate cover.”

The common variants:

• ROUGE-N - n-gram recall (ROUGE-1 for unigrams, ROUGE-2 for bigrams).
• ROUGE-L - based on the longest common subsequence, so it rewards matching word order without requiring contiguous n-grams.

ROUGE became the summarization standard because in summarization, coverage is the point - you want the summary to include the key content of the source. Its recall orientation makes it a coverage meter, not a fluency or faithfulness meter, which is exactly the confusion that trips teams up.

BERTScore - meaning, not exact strings

Both BLEU and ROUGE count exact string matches, so “the movie was excellent” and “the film was great” score near zero against each other despite meaning the same thing. BERTScore fixes that.

BERTScore embeds every token with a contextual language model, then matches each candidate token to its most similar reference token by cosine similarity. It averages those similarities into precision, recall, and an F1 score. Because it compares vectors rather than strings, synonyms and paraphrases score high. If embeddings are new to you, see the embedding primer.

The trade-offs - BERTScore needs a model to run, so it is slower and heavier than counting n-grams, and its score is only as good as the underlying embedding model. But for reference-based tasks where wording is free to vary, it is far more forgiving and more human-aligned than overlap counting.

The same sentence, three scores

Take one candidate and one reference:

• Reference - “The quarterly report shows revenue climbed sharply.”
• Candidate - “Earnings rose steeply according to the quarterly figures.”

The candidate is an excellent paraphrase. Watch the metrics disagree:

• BLEU - low. Almost no exact n-grams overlap (“quarterly” is about it), so precision is poor.
• ROUGE-1 - low to moderate. A couple of unigrams match, but “revenue/earnings” and “climbed/rose” are counted as misses.
• BERTScore - high. “Earnings” matches “revenue,” “rose” matches “climbed,” “steeply” matches “sharply” in embedding space, so the semantic overlap is scored correctly.

Same sentence, opposite verdicts - which is the entire reason you cannot pick a metric at random.

Why all three fail on open-ended answers

Here is the ceiling on every metric above - they are reference-based. They need one correct answer to compare against. That works for translation and summarization, where a reference exists. It falls apart for chat, reasoning, agents, and RAG answers, where many different responses are all valid.

Two failure modes follow directly:

1. A correct answer worded differently gets punished. Your model gives a perfectly good reply that does not match your single reference, and the score drops.
2. A wrong answer that reuses reference words gets rewarded. The metric sees overlap and scores it high, even though the answer is false.

BERTScore softens the first problem by matching meaning, but it still cannot tell you whether an answer is helpful, faithful to a source document, or safe. Those qualities have no reference string at all. For open-ended generation you need an LLM-as-judge metric - see G-Eval explained for the chain-of-thought judging approach, and faithfulness for the RAG hallucination check that works with no reference answer.

So which one, when

• Machine translation - BLEU. It is the field standard and the task fits its design.
• Reference-based or extractive summarization - ROUGE, usually ROUGE-1, ROUGE-2 and ROUGE-L together.
• Semantic overlap with a reference - BERTScore, when paraphrasing is fine but you still have a gold answer.
• Open-ended chat, agents, RAG, tone, instruction-following - an LLM-as-judge metric. Overlap metrics do not apply.

A common production pattern is to run a cheap overlap metric as a fast regression gate and a judge metric for the qualities that actually matter. Our LLM evaluation metrics explained guide lays out that layered approach, and what LLM evaluation is covers the fundamentals.

Where the tools fit

You rarely code these from scratch. DeepEval ships classic scorers alongside its LLM-as-judge metrics in one Apache-2.0 framework, so you can run an overlap score and a G-Eval judge in the same pytest-style suite and compare them. Arize Phoenix provides 50+ pre-built eval metrics with the strongest RAG evaluation in the category, and it runs locally in under a minute if you want to eyeball metric behavior on real traces before committing to one.

Bottom line - these are not three flavors of the same score. BLEU is precision for translation, ROUGE is recall for summarization, BERTScore is semantic overlap for reference-based tasks, and none of them can grade an open-ended answer. Match the metric to the task, and reach for a judge metric the moment your output has more than one right answer. For the full field of platforms that ship these scorers, see best RAG evaluation tools.