When a foundation model writes text, you need a number that says how good that text is. The exam names four ways to get one: ROUGE, BLEU, BERTScore and LLM-as-a-judge. The first three compare the model's output with a reference answer written by a person. The fourth asks another model to grade the output, often with no reference at all. Most exam questions come down to matching the metric to the task and noticing whether a reference answer exists.
Reference-Based Metrics
ROUGE, BLEU and BERTScore all need reference text: an ideal answer, written or approved by a person, to compare against. They're fast, cheap and objective, so they suit large automated test runs. Their shared weakness is that they reward similarity to one particular answer. A response that is correct but worded very differently can score poorly. Without reference answers, you can't use them at all.
ROUGE for Summaries
ROUGE (Recall-Oriented Understudy for Gisting Evaluation) counts how many words and short word sequences from the reference also appear in the model's output. It leans toward recall: did the output include the content the reference contains? That makes it the usual choice for summarization. Common variants are ROUGE-1 (single words), ROUGE-2 (word pairs) and ROUGE-L (the longest shared sequence). Its weakness is that it only sees matching words, so a summary that paraphrases heavily can score low even when its meaning is right.
BLEU for Translation
BLEU (Bilingual Evaluation Understudy) also counts overlapping word sequences, but it leans toward precision: how much of the output appears in the reference? It also penalizes outputs that are too short, so a model can't score well by writing only a few safe words. BLEU is the classic metric for machine translation, where it tracks human judgment fairly well. It's a poor fit for creative writing, where many different answers are equally good.
BERTScore for Meaning
BERTScore compares meaning rather than exact words. It turns the output and the reference into embeddings (numeric representations of meaning) and measures how similar they are. So "the meeting was postponed" and "the meeting was pushed back" score as close, even though they share few words. It handles paraphrasing far better than ROUGE or BLEU, at the cost of more computing. Amazon Bedrock's automatic evaluation uses BERTScore to measure accuracy for text summarization.
LLM-as-a-Judge
With LLM-as-a-judge, a second model reads each response and scores it against a rubric, explaining its reasoning. It can measure qualities that word overlap can't, such as helpfulness, relevance, logical coherence, following instructions, professional tone, and harm or stereotyping. Many of these need no reference answer. Bedrock provides built-in judge metrics and also lets you write custom ones. Watch for judge bias: a model tends to favour answers that look like its own. AWS recommends choosing a judge from a different model family and checking its scores against human ratings.
On the Exam
| Task or clue | Metric |
|---|---|
| Summaries compared with reference summaries | ROUGE |
| Translations compared with reference translations | BLEU |
| Paraphrased output, meaning matters more than wording | BERTScore |
| No reference answers; helpfulness, tone or instruction following | LLM-as-a-judge |
The usual trap is offering ROUGE, BLEU or BERTScore when the scenario says no reference answers exist. All three need one.