# ROUGE and Human Eval

Recall-oriented overlap metrics for summarisation, paired with human review as the ground truth they approximate.

**Rests on:** BLEU and Reference Metrics, Evaluation Datasets

*LLMs and Generative AI · LLM Evaluation*


## Why learn this

You will need this to answer the question that every AI project raises: 'How do we know our model is working?' ROUGE and human evaluation are the two-answer reply — a cheap automatic number you can run on every change, and a careful human judgment you run on a sample. Without ROUGE, you cannot compare two versions of your summarizer without reading everything yourself. Without human evaluation, you will not catch the summaries that read as nonsense but score well. This concept leads directly to 'Evaluation Datasets' (where your reference summaries come from), 'BLEU and Reference Metrics' (the same idea for translation), and 'Answer Relevancy and Correctness' (measuring what the model got right). In your library's 'Synthetic Eval Data' page you will use ROUGE to check that your generated test cases produce the summaries you expect.


## The intuition

When a school teacher marks an essay, they read it. If you want to know whether a student summary of a paragraph 'captures the point', you compare it to a reference summary — one you wrote yourself. If the student's summary uses the same words that you used, it is likely they captured the same ideas. That is the entire core of ROUGE: it counts how many words overlap between the student's summary and the reference. But words alone are not enough — order matters. 'The cat chased the dog' and 'The dog chased the cat' use the same words but tell different stories. So ROUGE also looks at pairs of words — 'the cat', 'cat chased', 'chased the', 'the dog' — to see if they appear together in the same order. Words and word-pairs are the 'units' ROUGE compares. Then ROUGE computes a percentage: of the reference's units, how many did the student's summary include? That percentage is ROUGE's score. The reason ROUGE is useful is it is automatic — you can run it on a thousand summaries in a second. But it only measures that the right words appear — it cannot tell whether the summary is grammatical, or true, or even in the right order beyond the pairs. This is why human evaluation is the reference standard: ROUGE looks like it scores everything, but it only scores word overlap. Humans catch what ROUGE misses. The practical habit is to use ROUGE as a first pass — a cheap screen that flags low-scoring summaries for human review — and then sample the rest for a human read.


## An analogy

Imagine you are a teacher with a hundred student summaries of the same paragraph. You want to grade them all, but reading each one carefully takes minutes. So you start by counting: you make a list of the important words — 'rain', 'flooded', 'village', 'school', 'closed' — and you check each summary for how many appear. That is ROUGE's 'recall': of the important words from your reference, how many the student used. If the student wrote 'the school was flooded because of heavy rain', that has all five — a high score. But this fast check has a flaw: a student who wrote 'rain village flooded closed school' gets a perfect score, even though the sentence is gibberish. Here is where the analogy breaks down: ROUGE would call that a good summary, because it only counts words, not grammar or meaning. So you add a second step: you read a random sample of what the high-scoring summaries and the low-scoring ones — all of them, to see what ROUGE sees and what it misses. That second step, where a human actually reads, is human evaluation. The analogy holds as long as you remember: the word count is a shortcut, not the real understanding. The real understanding is the human read. ROUGE is the shortcut that gets you to the right pile of summaries to review.


## Definition

ROUGE (Recall-Oriented Understudy for Gisting Evaluation) is a family of automatic metrics that score a generated summary by counting how many of its words and word-pairs overlap with one or more reference summaries, and human evaluation is the direct examination by a person of a summary's quality, which ROUGE approximates.


## How it connects

This concept is about evaluation — measuring whether a generative model works. It builds on the idea of a 'reference' answer, which you will meet again in 'Evaluation Datasets'. It directly contrasts with 'Human Evaluation', which ROUGE aims to approximate. After this, you will learn 'BLEU and Reference Metrics' (translation) and 'Answer Relevancy and Correctness' (separate dimensions of output quality).


## The mathematics

ROUGE is a way to measure how much two texts overlap by looking at their words or groups of words. It does not understand meaning, grammar, or word order beyond what is explicitly counted. It treats text as a bag of items.

First, we define the 'unit'. A unit can be a single word (called an unigram) or a pair of adjacent words in the same order (called a bigram). For example, in the phrase 'the cat', the units are 'the' and 'cat' if we look at single words, and 'the cat' if we look at pairs.

We compare two texts: a reference summary (the ideal answer) and a generated summary (the model's answer).

Step 1: Count the unique units in the reference summary. We list every unit that appears. If a unit appears multiple times, we count it only once because we care about presence, not frequency. Let this count be R.

Step 2: Count the unique units in the generated summary. Similarly, we list every unique unit and count them. Let this count be G.

Step 3: Find the shared units. We look at the list of units from the reference summary and check which ones also appear in the generated summary. The number of units that appear in both lists is the intersection. Let this count be S.

ROUGE-Recall measures how much of the reference was captured by the generated text. It is calculated as:
Recall = S divided by R.
This tells us what fraction of the important content in the reference is present in the generated summary.

ROUGE-Precision measures how much of the generated text is correct according to the reference. It is calculated as:
Precision = S divided by G.
This tells us what fraction of the generated summary matches the reference. A model that repeats words unnecessarily will have high recall but low precision.

ROUGE-F1 combines these two scores into one number. It uses the harmonic mean. The harmonic mean of two numbers is not their average; it is a specific calculation that gives a lower score if the two numbers are very different from each other. This prevents a model from getting a high score by having only high recall or only high precision.
The formula for F1 is:
F1 = 2 times (Recall multiplied by Precision) divided by (Recall plus Precision).

In short, ROUGE counts unique words or word pairs in the reference and generated texts, finds the overlap, and divides that overlap by the total count in the respective text to get Recall and Precision. It then combines them for F1.


## A worked example

We will compute ROUGE-1 (single words) recall, precision, and F1 for two short summaries.

Reference summary: 'The cats sat on the mat'
Generated summary: 'The cat on the mat'

Step 1: Identify unique units in the reference summary.
The words are: 'The', 'cats', 'sat', 'on', 'the', 'mat'.
We treat 'The' and 'the' as different because case matters in standard string matching. However, for simplicity in this example, let us assume we convert all text to lowercase first. So 'The' becomes 'the'.
Lowercase reference words: 'the', 'cats', 'sat', 'on', 'the', 'mat'.
The unique units (set of distinct words) are: {'the', 'cats', 'sat', 'on', 'mat'}.
Count R = 5.

Step 2: Identify unique units in the generated summary.
The words are: 'The', 'cat', 'on', 'the', 'mat'.
Lowercase generated words: 'the', 'cat', 'on', 'the', 'mat'.
The unique units (set of distinct words) are: {'the', 'cat', 'on', 'mat'}.
Count G = 4.

Step 3: Find the shared units.
We look for words that are in both sets.
Reference set: {'the', 'cats', 'sat', 'on', 'mat'}
Generated set: {'the', 'cat', 'on', 'mat'}
The common words are: 'the', 'on', 'mat'.
Note: 'cats' is in the reference but not the generated (because 'cat' is different from 'cats'). 'sat' is in the reference but not the generated. 'cat' is in the generated but not the reference.
Count S = 3.

Step 4: Calculate Recall.
Recall = S divided by R.
Recall = 3 divided by 5.
Recall = 0.6.
This means 60% of the unique words in the reference appear in the generated summary.

Step 5: Calculate Precision.
Precision = S divided by G.
Precision = 3 divided by 4.
Precision = 0.75.
This means 75% of the unique words in the generated summary appear in the reference.

Step 6: Calculate F1.
F1 = 2 times (Recall multiplied by Precision) divided by (Recall plus Precision).
First, multiply Recall and Precision: 0.6 times 0.75 = 0.45.
Second, add Recall and Precision: 0.6 plus 0.75 = 1.35.
Third, divide the product by the sum: 0.45 divided by 1.35.
0.45 / 1.35 = 1/3 ≈ 0.333.
Wait, let me re-calculate carefully.
2 * 0.45 = 0.9.
0.9 / 1.35 = 90 / 135.
Divide both by 45: 2 / 3 ≈ 0.667.
F1 ≈ 0.667 (or 66.7%).

Result for ROUGE-1:
Recall = 60%
Precision = 75%
F1 ≈ 67%

For ROUGE-2, we would look at pairs of adjacent words instead of single words.
Reference bigrams: ('the cats', 'cats sat', 'sat on', 'on the', 'the mat').
Generated bigrams: ('the cat', 'cat on', 'on the', 'the mat').
Shared bigrams: ('on the', 'the mat').
S = 2.
R (reference bigrams) = 5.
G (generated bigrams) = 4.
Recall = 2/5 = 0.4.
Precision = 2/4 = 0.5.
F1 = 2 * (0.4 * 0.5) / (0.4 + 0.5) = 2 * 0.2 / 0.9 = 0.4 / 0.9 ≈ 0.444.


## From scratch, in Python

```python
from collections import Counter

def ngrams(text, n):
    words = text.lower().split()
    return [tuple(words[i:i+n]) for i in range(len(words) - n + 1)]

def rouge_n(reference, generated, n):
    ref_ngrams = set(ngrams(reference, n))
    gen_ngrams = set(ngrams(generated, n))
    shared = ref_ngrams.intersection(gen_ngrams)
    if not ref_ngrams:
        return 0.0
    recall = len(shared) / len(ref_ngrams)
    if not gen_ngrams:
        precision = 0.0
    else:
        precision = len(shared) / len(gen_ngrams)
    if recall + precision == 0:
        f1 = 0.0
    else:
        f1 = 2 * recall * precision / (recall + precision)
    return recall, precision, f1

reference = "The cats sat on the mat."
generated = "The cat on the mat."
print(rouge_n(reference, generated, 1))
```


## The same thing, with a framework

Using the `evaluate` library:

```python
from evaluate import load
rouge = load("rouge")
results = rouge.compute(predictions=["The cat on the mat."], references=["The cats sat on the mat."])
```

This returns a dictionary with keys like `rouge1`, `rouge2`, `rougeL`, each holding an object with `precision`, `recall`, and `fmeasure`. It does the same counting under the hood: it splits into n-grams and computes intersections, but adds stemming and other processing.


## What the code does

We define a function `ngrams` that splits text into a list of words, then for each position in that list it creates a tuple of `n` consecutive words. A tuple is like a list that cannot be changed, and it can be used as a key in a set. The list comprehension `tuple(words[i:i+n])` takes a slice of the words list, starts at index `i` and goes to `i+n`, and converts that to a tuple. The `range` gives us every valid starting index, which is from 0 to the length minus `n` plus one, so that we do not go past the end. Then `rouge_n` finds the intersection of the two sets of n-grams. `len(shared)` gives the number of shared n-grams. The `if` checks are to avoid division by zero. F1 is the harmonic mean of recall and precision. Finally, we call it with our example and print the result.


## Common misconceptions

**ROUGE measures faithfulness** — If the ROUGE score is high, the summary must be true.

ROUGE only counts word overlap. A summary could use all the same words as a reference that says the opposite. In fact: ROUGE measures overlap with a reference, not truthfulness. Faithfulness requires human reading or a model judge.

**Human evaluation is unnecessary if ROUGE is high** — My ROUGE–F1 is 0.9, so my model is perfect — no need to look at the summaries.

ROUGE can be gamed by word choice; a fluent, misleading summary can score high. In fact: ROUGE is a cheap screen. Human review on a sample is the reference standard.

**ROUGE is easy to understand — it is just word counting** — I counted the words in common, so I have computed ROUGE.

ROUGE uses n-grams, where pairs and triples matter, and it uses recall, precision, and F1. In fact: ROUGE is a family of metrics; you need to specify the n-gram length and the aggregation (R, P, F1).


## In short

ROUGE is an automatic metric for summarization that counts overlap of n-grams between your summary and a reference. It is cheap and reproducible. But it does not measure meaning — only word overlap. Always pair it with human evaluation on a sample to catch what word counting misses.


## Where this is used

In large language models, ROUGE is used to evaluate summarization and generation benchmarks. A developer might run a fine-tuning job and compare the model's output against reference summaries from a dataset like CNN/DailyMail. They would compute ROUGE-1 and ROUGE-2 F1 scores before and after training — if the score drops, the model has regressed. In production, ROUGE is not usually the only metric, but it is a fast regression gate: every time you update a model, you run ROUGE over a fixed set of test cases. Humans then review a random sample of outputs to judge quality that ROUGE cannot measure — for instance, whether the summary is false or biased. For small language models running on phones, ROUGE is still used to evaluate if the smaller model summaries match the larger model's. Since the small model runs on-device, you cannot easily send everything to a human, so you first filter with ROUGE, and only sends suspicious or low-scoring cases for human review. In retrieval-augmented generation, ROUGE might seem odd because you are not generating a summary, but you can use it to evaluate whether the retrieved snippets contain the words the final answer uses — if the answer talks about 'flood' but the retrieved passages never mention 'flood', that is a red flag. In agentic systems that plan and call tools, ROUGE is less direct — agents write text like tool calls, which have a fixed format. You would use deterministic validators to check format, and ROUGE might evaluate the natural language summaries the agent produces after executing a tool. Human evaluation is even more important for agents, because the agent's reasoning cannot be measured by word overlap — you have to read the chain of steps to see if it made sense. In everyday applied machine learning, such as a news summarization app, the developer would set a ROUGE threshold (say, 0.3 on F1) below which a summary is flagged for human review. They would also curate a gold set — a collection of articles with human-written reference summaries — to run ROUGE on every new model version. The limitation is that ROUGE underestimates good summaries that paraphrase, so a team that relies only on ROUGE may reject better models. The right pattern, as your library notes say, is automatic screening plus human sampling.


---

Read this on the web: https://basavapurushottam.com/learn-ai/rouge-and-human-eval
