Hello! Welcome to the final lesson in our Natural Language Processing module.
In our previous lesson, we learned how to use and fine-tune powerful Transformer models from the Hugging Face ecosystem for text classification and Named Entity Recognition. We used accuracy as our evaluation metric, which works well for many classification tasks. However, NLP encompasses a wide range of generative tasks, like translation, summarization, and language modeling, where a simple right/wrong metric doesn't capture the quality of the output.
Today, we will equip you with the tools to properly evaluate these more complex tasks. Our learning outcome is to explain and apply evaluation metrics for NLP tasks, specifically perplexity, BLEU, and ROUGE.
- Perplexity is used to evaluate how well a language model predicts text.
- BLEU is the standard for evaluating machine translation.
- ROUGE is commonly used for evaluating automatic text summarization.
Understanding these metrics is crucial for building and comparing high-quality NLP systems.
1. Perplexity: How "Confused" is Your Language Model?
Language models, like the ones we'll explore in the next module, work by assigning probabilities to sequences of words. A good language model should assign high probability to plausible sentences and low probability to nonsensical ones. Perplexity is a metric that quantifies this.
But why can't we just use accuracy? Let's start by understanding the problem.
How you can use Perplexity to see how good a Language Model is [Lecture]
First, let's understand why we need a special metric for language models. Traditional accuracy isn't a good fit. This video explains why and sets the stage for perplexity.
Watch the first part of the video from 00:22 to 02:02. Focus on the example of 'I came home from a long day of work and sat on my ____' and the idea that we need to compare probability distributions, not just single 'correct' answers.
The Theory Behind Perplexity
As the video highlighted, we need to compare the model's output probability distribution to the true distribution of the language. Perplexity is derived from cross-entropy, a concept from information theory that measures the "distance" between two probability distributions.
For a language model, cross-entropy measures how well the model's predicted probabilities for a sequence of words match the actual sequence. The formal definition of perplexity is simply the exponential of the cross-entropy loss.
Perplexity =
Intuitively, you can think of perplexity as a measure of a model's "confusion." If a model has a perplexity of 50 for a given text, it means that at each step, it's as uncertain as if it were choosing between 50 different words. Therefore, a lower perplexity score is better, indicating the model is more confident and accurate in its predictions.
Calculating Perplexity
Let's walk through a concrete example to see how the math works.
LLM Evaluation metrics explained
The article 'LLM Evaluation metrics explained' breaks down the formula with a simple example. This will help solidify the connection between the theory and a practical result.
Read the section titled 'Perplexity'. Focus on the step-by-step calculation for the sentence 'The cat sat on the mat.'
Applying Perplexity in Python
In practice, you won't calculate this by hand. The Hugging Face transformers library makes it straightforward to compute perplexity. The library's models can return the loss (which is the cross-entropy) when you provide labels. You can then take the exponential of this loss to get the perplexity.
Understanding LLM Benchmarks with Python
Now for the practical part. This GitHub guide shows how to calculate perplexity for a given text using a specific model.
Look at the code block under 'Slide 8: Perplexity Metric'. Notice how it loads a GPT-2 model and tokenizer, then computes the perplexity by calculating the exponential of the model's loss (torch.exp(loss)). The example comparing 'The cat sat on the mat' and 'The cat sat on the cloud' is a great illustration of how a more plausible sentence gets a lower perplexity score.
2. BLEU Score: Evaluating Machine Translation
When evaluating machine translation, we need to compare the model's generated translation (the "candidate") with one or more high-quality human translations (the "references"). The BLEU (Bilingual Evaluation Understudy) score is the industry standard for this.
BLEU's core idea is to measure the overlap of n-grams (sequences of n words) between the candidate and reference translations.
This short video from Hugging Face explains the core concepts of BLEU brilliantly.
Watch from 00:05 to 03:05. Pay close attention to: The idea of comparing a generated 'candidate' to human-written 'references'. How simple unigram precision can be fooled by repetitive text. The concept of 'modified precision' (clipping), which prevents a model from getting a high score by repeating words. How using higher n-grams (like bigrams or 4-grams) helps evaluate word order.
The BLEU Formula
The final BLEU score combines two key elements:
- Modified N-gram Precision: It calculates the precision for different n-grams (usually 1 through 4) and combines them using a geometric mean.
- Brevity Penalty (BP): It penalizes candidate translations that are too short compared to the references. A model could achieve high precision by generating only a few correct words, and the BP corrects for this.
LLM Evaluation metrics explained
The video gave us the high-level idea. To complete our understanding, let's look at the 'Brevity Penalty' and the final formula. The 'LLM Evaluation metrics explained' article provides a clear breakdown.
Read the 'BLEU score' section. Focus on the 'Brevity Penalty (BP)' explanation and the final formula that combines the BP with the precisions of different n-grams.
Applying BLEU in Python
You can easily calculate BLEU using libraries like nltk. This makes it simple to integrate into your model evaluation pipeline.
Understanding LLM Benchmarks with Python
This example from the GitHub guide demonstrates how to calculate BLEU in Python.
Examine the code under 'Slide 6: BLEU Score for Machine Translation'. Note how sentence_bleu from NLTK is used to compare a candidate sentence against a reference. The example shows how a semantically similar sentence gets a higher score than one with a different meaning, even if both are grammatically correct.
3. ROUGE: Evaluating Text Summarization
For text summarization, we use a metric called ROUGE (Recall-Oriented Understudy for Gisting Evaluation). While BLEU is precision-focused (did the n-grams in the candidate appear in the reference?), ROUGE is recall-focused: did the n-grams from the reference appear in the candidate? This is a better fit for summarization, where the goal is to capture the main points of the original text.
ROUGE is not a single metric but a family of them.
LLM Evaluation metrics explained
The article 'LLM Evaluation metrics explained' gives an excellent overview of the most common ROUGE variants with clear examples.
Read the entire 'ROUGE Score' section. It's detailed, but focus on understanding the differences between: ROUGE-N: Measures the overlap of n-grams (e.g., ROUGE-1 for unigrams, ROUGE-2 for bigrams). ROUGE-L: Measures the Longest Common Subsequence (LCS), which rewards in-order matches even if they aren't consecutive. Don't worry about memorizing the formulas for ROUGE-W and ROUGE-S; just understand that they are more advanced variants.
Applying ROUGE in Python
Just like BLEU, ROUGE is easy to compute using dedicated Python libraries. The rouge library is a popular choice.
Understanding LLM Benchmarks with Python
Let's see how to implement ROUGE in Python. The GitHub guide provides a concise example.
Review the code block under 'Slide 7: ROUGE Metric for Text Summarization'. Notice how the rouge.get_scores function returns scores for rouge-1, rouge-2, and rouge-l all at once. Each metric gives you a precision, recall, and F1-score.
When analyzing ROUGE scores, you'll typically look at all three variants (ROUGE-1, ROUGE-2, ROUGE-L). A high ROUGE-1 score suggests good overlap of key terms, while a high ROUGE-L score indicates the summary follows the sentence structure of the original text well.
Conclusion
You have now learned how to evaluate a broad range of NLP models beyond simple classification. Measuring performance correctly is just as important as building the model itself.
Key Takeaways:
- Perplexity measures how well a language model predicts a sequence of text. It's the exponential of the cross-entropy loss, and lower is better.
- BLEU is a precision-based metric for machine translation that compares n-gram overlap between a candidate and reference translations, with a penalty for being too short. Higher is better.
- ROUGE is a recall-based metric for text summarization that measures the overlap of n-grams (
ROUGE-N) and the longest common subsequence (ROUGE-L) between a summary and a reference. Higher is better. - For all these metrics, practical implementation is made easy by libraries like
transformers,nltk, androuge.
Preview of the Next Lesson:
This lesson concludes our module on Natural Language Processing. You've journeyed from text preprocessing and embeddings to building with Transformers and now, to evaluating them. In the next module, "Large Language Models and Generative AI," we will dive deep into the models that power applications like ChatGPT. Our first lesson will focus on autoregressive text generation and decoding strategies, where your understanding of perplexity will be directly applicable.