Evaluating Text Classifier Performance with F1-Score and Per-Class Metrics

Interpreting the Training Logs

After calling trainer.train(), the Hugging Face Trainer provides a real-time log of the model's progress. You would have seen an output similar to the one below, with metrics calculated on your validation set at the end of each epoch (or at specified step intervals).

Let's break down these columns:

• Step: The number of training batches processed.
• Training Loss: The value of the loss function on the training data. You want to see this consistently decrease, indicating the model is learning.
• Validation Loss: The loss on the unseen validation data. This is a more honest measure of generalization. If it starts to increase while training loss decreases, your model is likely overfitting.
• Accuracy, F1, Precision, Recall: These are the evaluation metrics calculated on the validation set, as defined in your compute_metrics function. The F1-score is your primary benchmark, and your goal is to exceed 0.70.

These logs provide the first verdict on your model's performance. But to truly understand the story behind these numbers, we need to go deeper.

Why F1-Score? The Precision-Recall Trade-Off

You might wonder why we prioritize the F1-score over the more intuitive metric of accuracy. The reason is class imbalance. In your dataset, you will likely have far more complaints about "Poor" quality meals than "Good" ones. A model that simply learns to always predict "Poor" could achieve high accuracy while being completely useless.

The F1-score provides a more robust measure by balancing two competing metrics: precision and recall.

• Precision: Of all the complaints the model labeled as "Poor," how many were actually "Poor"? (Measures correctness of positive predictions).
• Recall: Of all the actual "Poor" quality complaints in the dataset, how many did the model successfully identify? (Measures how many positives the model "catches").

This video offers an excellent deep dive into these concepts and demonstrates why accuracy can be a deceptive metric in the face of class imbalance.

For your project, this trade-off is crucial. A model with high recall for the "Poor" class will catch most violation reports, but if its precision is low, it may incorrectly flag many adequate meals, creating false alarms. The F1-score is the harmonic mean of precision and recall, giving you a single number that summarizes this balance.

Generating a Detailed Classification Report

The overall F1-score gives you a great summary, but it doesn't tell you if the model is struggling with a specific complaint category. Is it good at identifying "Poor" quality but frequently misclassifies "Adequate" as "Good"? To diagnose this, we need a per-class breakdown of metrics. The classification_report from the scikit-learn library is the standard tool for this.

There are two primary ways to generate this report.

1. During Training (for live diagnostics)

You can modify your compute_metrics function to print the classification report at each evaluation step. This gives you a live view of how per-class performance evolves. The following Stack Overflow answer provides the exact code for this.

By adding print(classification_report(labels, predictions)) to your function, your training logs will now include a detailed table like this after each evaluation run:

              precision    recall  f1-score   support

        Good       0.85      0.79      0.82       150
    Adequate       0.65      0.75      0.70       200
        Poor       0.92      0.88      0.90       350

    accuracy                           0.81       700
   macro avg       0.81      0.81      0.81       700
weighted avg       0.82      0.81      0.81       700

2. After Training (for final evaluation)

A cleaner approach for final reporting is to evaluate your fully trained model on a hold-out test set. You can load your saved model into a Hugging Face pipeline, get predictions for your entire test set, and then generate the report.

This method is also demonstrated in the same Stack Overflow resource.

Interpreting the Classification Report

Let's dissect the report's components, using the example table above.

• Support: The number of actual occurrences of each class in the dataset. Here, we see a class imbalance: there are 350 "Poor" complaints but only 150 "Good" ones.
• Per-Class Metrics: You get precision, recall, and F1-score for each individual class. We can see the model is best at the "Poor" class (F1-score of 0.90) and weakest at the "Adequate" class (0.70). This tells you exactly where to focus any model improvement efforts.
• Macro Avg: The average of the metric (e.g., precision) across all classes, treating each class equally. This is a useful metric if you care about performance on rare classes just as much as on frequent ones.
• Weighted Avg: The average of the metric, weighted by the support of each class. This score will be closer to the performance of the more frequent classes. For your imbalanced dataset, this weighted F1-score is often the best single number to report for overall performance.

The video you watched earlier also provides a great walkthrough of these average types.

By analyzing the classification report, you can move from a simple statement like "the model has an F1-score of 0.81" to a much more insightful analysis: "The model demonstrates strong performance in identifying high-priority 'Poor' quality complaints (F1=0.90) but struggles to distinguish 'Adequate' complaints (F1=0.70), suggesting confusion between intermediate categories." This level of detail is exactly what's expected in a Master's level project report.

Conclusion

In this lesson, you've learned how to properly evaluate your text classification model, moving from a surface-level F1-score to a deep, diagnostic analysis.

Here are your key takeaways:

• The training logs from the Trainer provide an essential, high-level view of how your model's performance evolves during training.
• F1-score is a more reliable metric than accuracy for imbalanced datasets because it balances the trade-off between precision and recall.
• The sklearn.metrics.classification_report is the standard tool for a detailed, per-class performance breakdown.
• You can generate this report during training for live diagnostics or after training on a test set for final evaluation.
• Interpreting the macro vs. weighted averages in the context of your dataset's class balance is crucial for a nuanced performance assessment.

You are now fully equipped to evaluate and report on your text classification baseline. In our next lesson, we will pivot to the second modality of your project: images. You'll learn the fundamentals of object detection and how to run a pretrained YOLO model to identify food items in your meal photos, laying the groundwork for your image-based quality analysis.