Evaluating Regression Models: MSE, RMSE, MAE, and R-squared

1. The Foundation of Evaluation: Understanding Errors (Residuals)

Before we dive into specific metrics, we need to be crystal clear on what we're measuring. Every evaluation metric for regression is built upon the concept of error, also known as the residual.

The residual is simply the difference between the actual value and the value your model predicted for a given data point.

A small residual means your model's prediction was close to the truth; a large residual means it was far off. The goal of our evaluation metrics is to aggregate all these individual residuals across your dataset into a single, informative score.

To get a solid visual and conceptual grasp of residuals, the following article provides an excellent explanation.

2. Metrics Based on Error Magnitude: MAE, MSE, and RMSE

The first family of metrics we'll explore directly measures the average size of the errors your model makes. They answer the question: "On average, how far off are my model's predictions?"

The following video provides a clear, step-by-step explanation of these three related metrics.

Let's summarize the key characteristics of each metric from the video:

• Mean Absolute Error (MAE):

  • Calculation: The average of the absolute values of the residuals.
    $\text{MAE}=\frac{1}{n}{\sum }_{i=1}^n|y_i-{\hat{y}}_i|$
  • Interpretation: It represents the average error in your predictions. A MAE of 500, as in the video's example, means that on average, the model's prediction is off by 500 rooms.
  • Key Feature: It's in the same units as the target variable, making it very easy to explain to non-technical stakeholders.

• Mean Squared Error (MSE):

  • Calculation: The average of the squared residuals.
    $\text{MSE}=\frac{1}{n}{\sum }_{i=1}^n(y_i-{\hat{y}}_i{)}^2$
  • Interpretation: The units are squared (e.g., "squared rooms"), which makes direct interpretation difficult. A lower MSE is better, but the number itself isn't intuitive.
  • Key Feature: Squaring the errors means that large errors are weighted much more heavily than small ones. A single very wrong prediction will have a much bigger impact on MSE than on MAE. This is useful when large errors are particularly undesirable.

• Root Mean Squared Error (RMSE):

  • Calculation: The square root of the MSE.
    $\text{RMSE}=\sqrt{\frac{1}{n}{\sum }_{i=1}^n(y_i-{\hat{y}}_i{)}^2}$
  • Interpretation: Like MAE, it is in the same units as the target variable. An RMSE of 810 means the "typical" or standard deviation of the prediction error is 810 rooms.
  • Key Feature: It offers a balance: it's interpretable like MAE but retains MSE's sensitivity to large errors. It is one of the most commonly used metrics in regression.

3. The "Goodness of Fit" Metric: R-squared (R²)

While MAE and RMSE tell you the magnitude of the error, they don't tell you if that error is large or small in the context of the overall data. An RMSE of $5,000 might be terrible for predicting the price of a car but excellent for predicting the price of a house.

R-squared (R²), or the Coefficient of Determination, provides this context. It tells you what proportion of the variance in the target variable is explained by your model.

This concept is best understood with a clear, intuitive explanation.

Here’s the essence of R-squared:

• Scale: It ranges from 0 to 1 (though it can be negative if your model is worse than just predicting the mean).
• Interpretation: An R² of 0.81 means your model explains 81% of the total variation in the target data. The remaining 19% is unexplained by your model's features.
• A higher R² is generally better. An R² of 1 means a perfect fit, while an R² of 0 means your model is no better than just guessing the average value every time.

4. Practical Application with Scikit-Learn

You now understand the theory behind these four crucial metrics. Let's see how to calculate them using Python and scikit-learn. It's surprisingly straightforward.

The following article provides a handy reference with the exact sklearn.metrics functions you'll need.

To bring this all together, here is a single code block showing how you would evaluate a model's predictions in practice. Assume y_test holds the true values and y_pred holds your model's predictions.

from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
import numpy as np




# Let's imagine these are our true values and predicted values for house prices
y_test = np.array([300000, 450000, 550000, 780000])
y_pred = np.array([320000, 430000, 580000, 750000])




# Calculate MAE
mae = mean_absolute_error(y_test, y_pred)
print(f"Mean Absolute Error (MAE): ${mae:,.2f}")




# Calculate MSE
mse = mean_squared_error(y_test, y_pred)
print(f"Mean Squared Error (MSE): {mse:,.2f} (in dollars-squared, not intuitive)")




# Calculate RMSE
# You can take the sqrt of MSE or use a built-in argument
rmse = np.sqrt(mse) 



# Or: rmse = mean_squared_error(y_test, y_pred, squared=False)
print(f"Root Mean Squared Error (RMSE): ${rmse:,.2f}")




# Calculate R-squared
r2 = r2_score(y_test, y_pred)
print(f"R-squared (R²): {r2:.2f}")

Interpreting this example output:

• MAE: On average, our model's price prediction is off by $25,000.
• RMSE: The typical prediction error is about $25,500. It's slightly higher than MAE because it's more sensitive to the larger errors (like the $30,000 error on the third house).
• R²: Our model explains 96% of the variance in the house prices. This indicates a very strong fit.

Conclusion

Great work! You can now not only build a regression model but also rigorously evaluate its performance. This is a massive step forward in your practical data science skills.

Key Takeaways:

• Residuals ($y-\hat{y}$) are the building blocks of all regression metrics.
• MAE gives you the average error in the target's original units, treating all errors equally. It's easy to interpret.
• MSE penalizes large errors heavily but is in squared units, making it hard to interpret directly.
• RMSE is the best of both worlds: it's in the original units like MAE but is sensitive to large errors like MSE. It's often the default choice.
• R-squared measures the proportion of variance your model explains, providing a "goodness of fit" score between 0 and 1.

Preview of the next lesson:

So far, we've only used a single feature to make predictions (e.g., predicting price from area). But what if we have more information, like the number of bedrooms or the age of the house? In the next lesson, you will learn to fit a multiple linear regression model to leverage multiple features at once. Your new evaluation skills will be essential for determining if adding more features actually improves your model's performance.