In our last session, we honed in on the concept of a single residual—the prediction error for one specific data point. Now, we'll scale up from that micro-view to the macro-view. Instead of asking "How wrong was this one prediction?", we will ask, "How good is the model overall?". This lesson opens our second module by introducing the three key metrics you'll use to answer that question: the Residual Standard Error (RSE), R-squared, and Adjusted R-squared. Mastering these is crucial for assessing model fit and for one of the most common exam tasks: comparing competing models.
Let's begin by examining a typical R regression output. We will refer back to this image throughout the lesson.
1. Residual Standard Error (RSE): The Typical Prediction Error
In the last lesson, we saw that every prediction has some error, or residual. The Residual Standard Error (RSE) aggregates all these individual errors into a single number that represents the typical or average size of a residual.
You can think of it as the standard deviation of the residuals. It tells you, on average, how far your model's predictions are from the actual observed values. The RSE is always in the same units as your response variable (e.g., dollars, kilograms, etc.).
The following article provides a great intuitive buildup to the RSE.
Residual Standard Deviation/Error: Guide for Beginners – QUANTIFYING HEALTH
This article from QUANTIFYING HEALTH builds the concept of RSE from the ground up, starting with a visual and moving to the formula and interpretation.
Read the following parts to get a full picture: The introduction, up to the section titled "vs residual standard error". This will give you the graphical intuition. The section titled "vs residual standard error vs RMSE". Focus on how the formula accounts for the number of predictors estimated (df = n - 2 for SLR, df = n - 3 for a two-predictor model, etc.). This df is the "degrees of freedom" we'll see in the ANOVA table later. The section on interpretation, which gives a clear example using blood pressure and BMI. Finally, read the short discussion on what constitutes a 'good' RSE. The key takeaway is that 'good' is context-dependent.
What to Look For in R Output:
In the R output image, find the line that says: Residual standard error: 6283000 on 334 degrees of freedom.
- The Value: The RSE is 6,283,000.
- The Units: Since the model predicts
salary, this value is in dollars. - The Interpretation: Our model's salary predictions are typically off by about $6,283,000, on average.
How to Interpret (Exam-Style Sentence):
The Residual Standard Error (RSE) is $6.283M. This means that, on average, the model's predictions for player salary differ from the actual salaries by approximately $6.283 million. A lower RSE indicates a better model fit.
2. R-squared (): The Proportion of Variance Explained
While RSE tells us about the absolute size of the error, R-squared () tells us how much of the "story" our model is actually explaining. It measures the proportion of the total variation in the response variable (Y) that is accounted for by the regression model.
It is a value between 0 and 1 (or 0% and 100%).
- An of 0 means the model explains none of the variability.
- An of 1 means the model explains all of the variability.
This video gives a great visual intuition for what R-squared represents.
Linear Regression Summary in R
This clip from the channel Dataslice provides a simple and effective visual for understanding R-squared.
Watch the segment from this video. Notice how the high R-squared example has points tightly clustered around the line, while the low R-squared example shows points scattered widely, indicating the predictor is less effective.
What to Look For in R Output:
In the R output image, find the line that says: Multiple R-squared: 0.3737.
- The Value: The R-squared is 0.3737, or 37.37%.
How to Interpret (Exam-Style Sentence):
The R-squared value is 0.3737. This indicates that approximately 37.4% of the variability in NBA player salaries can be explained by the number of points they score. The remaining ~62.6% is unexplained by this model.
3. Adjusted R-squared and the "More Variables" Trap
A major problem with R-squared is that it will always increase (or stay the same) when you add more predictor variables to the model, even if those new variables are completely useless. This can trick you into building an overly complex model that suffers from "overfitting."
This is a critical exam trap. Adding a variable might bump your from 70% to 71%, but it could be adding pure noise and making your model worse at predicting new data.
Adjusted R-squared solves this problem. It is a modified version of R-squared that "penalizes" the model for having extra predictors that don't add significant value.
- If you add a useful predictor, Adjusted will increase.
- If you add a useless predictor, Adjusted will decrease.
This makes Adjusted R-squared the correct tool for comparing models with different numbers of predictors.
Linear Regression Summary in R
The same video from Dataslice continues to give a perfect explanation of this exact concept.
Watch the next segment from the video. The key idea to focus on is that Adjusted R-squared "controls against this increase and it adds penalties each time you add another predictor to the model."
What to Look For in R Output:
In the R output image, right next to R-squared, you see: Adjusted R-squared: 0.3718.
- In a simple linear regression with only one predictor, the R-squared and Adjusted R-squared will be very close. The difference becomes significant when you start comparing models.
Rule for Model Comparison (Exam-Style Justification):
When comparing two or more models, you should choose the one with the highest Adjusted R-squared and the lowest Residual Standard Error (RSE). Do not use Multiple R-squared for this comparison.
Practice Drill
You are a data analyst for a real estate company. You have built two models to predict house prices.
Model A (Simple Linear Regression):Price = β₀ + β₁ * Square_Footage
- Residual Standard Error (RSE): $45,000
- Multiple R-squared: 0.65
- Adjusted R-squared: 0.64
Model B (Multiple Linear Regression):Price = β₀ + β₁ * Square_Footage + β₂ * Number_of_Bedrooms
- Residual Standard Error (RSE): $38,000
- Multiple R-squared: 0.72
- Adjusted R-squared: 0.70
Your Task:
- Write a one-sentence interpretation of the R-squared for Model A.
- Your colleague suggests that since Model B has a higher R-squared (0.72 vs 0.65), it is automatically better. Explain the flaw in this logic.
- Which model would you choose? Justify your answer in 2-3 sentences, referencing the appropriate metrics.
Solution
-
Interpretation of R-squared for Model A:
Approximately 65% of the variation in house prices can be explained by the house's square footage. -
Flaw in Colleague's Logic (The Exam Trap):
Comparing Multiple R-squared between models with different numbers of predictors is misleading because R-squared will almost always increase when a new variable is added, regardless of its utility. We must use Adjusted R-squared, which penalizes for model complexity. -
Model Choice and Justification:
I would choose Model B. Model B is preferable because it has a substantially higher Adjusted R-squared (0.70 vs. 0.64), indicating it explains more of the price variation after accounting for model complexity. Furthermore, its Residual Standard Error is significantly lower ($38,000 vs. $45,000), meaning its predictions are, on average, more accurate.
Conclusion
In this lesson, we established the core toolkit for evaluating and comparing regression models. You now have a framework for moving beyond individual coefficients and assessing a model's overall performance.
Key Takeaways for Your Exam:
- Residual Standard Error (RSE): The typical prediction error, measured in the units of the response variable. Lower is better.
- R-squared (): The percentage of variance in Y explained by the model. Useful for describing the fit of a single model.
- Adjusted R-squared (): A modified R-squared that penalizes for adding non-useful predictors. Use this to compare models with different numbers of variables. Higher is better.
- Model Comparison Rule: The preferred model is generally the one with the highest Adjusted and lowest RSE.
In our next lesson, we will dive into the ANOVA table. This will introduce another critical metric, the F-statistic, which tests whether your model as a whole is statistically significant—that is, whether it's better than just guessing the average value for every prediction.