Skip to main content
Back to path

Welcome to a Grasp lesson

Create a free account to keep studying and create your own study path.

Lesson 8

Avoiding the R-squared Trap: Why More Variables Aren't Always Better

Recommended resources

In our last session, we established a framework for comparing models using Adjusted R-squared and the Residual Standard Error (RSE), emphasizing the principle of parsimony. Today, we'll focus on a critical exam topic that builds directly on this: understanding precisely why you must use Adjusted R-squared instead of the standard R-squared when comparing models. Mastering this distinction is crucial, as confusing the two is one of the most common and easily penalized mistakes in a regression exam.

This lesson will dissect the R-squared trap, explain the mechanics behind it, and equip you with the precise language to articulate why it's a flawed metric for model selection.

The R-squared Illusion: Why More is Not Always Better

A fundamental, mathematical property of R-squared makes it deceptive for model comparison. This is a point your professor will almost certainly test.

The Rule: When you add a new predictor variable to a model, the R-squared value will always increase or stay the same. It can never decrease.

This happens because the Ordinary Least Squares (OLS) method, which R uses to build the model, is designed to find the coefficients that minimize the sum of squared errors (SSE). When you introduce a new variable, OLS will do one of two things:

  1. If the variable has even a tiny, chance relationship with the response, OLS will assign it a small coefficient that slightly reduces the SSE, thus increasing R-squared.
  2. If the variable is completely useless, OLS can simply set its coefficient to zero, leaving the SSE unchanged.

In either case, SSE will not go up. Since R-squared is calculated as , and SST (Total Sum of Squares) is fixed for your dataset, a non-increasing SSE means a non-decreasing R-squared.

The following video provides an excellent and intuitive explanation of this mechanism.

Adjusted R Squared, Clearly Explained

This short clip from Super Data Science clearly explains the mechanics behind why R-squared is guaranteed to increase when new variables are added.

Watch from the beginning to the explanation of the problem. Pay close attention to the part where the narrator explains that if a new variable is useless, the OLS method will simply assign it a coefficient of zero, preventing the error from increasing.

This leads to the core problem: you can keep adding irrelevant predictors (like a student's shoe size to predict their exam score) and watch your R-squared value climb, giving you a false sense of having a "better" model.

Articulating the Trap: Your Exam Answer

This is a classic exam question setup. You might be asked to critique a model or explain why a colleague's reasoning is flawed.

Common Exam Trap: Mistaking a higher R-squared for a better model when comparing models with different numbers of predictors.

How to Spot It: You are presented with two models. Model B has more variables than Model A, and Model B also has a higher R-squared.

Your Exam-Style Answer:

"Relying on R-squared to compare these models is incorrect. R-squared is guaranteed to increase or stay the same whenever a new predictor is added, regardless of whether that predictor is statistically meaningful. This creates a misleading impression of model improvement and can lead to overfitting with unnecessarily complex models."

This point is stated very clearly in the resources.

Chapter 15 Linear regression

The textbook "Learning statistics with R" provides a concise academic explanation for this phenomenon.

In Section 15.4.3, read the first paragraph, "The adjusted R-squared value". The author explicitly states that adding more predictors "will always call the R^2 value to increase (or at least not decrease)." This is the key phrase for your exam.

The Solution: The Penalty of Adjusted R-squared

As we've discussed, Adjusted R-squared solves this problem by introducing a penalty for each variable added to the model. It "adjusts" the R-squared value downwards to account for the number of predictors.

Let's look at the formula from the LINK resource again:

Where is the number of observations and is the number of predictors.

When you add a predictor, increases. This makes the denominator smaller, which in turn makes the entire penalty fraction larger. Therefore, for Adjusted R-squared to increase, the drop in SSE from the new variable must be substantial enough to overcome this penalty. If the variable is useless, the small drop in SSE won't be enough, and your Adjusted R-squared will go down.

This image perfectly illustrates the difference in behavior.

This chart shows three models. Notice in Model 03, adding a predictor increased R-Squared from 0.09 (in Model 02) to 0.32, but the Adjusted R-Squared remained negative, signaling the new variable did not add real value relative to its complexity.

Let's see this demonstrated with a practical example from a DataCamp tutorial.

Adjusted R-Squared: A Clear Explanation with Examples

This article gives a concrete coding example where adding a predictor hurts the model according to Adjusted R-squared.

Navigate to the section "Adjusted r-squared in R". The author builds model1 with 4 predictors and gets an R-squared of 0.88527 and an Adjusted R-squared of 0.88229. They then build model2 by adding one more predictor. Notice the result: the R-squared barely increases to 0.88529, but the Adjusted R-squared decreases to 0.88154. This is a perfect demonstration of the penalty at work and proves the new variable was not helpful.

Practice Drill

You are reviewing the work of a junior analyst. They have built two models to predict employee satisfaction score (out of 100).

  • Model 1: Predicts Satisfaction using Salary.

    • R-squared = 0.45
    • Adjusted R-squared = 0.44

  • Model 2: Predicts Satisfaction using Salary, Age, and Commute_Time.

    • R-squared = 0.47
    • Adjusted R-squared = 0.42

The analyst recommends using Model 2 because "it explains more of the variance, as shown by its higher R-squared."

Your Task:
Write a 2-3 line response explaining the flaw in the analyst's logic and stating which model should be preferred.

Solution

Exam-Style Response:
The analyst's conclusion is flawed because they are incorrectly using R-squared to compare models with different numbers of predictors. R-squared will always increase with more variables, which is misleading. The correct metric, Adjusted R-squared, decreased from 0.44 to 0.42, indicating that the two additional variables in Model 2 do not add sufficient explanatory power to justify the increased model complexity. Therefore, the more parsimonious Model 1 is preferred.

Conclusion

You are now equipped to identify and dismantle one of the most common regression exam traps. Remember this simple hierarchy: R-squared describes fit for a single model, but Adjusted R-squared is required to compare models.

Key Takeaways for Your Exam:

  • The R-squared Trap: R-squared never decreases when you add predictors, even useless ones. Relying on it for model selection leads to overly complex models.
  • The Adjusted R-squared Solution: Adjusted R-squared penalizes the model for added complexity. It only increases if a new variable adds significant value.
  • The Decisive Argument: When comparing models, always justify your choice using Adjusted R-squared and RSE, not R-squared.

In the next lesson, we will dive into the ANOVA table. This will introduce the F-test, which moves beyond assessing fit to formally test the overall statistical significance of your entire regression model.