Skip to main content
Back to path

Welcome to a Grasp lesson

Create a free account to keep studying and create your own study path.

Lesson 15

OLS Model Diagnostics

Recommended resources

Introduction

In our previous lesson, we constructed a multiple linear regression model to predict a lingerie piece's Appeal_Score based on its design features. You learned how to interpret the model's coefficients to isolate the effect of specific choices, such as using a balconette style or increasing color saturation. However, building a model is only half the battle. A model's predictions and interpretations are only as reliable as the assumptions that underpin it.

This lesson focuses on the crucial next step: model diagnostics. We will investigate whether our OLS model is valid and trustworthy. Given your background in statistics, you'll recognize these assumptions as foundational to sound econometric and regression analysis. We will explore three common pitfalls that can invalidate our conclusions: multicollinearity, heteroscedasticity, and the non-normality of residuals. For each, we will discuss its practical implications in a design context, how to detect it using both visual plots and formal statistical tests, and what it means for the insights you derive about your lingerie styles.

Diagnosing Multicollinearity with VIF

The first assumption we'll check is that our predictor variables are not perfectly correlated with each other. When two or more predictors have a strong linear relationship, we have multicollinearity.

In our lingerie context, this could happen if we include highly related features in our model. For example, a bra's Fabric_Weight is likely highly correlated with its Cup_Coverage_Percentage—more coverage generally means more fabric and thus more weight. If both are in the model, it becomes difficult for the regression to disentangle their individual effects on the Appeal_Score. The model might attribute a positive effect to Fabric_Weight and a negative one to Cup_Coverage, or vice versa, with both having large standard errors. This makes the coefficients unstable and their interpretations unreliable.

The primary tool for diagnosing multicollinearity is the Variance Inflation Factor (VIF). The VIF for a predictor variable is a measure of how much the variance of its estimated coefficient is "inflated" because of its linear dependence on other predictors.

A common rule of thumb is:

  • VIF = 1: No correlation between the predictor and others.
  • 1 < VIF < 5: Moderate correlation, usually not a problem.
  • VIF > 5 or 10: High correlation that may be problematic, warranting further investigation.

The UCLA Stata guide on regression diagnostics provides an excellent, concise explanation of multicollinearity and the use of VIF.

Regression with Stata Chapter 2 – Regression Diagnostics

This guide explains the consequences of multicollinearity and demonstrates how to use VIF to detect it. While the commands are for Stata, the concepts and interpretation are universal.

Please read section 2.4, Checking for Multicollinearity. Focus on the description of the problem—unstable coefficients and inflated standard errors. Then, examine the example where the initial model has variables with very high VIFs. Pay close attention to how removing one redundant variable (avg_ed) dramatically reduces the VIFs and stabilizes the model, even making a previously non-significant coefficient significant. This demonstrates the practical benefit of this diagnostic check.

Checking for Heteroscedasticity

A key assumption of OLS is homoscedasticity, which means the variance of the error term () is constant across all levels of the predictor variables. In simpler terms, the model's predictive accuracy should be consistent regardless of the predicted outcome's value.

The opposite is heteroscedasticity, where the error variance is non-constant. Imagine our model predicting Appeal_Score. If the model's errors are small for low-scoring lingerie but become much larger and more spread out for high-scoring lingerie, we have heteroscedasticity. This often appears as a cone or fan shape in a plot of residuals versus fitted values.

This plot shows a classic example of heteroscedasticity. The spread of the residuals increases as the fitted value increases, forming a "fan shape."

Why is this a problem? Heteroscedasticity does not bias the OLS coefficient estimates themselves, but it renders the standard errors of those coefficients unreliable. Consequently, the t-tests and F-tests become invalid. You might incorrectly conclude that a design feature like Strap_Width_mm is statistically significant when it isn't, leading you to make design decisions based on spurious results.

Visual and Formal Tests

  1. Visual Inspection: The first step is always to plot the residuals against the predicted (fitted) values. A random, patternless cloud of points around the zero line suggests homoscedasticity. The plot below, for instance, shows residuals that are much more evenly scattered, indicating no obvious heteroscedasticity.
The "Residuals for LinearRegression Model" plot on the left shows a relatively random scatter of points around the zero line, which is characteristic of a homoscedastic model.

  1. The Breusch-Pagan Test: This is a formal statistical test to detect heteroscedasticity.

    • Null Hypothesis (): Homoscedasticity is present (error variance is constant).
    • Alternative Hypothesis (): Heteroscedasticity is present (error variance is not constant).

    If the test yields a small p-value (typically < 0.05), we reject the null hypothesis and conclude that our model suffers from heteroscedasticity.

The following video provides a clear explanation of the intuition behind the Breusch-Pagan test.

The Breusch Pagan test for heteroscedasticity

This video from Ben Lambert clearly explains the logic of the Breusch-Pagan test.

Watch from the beginning to understand why heteroscedasticity is a problem (the introduction). The core idea of the test is explained starting from this point, where the squared residuals from the original model are regressed on the independent variables. Finally, understand how the F-test and the LM (Chi-squared) statistic are used to test the null hypothesis of homoscedasticity (the testing procedure). This second regression is referred to as the "auxiliary regression".

For a hands-on view of the calculation, the video "Heteroscedasticity: Breusch-Pagan Test on Spreadsheet" by Pat Obi walks through the entire process step-by-step in Excel. The key steps are:

  1. Run the primary regression and save the residuals.
  2. Square the residuals.
  3. Run an auxiliary regression of the squared residuals on the original independent variables.
  4. Calculate the test statistic, from the auxiliary regression, which follows a Chi-squared distribution.

Checking for Normality of Residuals

The final assumption we'll examine is that the model's residuals are normally distributed. This assumption is not required for the OLS coefficient estimates to be unbiased, but it is essential for the validity of the t-tests and F-statistics that we use to assess the significance of our predictors. If the residuals deviate significantly from a normal distribution, the p-values associated with your coefficients can be misleading.

However, with large sample sizes (e.g., n > 30 or 40), the Central Limit Theorem often ensures that the sampling distribution of the coefficients is approximately normal, even if the residuals themselves are not. Therefore, minor deviations from normality in a large dataset are usually not a major concern.

Visual and Formal Tests

  1. Q-Q Plot: A quantile-quantile (Q-Q) plot is the best visual tool for checking normality. It plots the quantiles of your residuals against the theoretical quantiles of a normal distribution. If the residuals are normally distributed, the points will fall closely along the 45-degree reference line. The right-hand plot in the "Residuals for LinearRegression Model" image above is a Q-Q plot showing residuals that align well with the theoretical line, suggesting normality.

  2. Shapiro-Wilk Test: This is a formal test for normality.

    • Null Hypothesis (): The residuals are normally distributed.
    • Alternative Hypothesis (): The residuals are not normally distributed.

    A low p-value (< 0.05) indicates that you should reject the null hypothesis and conclude that the residuals are likely not from a normal distribution.

You can read about these checks in the same UCLA guide we used before.

Regression with Stata Chapter 2 – Regression Diagnostics

This section details both graphical and formal methods for assessing the normality of residuals.

Please read section 2.2, Checking Normality of Residuals. It clearly states when normality is required and demonstrates the use of a kernel density plot, P-P plot, and Q-Q plot for visual inspection. Also, note the mention of the Shapiro-Wilk test (swilk test) as a formal numerical check.

Numerical Exercise

You have fit a multiple linear regression model to predict the Appeal_Score (0-100) for a new collection of 45 lingerie sets. The predictors are Sheerness_pct (0-100), Underwire_Strength (a 1-10 scale), and Is_Seamless (1 for seamless, 0 for seamed).

After running the regression, your statistical software provides the following diagnostic information:

  1. Variance Inflation Factors (VIF):

    • Sheerness_pct: 1.2
    • Underwire_Strength: 8.9
    • Is_Seamless: 1.5

  2. Breusch-Pagan Test for Heteroscedasticity:

    • Test Statistic (Chi-squared): 9.52
    • p-value: 0.002

  3. Shapiro-Wilk Test for Normality of Residuals:

    • Test Statistic: 0.98
    • p-value: 0.45

Questions:

  1. Analyze the VIF values. Is multicollinearity a concern in this model? Why or why not?
  2. Interpret the result of the Breusch-Pagan test. What does this p-value tell you about the model's residuals? What would you expect a plot of residuals versus fitted values to look like?
  3. Interpret the result of the Shapiro-Wilk test. Should you be concerned about the normality of the residuals? How does this impact your confidence in the model's coefficient p-values?
  4. Synthesizing all three diagnostics, what is your overall assessment of this model's validity? What would be your most pressing concern to address next?
Click for a detailed solution.

  1. Multicollinearity Analysis: The VIF for Sheerness_pct (1.2) and Is_Seamless (1.5) are very low, indicating no collinearity issues with these variables. The VIF for Underwire_Strength is 8.9. This is above the more conservative threshold of 5 and approaching the higher threshold of 10. This suggests a moderate to high degree of multicollinearity associated with this variable. It's likely correlated with another predictor not included here, or perhaps one of the other predictors in the model. This is a concern because the coefficient for Underwire_Strength may be unstable.

  2. Heteroscedasticity Analysis: The p-value for the Breusch-Pagan test is 0.002, which is well below the standard significance level of 0.05. Therefore, we reject the null hypothesis of homoscedasticity. We conclude that there is statistically significant evidence of heteroscedasticity. I would expect a plot of residuals versus fitted values to show a non-random pattern, most likely a "fan" or "cone" shape, where the spread of the residuals either increases or decreases as the predicted Appeal_Score changes.

  3. Normality Analysis: The p-value for the Shapiro-Wilk test is 0.45. Since this is much greater than 0.05, we fail to reject the null hypothesis. This means there is no evidence to suggest the residuals are not normally distributed. This is good news, as it increases our confidence in the p-values for the coefficients (assuming the heteroscedasticity issue is resolved). The sample size (n=45) is also large enough that the Central Limit Theorem would provide some robustness.

  4. Overall Assessment: The model has a serious issue with heteroscedasticity, which invalidates the standard errors and thus all hypothesis tests (t-tests, F-test). While there's also a potential multicollinearity problem with Underwire_Strength, the heteroscedasticity is the most critical issue to address. The normality of residuals is not a concern. The most pressing next step would be to correct for heteroscedasticity, for example, by using robust standard errors (also known as Huber-White standard errors), which provide valid statistical tests even in the presence of heteroscedasticity.

Conclusion

In this lesson, we moved beyond model creation to the critical process of model validation. By checking the core assumptions of OLS regression, you can ensure that your interpretations are statistically sound and that the design insights you generate are reliable.

Key Takeaways:

  • Multicollinearity inflates standard errors and makes coefficients unstable. It is diagnosed using the Variance Inflation Factor (VIF).
  • Heteroscedasticity means the error variance is not constant, which invalidates standard errors and hypothesis tests. It is diagnosed visually with residual plots and formally with the Breusch-Pagan test.
  • Non-normality of residuals can affect the validity of p-values, especially in small samples. It is checked with Q-Q plots and the Shapiro-Wilk test.

So far, we have focused on predicting a continuous outcome (Appeal_Score). However, often the goal is simpler: is a piece a "top pick" or not? This is a binary classification problem. In our next lesson, we will introduce logistic regression, a powerful technique for modeling the probability of a binary outcome based on a set of predictor variables.