Diagnosing Regression Assumptions with Residual Plots

What are Residuals and Why Plot Them?

Before we can diagnose problems, we need to understand our main diagnostic tool: the residual.

A residual is simply the difference between the actual observed value and the value your model predicted.

$\text{Residual}=\text{Observed value}-\text{Predicted value}$
$e_i=y_i-{\hat{y}}_i$

If a residual is positive, the model's prediction was too low. If it's negative, the prediction was too high. A residual of zero means the model predicted the value perfectly.

We can learn a lot by plotting these residuals. The most common and useful type of residual plot is one that shows the residuals on the y-axis versus the predicted (or fitted) values on the x-axis.

Why do we plot residuals against the predicted values ($\hat{y}$) instead of the original predictor variable ($x$)? The following video explains this clearly. It's especially important in multiple regression, where you have many predictors. Plotting against the single vector of predicted values allows us to check the model's overall performance in one simple 2D plot.

In a well-behaved model, the residual plot should look like a random cloud of points with no discernible pattern, centered on zero. This indicates that our model has captured the underlying relationship in the data, and what's left over (the residuals) is just random, unpredictable noise.

Diagnosing Key Assumption Violations

When the residual plot does not look like a random cloud, it's sending us a signal that one or more of the linear regression assumptions have been violated. Let's look at the most common culprits.

1. The Linearity Assumption

• The Assumption: The true relationship between the predictors and the outcome variable is linear.
• The Symptom of Violation: A clear pattern in the residual plot. The most common is a U-shape or an inverted U-shape (a "frowny face"). This tells you the model is systematically over-predicting for some ranges of values and under-predicting for others.
• The Consequence: If the relationship isn't linear, your linear model is fundamentally misspecified. The coefficients are biased and cannot be trusted to explain the relationship between variables.

The following video provides an intuitive explanation of this and other assumptions. We'll start by focusing on linearity.

As the video explains, seeing a curve in your residuals is a strong sign that you may need to either transform your variables or consider a non-linear model.

2. The Homoscedasticity Assumption (Constant Variance)

• The Assumption: The residuals have a constant variance at every level of the predictors. This is called homoscedasticity (a mouthful, but it just means "same scatter").
• The Symptom of Violation: The spread of the residuals is not constant across the plot. A common pattern is a fan or cone shape, where the residuals become more (or less) spread out as the predicted values increase. This violation is called heteroscedasticity ("different scatter").
• The Consequence: The model's coefficients are still unbiased, which is good. However, the standard errors of the coefficients are now unreliable. This means any p-values and confidence intervals are wrong, so you can't confidently perform hypothesis tests on the coefficients (e.g., to determine if a predictor is statistically significant).

Now let's watch the next section of the same video, which covers heteroscedasticity.

3. The Normality of Errors Assumption

• The Assumption: The residuals are normally distributed.
• The Symptom of Violation: This is harder to see in a standard residual plot. Instead, we use a histogram of the residuals or, more formally, a Quantile-Quantile (Q-Q) plot. In a Q-Q plot, the quantiles of the residuals are plotted against the theoretical quantiles of a normal distribution. If the residuals are normally distributed, the points will fall closely along a straight diagonal line.
• The Consequence: This is often considered the least critical assumption, especially with larger sample sizes (thanks to the Central Limit Theorem). If the sample size is small and the errors are far from normal, it can affect the validity of confidence intervals and hypothesis tests.

The webpage below has an excellent gallery of unhealthy residual plots. It's a great practical guide to recognizing these patterns in the wild.

A Complete Diagnostic Dashboard

In practice, data science tools often provide a dashboard of multiple diagnostic plots at once. Let's look at a common set of four plots produced by statistical software.

Let's break down what each plot tells us:

1. Residuals vs Fitted: This is the plot we've been focusing on. We look for non-linearity (curves) and heteroscedasticity (fan shapes). Here, we see a distinct U-shape, strongly suggesting the linearity assumption is violated.
2. Normal Q-Q: This is the plot for checking the normality of residuals. The points should fall on the dashed line. Here, the points deviate significantly at both ends, indicating the residuals have "heavier tails" than a normal distribution. The normality assumption is violated.
3. Scale-Location: This plot is similar to the Residuals vs Fitted plot, but it uses the square root of the standardized residuals on the y-axis. This can make it easier to see trends in the spread of residuals. The upward curve here again suggests heteroscedasticity (non-constant variance).
4. Residuals vs Leverage: This plot helps identify influential data points. Points with high leverage (far to the right on the x-axis) and/or large residuals (far from zero on the y-axis) can have a strong influence on the regression line. Points outside the red dashed lines (Cook's distance) are often considered influential.

By looking at these four plots together, we get a comprehensive picture. For the model that generated these plots, the verdict is clear: a simple linear regression is not a good fit for this data.

Conclusion

In this lesson, you've learned how to move beyond simply fitting a model to critically assessing its validity. Using residual plots, you can diagnose fundamental issues that would otherwise go unnoticed.

Key Takeaways:

• A residual is the error of a prediction: $e_i=y_i-{\hat{y}}_i$.
• A residual plot (residuals vs. fitted values) should look like a random cloud of points around zero.
• A curved pattern (e.g., a U-shape) signals a violation of the linearity assumption.
• A fan or cone shape in the plot signals heteroscedasticity (a violation of the constant variance assumption).
• A Q-Q plot is used to check the normality of residuals, where points should fall along a straight line.
• Regression diagnostics are an essential step to ensure your model's results are reliable and trustworthy.

Preview of the Next Lesson:

We've now seen that a simple linear model can sometimes be too simple. When we diagnose a problem like non-linearity, one potential solution is to make our model more complex (for example, by adding polynomial terms like $x^2$). However, increasing model complexity comes with its own risks. In our next lesson, we will explore one of the most fundamental concepts in machine learning: the bias-variance tradeoff. This will give you the conceptual framework to understand the costs and benefits of making models more or less complex.