Predictive Modeling with Multiple Regression

From a Line to a Plane: The Concept of Multiple Regression

In simple regression, we fit a "line of best fit" to our data in two dimensions (one X, one Y). When we add a second independent variable (X2), we are no longer fitting a line, but a two-dimensional plane in a three-dimensional space. If we have more than two X's, we fit a hyperplane in a multi-dimensional space. While we can't visualize this easily, the principle remains the same: we are finding the best-fitting surface to model the relationship between our inputs and the single output.

The mathematical equation is a straightforward extension of the simple linear model:

$Y={\beta }_0+{\beta }_1{X}_1+{\beta }_2{X}_2+\cdots +{\beta }_k{X}_k+ϵ$

Here:

• $Y$ is still our single dependent variable (the quality outcome).
• $X_1,X_2,\dots ,X_k$ are the multiple independent variables (the process parameters).
• ${\beta }_0$ is the intercept.
• ${\beta }_1,{\beta }_2,\dots ,{\beta }_k$ are the coefficients for each independent variable. Each $\beta$ represents the average change in $Y$ for a one-unit increase in its corresponding $X$, while holding all other independent variables constant.
• $ϵ$ is still the random error.

Developing a Multiple Regression Model in Excel

Since you prefer a hands-on approach, we'll continue using Excel to build our model. The process is very similar to what you did in the last lesson, but with a crucial additional step: refining the model to include only the significant predictors. A big part of "developing" a model isn't just running the analysis once, but iterating to find the most effective and efficient (or parsimonious) model.

The following video will walk you through a practical factory example: predicting total factory cost based on the quantities of different products being manufactured.

This iterative process of removing insignificant variables is a core practice in statistical modeling. It strengthens your model by eliminating "noise" and focusing only on the "vital few" inputs that truly drive the output.

Interpreting the Multiple Regression Output

The interpretation of the output is largely the same as for simple regression, but with one new, important metric: Adjusted R-squared.

Let's review the key components using the learnleansigma.com guide as a reference.

Here's a summary of the interpretation flow:

1. Check Overall Model Significance (ANOVA Table): Look at the Significance F value first. If it's less than 0.05, your model as a whole is statistically significant. It's better than nothing! If it's not, you need to stop and reconsider your variables.

2. Check Individual Predictor Significance (Coefficients Table): Examine the p-value for each independent variable. If a variable's p-value is greater than 0.05, it means that variable is not a significant predictor in the presence of the other variables in the model. This is what you use to decide which variables to remove, as demonstrated in the video.

3. Assess Model Fit (Regression Statistics Table):

   • R-squared ($R^2$): This still tells you the percentage of variation in Y explained by all the X's in your model.
   • Adjusted R-squared: This is a modified version of $R^2$ that accounts for the number of predictors in the model. As you add more variables, $R^2$ will always go up, even if the new variables are useless. Adjusted R-squared penalizes the score for adding variables that don't improve the model. In multiple regression, you should always focus on the Adjusted $R^2$ to compare the effectiveness of models with different numbers of predictors. A higher Adjusted $R^2$ is better.

4. Write the Equation (Coefficients Table): Once you have your final, refined model (with only significant predictors), use the Intercept and Coefficients to write your predictive equation.

   • Predicted Y = Intercept + (Coeff_X1 * X1) + (Coeff_X2 * X2) + ...

Advanced Model Building: Best Subsets Regression

In complex projects, you might have 5, 10, or even more potential X variables. Testing every possible combination manually would be incredibly tedious. Statistical software like Minitab automates this with a technique called Best Subsets Regression.

This technique evaluates all possible models (model with 1 predictor, models with 2 predictors, etc.) and presents a summary of the best-fitting models for each number of predictors. This allows you, the analyst, to make an informed trade-off between model simplicity and predictive power.

The key takeaway from this is that developing a good model is often about finding a balance. We want a model that predicts well (high Adjusted R-squared) but is also as simple as possible. The concept of Mallows' Cp helps quantify this balance, where a good model has a Cp value close to the number of predictors plus the constant.

Conclusion

Today, we've made a significant leap from simple to multiple regression, equipping you with a tool that more accurately reflects the complexities of real-world manufacturing processes. You learned how to build, refine, and interpret a model with several inputs.

Key Takeaways:

• Multiple regression extends the simple linear model to include multiple independent variables ($X_1,X_2,\dots$).
• The coefficient for each variable represents its effect on the output while holding other variables constant.
• Developing a strong model is an iterative process: build an initial model, remove insignificant predictors (based on p-values), and re-evaluate.
• Adjusted R-squared is the preferred metric for assessing model fit in multiple regression because it penalizes for adding non-value-added predictors.
• The final regression equation can be used to predict a quality outcome based on a specific combination of process settings.

Preview of the Next Lesson:

Building a model and checking its statistical significance is a huge step. But is the model actually valid? Regression analysis rests on several key assumptions (e.g., that the errors are normally distributed and independent). If these assumptions are violated, our model's predictions can be unreliable. In the next lesson, we will tackle the crucial topic of validating regression model assumptions and assessing model adequacy using residual analysis.