Detecting Multicollinearity with VIF

1. What is Multicollinearity and Why is it a Problem?

Before we start calculating anything, it's essential to build an intuition for what multicollinearity is and why it matters. In short, when features are highly correlated, it becomes difficult for a model to distinguish their individual effects on the target variable.

As the video explains, if two predictors X1 and X2 contain very similar information, the model gets "confused." It might assign a large positive coefficient to X1 and a large negative one to X2, or vice-versa. Small changes in your data could cause these coefficients to swing wildly. This makes the model difficult to interpret and less trustworthy, even if its overall predictive accuracy isn't harmed.

2. Quantifying Multicollinearity with VIF

While a correlation matrix shows pairwise relationships, it doesn't capture more complex interactions where one feature might be a linear combination of several other features. This is where the Variance Inflation Factor (VIF) comes in.

Conceptually, the VIF for a specific feature is calculated by:

1. Running a linear regression model that uses this feature as the target and all other features as predictors.
2. Calculating the R-squared ($R^2$) value from this regression. This $R^2$ tells you how well the other features can explain your chosen feature.
3. Calculating the VIF using the formula: $$\text{VIF}=\frac{1}{1-R_i^2}$$ where $R_i^2$ is the R-squared from regressing feature i on all other features.

If the other features perfectly predict feature i ($R_i^2=1$), the VIF is infinite. If the other features have no relationship with feature i ($R_i^2=0$), the VIF is 1.

Interpreting VIF Values

So, what do the VIF numbers actually mean? Here are the commonly used rules of thumb:

To summarize the key points from the article:

• VIF = 1: No correlation. The feature is completely independent of the others.
• 1 < VIF < 5: Moderate correlation. This is often acceptable.
• VIF > 5: High correlation. This is a cause for concern and may require further investigation.
• VIF > 10: Very high correlation. Indicates serious multicollinearity, and the feature should likely be removed.

3. Calculating and Using VIF in Python

Now let's move to the practical implementation. We will use the statsmodels library in Python to calculate VIF.

A Crucial Pitfall: Adding a Constant

There's a critical detail when using statsmodels' VIF function: it assumes that the data you provide includes a constant (i.e., an intercept term), but it doesn't add one for you. This is a common source of error that leads to incorrect VIF values. Your background in computer science and experience with APIs will help you appreciate the importance of understanding such implementation details.

Iterative VIF Calculation and Feature Removal

The correct way to use VIF for feature selection is an iterative process:

1. Calculate the VIF for all features.
2. Find the feature with the highest VIF.
3. If this VIF is above your threshold (e.g., 5 or 10), remove that single feature.
4. Repeat the process until all remaining features have VIFs below the threshold.

It is incorrect to calculate VIFs once and drop all features above the threshold simultaneously. When you remove one feature, the VIFs of all other features will change, so they must be recalculated.

The following video provides a hands-on demonstration of this exact iterative process.

Here is a complete code example that puts everything together. We'll use the California housing dataset from scikit-learn.

import pandas as pd
import numpy as np
from sklearn.datasets import fetch_california_housing
from statsmodels.stats.outliers_influence import variance_inflation_factor
from statsmodels.tools.tools import add_constant




# Load the dataset
housing = fetch_california_housing()
X = pd.DataFrame(housing.data, columns=housing.feature_names)




# --- The Iterative VIF Removal Process ---
vif_threshold = 10.0
features_to_keep = X.copy()



# We use a while loop to iteratively remove features
while True:



    # Add a constant for the VIF calculation
    X_with_const = add_constant(features_to_keep)
    



    # Calculate VIF for each feature
    vif_data = pd.Series([variance_inflation_factor(X_with_const.values, i) 
                          for i in range(X_with_const.shape[1])], 
                         index=X_with_const.columns).drop('const')
    



    # Get the feature with the highest VIF
    max_vif_feature = vif_data.idxmax()
    max_vif_value = vif_data.max()
    
    print(f"Iteration with {len(features_to_keep.columns)} features. Max VIF is for '{max_vif_feature}' with value {max_vif_value:.2f}")




    # If the highest VIF is below the threshold, we are done
    if max_vif_value < vif_threshold:
        break
    else:



        # Otherwise, drop the feature with the highest VIF and repeat
        print(f"Dropping '{max_vif_feature}'...")
        features_to_keep = features_to_keep.drop(columns=[max_vif_feature])

print("\nFinal set of features after VIF selection:")
print(features_to_keep.columns.tolist())

In some cases, you might encounter perfect multicollinearity, where one feature is a perfect linear combination of others. This results in an infinite VIF.

Conclusion

In this lesson, you have moved from visually identifying correlations to diagnostically measuring and acting on multicollinearity. This is a fundamental skill in the feature preprocessing stage of the machine learning workflow.

Key Takeaways:

• Multicollinearity occurs when predictor variables are highly correlated, which can make model coefficients unstable and hard to interpret.
• Variance Inflation Factor (VIF) quantifies how much a feature's variance is inflated by its relationship with other features.
• VIF Interpretation: A VIF greater than 5 is a warning sign, and a VIF greater than 10 indicates serious multicollinearity.
• Practical Implementation: Always use add_constant when calculating VIF with statsmodels. Feature removal must be an iterative process, removing the single highest-VIF feature and recalculating in each step.

You are now equipped to handle one of the common data issues that can trip up modeling. In our next module, we will formally begin our journey into machine learning by exploring the core theoretical concepts that underpin how models learn and generalize, starting with the crucial bias-variance tradeoff and the role of loss functions.