Feature Selection Techniques

1. An Overview of Feature Selection Methods

Feature selection techniques are typically grouped into three main categories. Understanding them will give you a mental map for where the methods we're learning today fit in.

• Filter Methods: These methods rank features based on their intrinsic statistical properties, independent of any machine learning algorithm. They are fast and computationally inexpensive. The correlation analysis we'll study today falls into this category.
• Wrapper Methods: These methods use a specific machine learning model to evaluate the usefulness of different subsets of features. They "wrap" the feature selection process around a model. They are more computationally intensive but can often lead to better model performance. Recursive Feature Elimination (RFE) is a classic wrapper method.
• Embedded Methods: In these methods, the feature selection process is an integral part of the model training process. Examples include L1 (Lasso) regularization, which pushes the coefficients of less important features towards zero, and the feature importances derived from tree-based models.

To solidify these concepts, let's look at a brief overview.

2. Filter Method: Removing Correlated Features

One of the first checks you should perform is for multicollinearity—a situation where two or more independent features are highly correlated with each other.

Why is this a problem?

• Redundancy: If two features are highly correlated (e.g., a person's height in centimeters and height in inches), they are providing very similar information. Keeping both is redundant.
• Model Instability: For linear models (like Linear or Logistic Regression), multicollinearity can make the model coefficients unstable and difficult to interpret. The model might assign a large positive coefficient to one feature and a large negative one to its correlated counterpart, making it impossible to gauge the true effect of either feature individually.

Our goal is to identify and remove these redundant features.

Correlation Analysis using Heatmaps

The most common way to do this is by calculating the Pearson correlation coefficient between every pair of features and visualizing the result as a heatmap.

A quick note on the video: the presenter discusses negative correlation. For the purpose of removing redundant features, we are interested in the magnitude of the correlation, not its sign. A correlation of -0.9 is just as strong as +0.9; it just means the variables move in opposite directions. This is why the absolute value of the correlation is often used.

A Strategy for Removing Correlated Features

Once you've identified highly correlated pairs (e.g., with an absolute correlation > 0.8), you need a strategy to decide which feature to remove. You could manually remove one, or you could automate the process.

A robust method involves using the Variance Inflation Factor (VIF), which measures how much the variance of an estimated regression coefficient is increased because of multicollinearity.

As with all preprocessing steps, feature selection must be done only on the training data to avoid data leakage from the test set. You fit your selection strategy on X_train and then apply the same transformation (dropping the same columns) to both X_train and X_test.

3. Wrapper Method: Recursive Feature Elimination (RFE)

While correlation analysis is great for removing redundant features, it doesn't tell us which features are most predictive for our model. Wrapper methods address this by using a model's performance to score features.

Recursive Feature Elimination (RFE) is a popular and intuitive wrapper method. Here’s how it works:

1. Train Model: An estimator (e.g., Logistic Regression) is trained on the initial set of all features.
2. Rank Features: The importance of each feature is ranked (e.g., using model coefficients or feature_importances_).
3. Eliminate: The least important feature is removed from the dataset.
4. Repeat: The process is repeated with the remaining features until a pre-specified number of features is reached.

This approach is powerful because it considers the relationship between features and how they contribute collectively to a model's predictive power.

Implementing RFE with Scikit-Learn

The following video is an excellent, in-depth guide to the theory and practice of RFE. Given your background, you'll appreciate its conceptual clarity.

4. Hands-on Practice

Now it's time to put these two techniques into practice. The code below sets up a dataset with several redundant features. Your task is to apply both correlation analysis and RFE to select better feature subsets.

import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.feature_selection import RFE
from sklearn.metrics import accuracy_score




# 1. Generate data with redundant features
X, y = make_classification(
    n_samples=1000,
    n_features=20,
    n_informative=5,
    n_redundant=8,  # These features are linear combinations of informative ones
    n_classes=2,
    random_state=42
)
X = pd.DataFrame(X, columns=[f'feature_{i}' for i in range(20)])




# Create some manually correlated features for demonstration
X['feature_20'] = X['feature_1'] * 2 + np.random.normal(0, 0.1, 1000)
X['feature_21'] = -X['feature_2'] * 3 + np.random.normal(0, 0.1, 1000)





# Split data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42, stratify=y)




# --- YOUR CODE GOES HERE ---




# 2. Correlation-based Feature Selection
# Create a correlation matrix for the training data
cor_matrix = X_train.corr().abs()




# Select upper triangle of correlation matrix
upper_tri = cor_matrix.where(np.triu(np.ones(cor_matrix.shape),k=1).astype(bool))




# Find index of feature columns with correlation greater than 0.95
to_drop = [column for column in upper_tri.columns if any(upper_tri[column] > 0.95)]
print(f"Features to drop via correlation: {to_drop}")




# Drop features from training and testing sets
X_train_corr = X_train.drop(to_drop, axis=1)
X_test_corr = X_test.drop(to_drop, axis=1)

print(f"Original shape: {X_train.shape}, After correlation selection: {X_train_corr.shape}")





# 3. Recursive Feature Elimination (RFE)
# Standardize the data first, as we are using Logistic Regression
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)




# Initialize the RFE model
# We will aim to select 7 features
estimator = LogisticRegression(solver='liblinear')
rfe = RFE(estimator, n_features_to_select=7, step=1)




# Fit RFE
rfe.fit(X_train_scaled, y_train)




# Get the selected features
selected_features_rfe = X_train.columns[rfe.support_]
print(f"\nFeatures selected by RFE: {selected_features_rfe.tolist()}")




# Transform the data
X_train_rfe = rfe.transform(X_train_scaled)
X_test_rfe = rfe.transform(X_test_scaled)
print(f"RFE selected shape: {X_train_rfe.shape}")





# 4. Compare Model Performance
# Model on original (scaled) data
model_full = LogisticRegression(solver='liblinear')
model_full.fit(X_train_scaled, y_train)
y_pred_full = model_full.predict(X_test_scaled)
print(f"\nAccuracy (All Features): {accuracy_score(y_test, y_pred_full):.4f}")




# Model on correlation-selected data (needs scaling)
scaler_corr = StandardScaler()
X_train_corr_scaled = scaler_corr.fit_transform(X_train_corr)
X_test_corr_scaled = scaler_corr.transform(X_test_corr)
model_corr = LogisticRegression(solver='liblinear')
model_corr.fit(X_train_corr_scaled, y_train)
y_pred_corr = model_corr.predict(X_test_corr_scaled)
print(f"Accuracy (Correlation Selected): {accuracy_score(y_test, y_pred_corr):.4f}")




# Model on RFE-selected data (already scaled and transformed)
model_rfe = LogisticRegression(solver='liblinear')
model_rfe.fit(X_train_rfe, y_train)
y_pred_rfe = model_rfe.predict(X_test_rfe)
print(f"Accuracy (RFE Selected): {accuracy_score(y_test, y_pred_rfe):.4f}")

Run the code and observe the results. You'll likely see that the models trained on fewer, more carefully selected features perform just as well as, or even slightly better than, the model trained on all features, while being computationally simpler.

Conclusion

In this lesson, you've added two essential feature selection techniques to your machine learning toolkit. You've learned how to streamline your feature set to build more efficient and often more accurate models.

Key Takeaways:

• Feature Selection is critical for reducing model complexity, improving training speed, and mitigating the curse of dimensionality.
• Correlation Analysis is a filter method that helps identify and remove redundant features (multicollinearity), which is particularly important for the stability and interpretability of linear models.
• Recursive Feature Elimination (RFE) is a powerful wrapper method that iteratively removes the weakest features based on the judgment of an underlying machine learning model.
• When using RFE with models whose coefficients are scale-dependent (like linear regression), it's crucial to standardize your features first.
• The optimal number of features for RFE is a hyperparameter that can be tuned using techniques like cross-validation with GridSearchCV.

In the next lesson, we will tie together everything we've learned in this module. You'll see how to build clean, reproducible, and robust preprocessing and modeling workflows using scikit-learn's Pipelines, which allow you to chain together steps like scaling, encoding, imputation, and even feature selection into a single, elegant object.