Regression Models with scikit-learn

The Standard Machine Learning Workflow

A typical supervised learning project follows a sequence of well-defined steps. We'll walk through each one, using scikit-learn's tools to build a linear regression model.

The workflow looks like this:

1. Prepare Data: Separate features (X) from the target (y).
2. Split Data: Divide the data into training and testing sets.
3. Train Model: Fit a model to the training data.
4. Make Predictions: Use the trained model to predict outcomes for the test set.
5. Evaluate Performance: Measure how well the model's predictions match the actual values.

Let's begin with a hands-on project.

A Practical Walkthrough: Predicting E-commerce Spending

To make this concrete, we'll follow a project that aims to predict how much an e-commerce customer will spend per year based on their usage patterns. The following video provides an excellent end-to-end demonstration of the scikit-learn workflow. We will watch it in segments as we cover each step.

As you saw, the first step is always to understand your data and clearly define your features (X), which are the inputs to your model, and your target (y), which is the value you want to predict.

Step 1: Splitting Data into Training and Test Sets

This is one of the most critical steps in machine learning. We train our model on one portion of the data (the training set) and then evaluate its performance on a completely separate, unseen portion (the test set). This simulates how the model would perform on new, real-world data.

Scikit-learn provides a convenient function for this: train_test_split.

The random_state parameter is important for reproducibility. By setting it to a specific number, you ensure that you get the exact same split every time you run your code, which is essential for debugging and consistent results.

Step 2: Training the Model

With our data split, we can now train our model. This involves three simple actions:

1. Import the model class from scikit-learn (e.g., LinearRegression).
2. Create an instance of the model.
3. Call the .fit() method on the model instance, passing the training data (X_train, y_train).

The .fit() method is where the model "learns" the relationship between the features and the target from the training data.

That's it! The model is now trained. For linear models, the learned parameters are stored in the .coef_ (coefficients) and .intercept_ attributes.

Step 3: Making Predictions

Once the model is trained, we can use it to make predictions on new data. We use the .predict() method and pass it the features from our test set (X_test). The model will return an array of predicted target values.

We now have the model's predictions (y_pred) and the actual ground truth values (y_test). The final step is to compare them.

Step 4: Evaluating Model Performance

How good is our model? Evaluation is crucial for understanding a model's strengths and weaknesses. We can evaluate it both visually and with quantitative metrics.

Visual Evaluation

A simple and intuitive way to assess a regression model is to plot the actual values (y_test) against the predicted values (y_pred). If the model is perfect, all points will lie on a straight diagonal line. The more scattered the points are, the worse the model.

Quantitative Evaluation with Metrics

While visuals are helpful, we need objective metrics to quantify performance. For regression, the most common metrics are:

• Mean Squared Error (MSE): The average of the squared differences between the actual and predicted values. It heavily penalizes large errors.
  $\text{MSE}=\frac{1}{n}{\sum }_{i=1}^n(y_i-{\hat{y}}_i{)}^2$

*This diagram provides a great visual intuition for how MSE is calculated by squaring the differences (errors) between actual and predicted points and then averaging them.*

• Root Mean Squared Error (RMSE): The square root of the MSE. A key advantage is that it is in the same units as the target variable, making it more interpretable. For our e-commerce example, an RMSE of 10 would mean our model's predictions are, on average, off by about $10.
  $\text{RMSE}=\sqrt{\text{MSE}}$

• R-squared (R² Score): The "coefficient of determination." It represents the proportion of the variance in the target variable that is predictable from the features.

  • R² = 1: The model explains all the variability of the data.
  • R² = 0: The model does not explain any of the variability (it's no better than just predicting the mean of the target).
  • R² < 0: The model is worse than predicting the mean.

The following guide gives clear, concise definitions for these metrics.

Now, let's see how to calculate and interpret these metrics in our project.

A Complete Code Example

Here is a consolidated Python code block that summarizes the entire workflow. This example uses Ridge regression, connecting back to our previous lesson, and includes feature scaling, which is crucial for regularized models.

import pandas as pd
from sklearn.datasets import fetch_california_housing
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import Ridge
from sklearn.metrics import mean_squared_error, r2_score

# 1. Load and Prepare Data
housing = fetch_california_housing()
X = pd.DataFrame(housing.data, columns=housing.feature_names)
y = housing.target

# 2. Split Data
# Using 80% for training and 20% for testing
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# 3. Preprocess Data (Feature Scaling)
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test) # Use the scaler fitted on the training data

# 4. Train Model
# We'll use Ridge regression with a chosen alpha
model = Ridge(alpha=1.0)
model.fit(X_train_scaled, y_train)

# 5. Make Predictions
y_pred = model.predict(X_test_scaled)

# 6. Evaluate Performance
mse = mean_squared_error(y_test, y_pred)
rmse = mse ** 0.5  # or np.sqrt(mse)
r2 = r2_score(y_test, y_pred)

print(f"Model: Ridge Regression (alpha=1.0)")
print("--------------------------------------")
print(f"R-squared (R²): {r2:.4f}")
print(f"Mean Squared Error (MSE): {mse:.4f}")
print(f"Root Mean Squared Error (RMSE): {rmse:.4f}")

Conclusion

Excellent work! You have now walked through the fundamental workflow for building and evaluating a supervised learning model in scikit-learn. This process is your bread and butter as a machine learning practitioner.

Key Takeaways:

• The standard workflow is: Prepare Data → Split → Train → Predict → Evaluate.
• train_test_split is essential for creating training and test sets to get an unbiased estimate of model performance.
• The model learns from the training data using the .fit() method.
• Predictions are made on unseen test data using the .predict() method.
• Performance is quantified using metrics like MSE, RMSE, and R², each providing a different perspective on model accuracy.
• Feature scaling (StandardScaler) is a critical preprocessing step, especially for models sensitive to feature magnitudes like Ridge and Lasso.

Preview of the Next Lesson:

We can now build a model and measure how well it predicts. But what is the model actually telling us? In the next lesson, we will focus on interpreting regression coefficients. You will learn how to analyze the coef_ attribute of your trained model to understand the relationship between each feature and the target variable, answering questions like, "How much does the price increase for each additional bedroom?" This is a key step in turning a machine learning model into actionable insights.