Building Data Pipelines with scikit-learn

Why Use Pipelines?

Before we dive into the code, let's quickly understand the key benefits of this approach:

• Organization: It bundles all preprocessing steps and the final model into a single object, making your workflow clean and easy to understand.
• Preventing Data Leakage: When you use a pipeline, you fit your preprocessing steps (like scaling or imputation) on the training data only. The pipeline ensures that this same fitted transformation is then correctly applied to the validation and test data without re-learning from them, which prevents a common and subtle bug called data leakage.
• Reproducibility and Automation: A saved pipeline can be easily reloaded and used to make predictions on new data, guaranteeing that the exact same preprocessing is applied every time.

1. The Simple Pipeline: Chaining Steps Together

Let's start with the simplest case: a sequence of steps applied to the entire dataset. This is common when all your features are of the same type (e.g., all numerical). The main tool for this is scikit-learn's Pipeline or its more concise cousin, make_pipeline.

A pipeline chains together multiple "steps," where each step is a transformer (like StandardScaler or SimpleImputer) and the final step is typically an estimator (a machine learning model like LogisticRegression).

As you saw, make_pipeline automatically names the steps for you (using the lowercase class names), making the code very clean. Here's a quick recap of the concept with code:

import numpy as np
from sklearn.pipeline import make_pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression

# Sample data with missing values
X_train = np.array([[1, 2], [np.nan, 4], [3, 6], [5, 8]])
y_train = np.array([0, 0, 1, 1])

# Create a pipeline
# 1. Impute missing values with the mean
# 2. Scale the features
# 3. Train a logistic regression model
pipe = make_pipeline(
    SimpleImputer(strategy='mean'),
    StandardScaler(),
    LogisticRegression()
)

# Fit the entire pipeline on the training data
pipe.fit(X_train, y_train)

# Now you can use the fitted pipeline to predict on new data
X_test = np.array([[2, 3], [4, np.nan]])
predictions = pipe.predict(X_test)

print(f"Pipeline object: {pipe}")
print(f"Predictions on test data: {predictions}")

Notice we only call .fit() once. The pipeline handles passing the data from one step to the next, fitting each transformer and finally the model.

2. The ColumnTransformer: Handling Mixed Data Types

The simple pipeline works great if all features need the same treatment. But what about a more realistic scenario where you have a mix of numerical and categorical columns? You need to scale the numerical data and one-hot encode the categorical data. Trying to apply a StandardScaler to a categorical column would raise an error.

This is where ColumnTransformer comes in. It's a powerful tool that applies different transformers to different columns of your data.

The workflow is as follows:

1. Create separate pipelines for each group of columns (e.g., a numerical pipeline and a categorical pipeline).
2. Use ColumnTransformer to specify which pipeline to apply to which columns.
3. Combine the ColumnTransformer with a final model inside a main Pipeline.

This sounds complex, but it's very systematic. Let's watch a detailed walkthrough.

For a text-based alternative or a quick reference, this guide is also excellent.

# Assume num_transformer is a pipeline with StandardScaler
# Assume cat_transformer is a pipeline with OneHotEncoder
# Assume numeric_features = ['age', 'salary']
# Assume categorical_features = ['city', 'has_car']

transformers = [
    ('numerical_processing', num_transformer, numeric_features),
    ('categorical_processing', cat_transformer, categorical_features)
]

3. Putting It All Together: A Complete Workflow

Now, let's build a complete end-to-end pipeline that combines the ColumnTransformer (for preprocessing) with a final estimator (our model).

Here is a full code example that ties everything together. We'll build a pipeline that:

1. Identifies numerical and categorical columns.
2. Imputes and scales numerical data.
3. Imputes and one-hot encodes categorical data.
4. Trains a Random Forest classifier on the processed data.

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.compose import ColumnTransformer
from sklearn.ensemble import RandomForestClassifier

# 1. Create a sample dataset
data = {
    'age': [25, 30, 45, 35, 50, None, 22],
    'city': ['New York', 'Paris', 'New York', 'London', 'Paris', 'London', None],
    'experience': [2, 5, 20, 10, 25, 5, 1],
    'purchased': [0, 1, 1, 0, 1, 0, 0]
}
df = pd.DataFrame(data)

X = df.drop('purchased', axis=1)
y = df['purchased']

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# 2. Define preprocessing for numerical columns
# Impute with median, then scale
numeric_transformer = Pipeline(steps=[
    ('imputer', SimpleImputer(strategy='median')),
    ('scaler', StandardScaler())
])

# 3. Define preprocessing for categorical columns
# Impute with the most frequent value, then one-hot encode
categorical_transformer = Pipeline(steps=[
    ('imputer', SimpleImputer(strategy='most_frequent')),
    ('onehot', OneHotEncoder(handle_unknown='ignore')) # handle_unknown ignores categories in test set not seen in training
])

# 4. Identify column names for each transformer
numeric_features = ['age', 'experience']
categorical_features = ['city']

# 5. Create the ColumnTransformer to apply different transformations
preprocessor = ColumnTransformer(
    transformers=[
        ('num', numeric_transformer, numeric_features),
        ('cat', categorical_transformer, categorical_features)
    ])

# 6. Create the final pipeline with the preprocessor and the model
model_pipeline = Pipeline(steps=[
    ('preprocessor', preprocessor),
    ('classifier', RandomForestClassifier(random_state=42))
])

# 7. Train and evaluate the pipeline
model_pipeline.fit(X_train, y_train)

print("Model pipeline fitted successfully!")
score = model_pipeline.score(X_test, y_test)
print(f"Model accuracy on test set: {score:.2f}")

# You can also use it to predict on new, raw data
new_data = pd.DataFrame({'age': [40], 'city': ['Paris'], 'experience': [15]})
prediction = model_pipeline.predict(new_data)
print(f"Prediction for new data: {prediction[0]}")

This final model_pipeline object contains your entire workflow. It takes in raw Pandas DataFrames and outputs predictions. This is exactly what you need for a clean, professional machine learning project.

Conclusion

Congratulations on completing this crucial lesson! You've successfully bridged the gap between individual preprocessing steps and a fully integrated, professional machine learning workflow.

Key Takeaways:

• Pipeline and make_pipeline are used to chain transformations and a final model in a sequence.
• ColumnTransformer is essential for real-world datasets with mixed data types. It applies different transformation pipelines to different sets of columns.
• The standard workflow is to create separate transformation pipelines for numerical and categorical data, combine them with a ColumnTransformer, and then place this preprocessor as the first step in a final Pipeline that ends with your model.
• This approach automates your preprocessing, prevents data leakage, and makes your entire model (including data prep) easily portable.

Preview of the Next Lesson:

This lesson concludes our module on Data Visualization and Preparation. You now have a comprehensive toolkit for preparing data for modeling. We will now shift gears significantly. The next few modules will provide the mathematical foundation you need to understand how machine learning algorithms work internally. Our next module is Linear Algebra Foundations, and we will begin by learning how to represent vectors and matrices in NumPy and perform basic operations. This will be the first step in building your intuition for the math behind the models.