Introduction
Hello! In our last lesson, we saw how a single decision tree can easily overfit the training data. We learned that pruning the tree by limiting its max_depth is a crucial step to make it generalize better. However, even a well-pruned decision tree can be unstable; small changes in the training data can lead to a completely different tree. This instability is a classic sign of high variance.
Today, we'll explore a powerful technique designed specifically to combat this problem: Bootstrap Aggregating, or Bagging. This is our first look at an ensemble method—an approach where we combine multiple machine learning models to build a more robust one. By the end of this lesson, you will understand how bagging works and why it is so effective at reducing model variance.
The Problem: Bias-Variance Trade-off Revisited
Before diving into the solution, let's quickly refresh our memory on the bias-variance trade-off. A complex model (like a deep decision tree) has low bias but high variance, while a simple model (like a very shallow tree) has high bias but low variance. Our goal is to find a sweet spot.
Bagging is one of the most effective methods for achieving this balance, especially for high-variance models like decision trees.
Machine Learning Fundamentals: Bias and Variance
To start, let's watch a quick refresher on bias and variance from StatQuest. Pay close attention at the end when it introduces bagging as a solution.
Watch from the beginning to the end (00:00 - 06:11). The video will review the concepts of bias and variance using an intuitive example and then explicitly mention bagging as a technique for finding the 'sweet spot' between a simple and a complex model.
As Josh Starmer explained, a model with high variance (the "squiggly line") fits the training data well but performs poorly on the testing data because it's too sensitive to the specific data points it was trained on. This is exactly the problem we face with individual decision trees. Bagging offers a way to keep the flexibility (low bias) of complex trees while drastically reducing their variance.
What is Bootstrap Aggregating (Bagging)?
The name "Bagging" comes from combining two key concepts: Bootstrap sampling and Aggregating predictions. Let's break down the process step-by-step.
Step 1: Bootstrap Sampling
The first step is to create multiple, new training datasets from our original dataset. We do this using a technique called bootstrapping, which means sampling with replacement.
Imagine you have a bag of 10 unique marbles. To create a bootstrap sample:
- You draw one marble, note its color, and put it back in the bag.
- You repeat this process 10 times.
Because you replace the marble each time, your new set of 10 marbles might have duplicates of some colors and be missing others entirely. This is a bootstrap sample. In bagging, we do this with our data points to create many different training sets.
StatQuest: Random Forests Part 1 - Building, Using and Evaluating
This video from StatQuest provides a clear, animated explanation of creating a bootstrap dataset.
Watch the segment from 01:10 to 02:24. It visually demonstrates how a bootstrap dataset is created by randomly selecting samples from the original data, with replacement.
Step 2: Train Multiple Models
Once we have our collection of bootstrap datasets (e.g., 100 of them), we train a separate model on each one. If we are using decision trees, we would train 100 different decision trees, one for each unique bootstrap sample.
Since each tree is trained on a slightly different dataset, each one will learn slightly different decision boundaries. Some will overfit in one way, others in another.
Step 3: Aggregate the Predictions
To make a final prediction for a new data point, we let all our individual models "vote."
- For a classification task, the final prediction is the class that gets the most votes from the individual trees.
- For a regression task, the final prediction is the average of all the predictions from the individual trees.
This entire process—creating bootstrap samples, training a model on each, and aggregating the results—is called Bagging.
This image provides a great visual summary of the whole process:
To tie it all together, let's return to the StatQuest video, which formally defines bagging after showing the aggregation step.
StatQuest: Random Forests Part 1 - Building, Using and Evaluating
This clip shows how the predictions from all the trees are combined and provides the formal definition of bagging.
Watch from 04:34 to 05:53. You'll see the aggregation (voting) step in action, followed by a concise summary defining bagging.
Why Bagging Reduces Variance
Now for the most important question: why does this work?
The magic of bagging lies in the aggregation step. Each individual decision tree is a low-bias, high-variance model. It's an "expert" on its own little bootstrap world but can be very wrong about the real world. By training many trees on different subsets of the data, we create a diverse set of "experts."
When we average their predictions (or take a majority vote), the individual errors tend to cancel each other out. A tree that overfitted on a specific outlier in its sample will be outvoted by the many other trees that never saw that outlier. The final aggregated model is much more stable and less sensitive to the noise in the original training data. In other words, it has lower variance than any single tree, while retaining the low bias.
A Guide to Bagging in Machine Learning
This article from DataCamp clearly explains the concept and benefits of bagging. Reading these sections will solidify your understanding of how it tackles variance.
Please read the sections 'What is Bagging?' and 'Advantages of Bagging'. Focus on how training models on different bootstraps and then aggregating them helps reduce variance and avoid overfitting.
In scikit-learn, this technique is implemented in the BaggingClassifier and BaggingRegressor classes. You can wrap almost any model (like a DecisionTreeClassifier) with these classes to create a bagged ensemble.
Conclusion
In this lesson, you've been introduced to your first ensemble method, Bagging, and learned how it cleverly solves the high-variance problem inherent in models like decision trees.
Key Takeaways:
- Bagging (Bootstrap Aggregating) is an ensemble technique that combines predictions from multiple models to improve robustness.
- It involves three steps: Bootstrap (sampling with replacement), training a model on each sample, and Aggregating (voting/averaging) their predictions.
- The primary benefit of bagging is variance reduction. By averaging out the predictions of many high-variance models, the ensemble's prediction becomes much more stable and less prone to overfitting.
- Bagging is particularly effective when used with complex, unstable models like unpruned decision trees.
Preview of the Next Lesson
While bagging is a powerful, general-purpose technique, a specialized version of it has become one of the most popular and effective machine learning algorithms: the Random Forest. In the next lesson, we will see how Random Forests build directly on the idea of bagging and add one more layer of randomness to make the individual trees even more diverse, leading to an even better final model.