Welcome back. In our last lesson, you successfully implemented a decision tree, visualized its logic, and evaluated its performance. A key takeaway was that while single decision trees are highly interpretable, they are prone to overfitting, meaning they capture noise in the training data and don't generalize well. You saw how pruning can help, but it's a manual process of finding a balance.
Today, we will explore a more powerful and systematic solution to this problem: ensemble learning. This lesson introduces the core idea of combining multiple models to produce a better one. Specifically, we'll focus on a fundamental ensemble strategy called bagging, or bootstrap aggregating. You will learn how this technique harnesses the "wisdom of the crowd" to build robust models that significantly reduce the variance we observed in single decision trees. This is the first step in the evolution from a simple decision tree to more advanced algorithms like Random Forests.
The Power of the Collective: An Introduction to Ensemble Learning
The core idea behind ensemble learning is surprisingly intuitive: a diverse group of moderately informed individuals is often smarter than a single expert. In machine learning, instead of relying on one model, we train multiple models—often called "weak learners"—and combine their predictions to form a single, highly accurate "strong learner."
So, why does this work? The answer lies in the bias-variance tradeoff, a concept we've touched upon before. Recall that a model with high variance is very sensitive to the specific training data it sees; it overfits. As you saw in the previous lesson, a fully grown decision tree has this exact characteristic: it learns the training data perfectly (low bias) but performs erratically on new data (high variance).
The video below from Krish Naik provides a great refresher on this topic, specifically in the context of decision trees.
Machine Learning-Bias And Variance In Depth Intuition| Overfitting Underfitting
This clip from "Machine Learning-Bias And Variance In Depth Intuition" directly connects decision trees to the concept of high variance.
Please watch the segment from decision tree and random forests. The key point is understanding why a deep decision tree is considered a low-bias, high-variance model.
Ensemble methods provide a way to manage this tradeoff. Bagging, the technique we are focusing on today, is specifically designed to take a collection of high-variance models and combine them to create a single model with much lower variance, without a significant increase in bias.
To formalize this, let's explore the foundational concepts in a bit more detail.
Ensemble learning: bagging, boosting and stacking
This resource clearly defines the goal of ensemble learning and its relationship to the bias-variance tradeoff.
Read the introduction and the section "Single weak learner". Focus on the definitions of weak and strong learners and how ensemble methods are categorized by their goal. Note that bagging is explicitly listed as a technique to decrease variance.
The Bagging Mechanism: Bootstrap and Aggregate
The name "bagging" is a portmanteau of Bootstrap Aggregating. This name neatly describes the two-step process at its heart.
- Bootstrap: We generate multiple, slightly different training datasets from our original dataset.
- Aggregate: We train a model on each of these new datasets and then combine their predictions.
Let's break down each step.
Step 1: Bootstrapping
If we had multiple independent datasets, we could train a model on each one and average their predictions. In reality, we usually only have one dataset. Bootstrapping is a powerful statistical method that allows us to simulate having multiple datasets.
It works through sampling with replacement. Imagine your training data is a bag of marbles. To create one "bootstrap sample," you:
- Draw one marble from the bag and note its color.
- Put the marble back into the bag. This is the "with replacement" part.
- Repeat this process until your new sample has the same number of marbles as the original bag.
Because you replace each marble, some will be selected multiple times in your new sample, and some won't be selected at all. By repeating this process, you can create many unique training datasets, each one a slightly different representation of the original data.
The following video from Krish Naik provides an excellent high-level overview of the entire bagging process, starting with bootstrapping.
Tutorial 42 - Ensemble: What is Bagging (Bootstrap Aggregation)?
This video, "Ensemble: What is Bagging (Bootstrap Aggregation)?", provides a clear, step-by-step walkthrough of the bagging procedure.
Watch the first few minutes of the video, from the beginning up to the explanation, to get a solid intuition for bootstrapping and how it fits into the overall bagging workflow.
Step 2: Training and Aggregating
Once we have our collection of bootstrap samples, the rest of the process is straightforward and, importantly, can be done in parallel (a major advantage given your background in distributed systems).
- Training: We take each bootstrap sample and train a separate model on it. For bagging with decision trees, we typically train a full, unpruned tree on each sample. Since each tree sees slightly different data, each will learn slightly different rules and make slightly different errors.
- Aggregation: To get a final prediction, we combine the outputs of all the individual models.
- For classification problems, this is usually done by a majority vote. Each tree "votes" for a class, and the class with the most votes wins.
- For regression problems, we simply average the predictions from all the individual trees.
The combination of these steps—creating diverse datasets via bootstrapping and then averaging out individual model errors via aggregation—is what gives bagging its power.
The diagrams below visualize this entire process.
Why Bagging Works: The Magic of Averaging
So, why does training a hundred overfit models result in one model that generalizes well?
Each individual decision tree we train is a high-variance model. It has memorized the quirks of its particular bootstrap sample. However, because each tree was trained on a different sample, the specific errors they make will be different and largely uncorrelated. When we aggregate their predictions, these individual errors tend to cancel each other out.
The result is a final model that captures the true underlying signal from the data while averaging away the noise. The overall bias of the ensemble stays low (like the individual trees), but the variance is dramatically reduced.
This article from the Paperspace Blog provides a great summary of the concepts we've discussed.
Introduction to Bagging and Ensemble Methods
This article explains the core concepts of bagging and why it is so effective.
Please read the following sections: Start at the section What Is Bagging? for a formal definition. Continue through Basic Concepts Behind Bagging. This section neatly explains the entire process, how aggregation works, and crucially, states that bagging works well for "high variance models such as Decision Trees." This directly connects to our previous lesson.
Conclusion
In this lesson, you've moved from the world of single models to the powerful paradigm of ensemble learning. You've learned the theory behind bagging, a technique that masterfully addresses the overfitting problem we saw with individual decision trees.
Key Takeaways:
- Ensemble learning combines predictions from multiple models ("weak learners") to create a more robust and accurate "strong learner."
- Bagging (Bootstrap Aggregating) is an ensemble method designed specifically to reduce model variance.
- The process involves two main steps:
- Bootstrapping: Creating multiple datasets by sampling with replacement from the original training data.
- Aggregating: Training a model on each bootstrap sample and combining their predictions through voting (for classification) or averaging (for regression).
- Bagging is highly effective when used with unstable, high-variance models like unpruned decision trees, as it averages out their individual errors.
You now have the conceptual foundation for one of the most popular and effective machine learning algorithms. In our next lesson, we will build directly on this foundation to explore the Random Forest, which takes the idea of bagging decision trees and adds one more clever twist to make it even more powerful.