Bagging: Reducing Variance in Ensemble Models

Part 1: Revisiting the Bias-Variance Tradeoff

Before we dive into bagging, let's quickly refresh our memory on a critical concept: the bias-variance tradeoff. In machine learning, the prediction error of a model can be broken down into three parts: bias, variance, and irreducible error.

A single, fully grown decision tree, like we've discussed, is a classic example of a low-bias, high-variance model.

• Low Bias: Because it can keep splitting until every leaf is pure, it can perfectly capture the training data, even complex patterns. It makes very few assumptions about the data's structure.
• High Variance: This flexibility is also its weakness. It learns the training data too well, including the noise. If we were to train the same tree on a slightly different subset of our data, we might get a dramatically different tree structure. This instability is what we mean by high variance, and it leads to poor performance on new, unseen data (overfitting).

To see this idea in action, let's watch a brief clip.

The core problem we need to solve is this high variance. How can we use the power of a flexible model like a decision tree without falling into the trap of overfitting? The answer lies in bagging.

Part 2: What is Bootstrap Aggregating (Bagging)?

Bagging is an acronym for Bootstrap Aggregating. It’s an ensemble technique that combines the predictions from multiple models to produce a more robust final prediction. The name itself tells you the two key steps involved.

Let's unpack these steps by reading a section from the "Hands-On Machine Learning with R" book.

To summarize the key ideas from the reading:

1. Bootstrapping: We create many new training datasets from our original one. Each new dataset is the same size as the original and is created by sampling with replacement. This means some original data points might appear multiple times in a bootstrap sample, while others might not appear at all.

2. Training: We train a separate model (a "base learner," like a decision tree) on each of these bootstrap datasets. Because each dataset is slightly different, we end up with a collection of slightly different models.

3. Aggregating: To make a new prediction, we get a prediction from every model in our ensemble and combine them.

   • For regression, we average the predictions.
   • For classification, we take a majority vote (or average the predicted probabilities).

This process is visualized beautifully in the following diagram.

Part 3: How Averaging Reduces Variance

So, we've trained many high-variance models on slightly different data. How does averaging their results lead to a low-variance ensemble?

The core intuition is that while each individual model overfits the data in its own way (capturing different bits of noise), these errors are largely random. When we average the predictions, these random errors tend to cancel each other out, leaving behind the true underlying signal.

There's a simple statistical principle behind this. The variance of the average of $N$ independent random variables is $1/N$ times the variance of a single one of those variables.

In our case, the $X_i$ are our individual tree models. By averaging $N$ trees, we could theoretically reduce the variance by a factor of $N$.

The Important Caveat: In bagging, our models are not perfectly independent. They are all trained on bootstrap samples taken from the same original dataset. This means the trees will be correlated. If there's a strong predictor in the dataset, most trees will likely use it for their first split. This correlation limits the amount of variance reduction we can get, but the reduction is still very significant.

Let's watch a video that explains this key concept.

So, to summarize the magic of bagging:

• We start with a high-variance, low-bias learner (like a deep decision tree).
• We create many versions of it by training on bootstrap samples.
• The bias of the ensemble remains low (averaging many unbiased estimates gives an unbiased estimate).
• The variance of the ensemble is significantly reduced because the individual models' errors, driven by noise in their respective training samples, tend to cancel out upon averaging.

Conclusion

You've now uncovered the statistical reasoning behind one of the most powerful ensemble techniques. You can now explain not just what bagging is, but why it is so effective at improving the performance of models like decision trees.

Key Takeaways:

• Single decision trees are often low-bias, high-variance models, which means they are prone to overfitting.
• Bagging (Bootstrap Aggregating) is an ensemble method that combats this by training many trees on different bootstrap samples (sampling with replacement) and aggregating their predictions.
• The core benefit of bagging is variance reduction. By averaging the predictions of many models, the random errors and noise captured by each individual model tend to cancel out.
• Bagging is most effective when applied to unstable, high-variance models. The bias of the ensemble remains roughly the same as the bias of the base learners.

Preview of the Next Lesson:
Bagging is a huge step forward, but we can do even better. The main limitation of bagging is that the trees can still be quite correlated, which limits the total variance reduction. In our next lesson, we will explore Random Forests, a clever modification of bagging that introduces an extra element of randomness to deliberately de-correlate the trees, leading to an even more powerful and robust model.