Welcome to the next step in our journey through tree-based models. In our last lesson, we explored bagging, a powerful technique for reducing the high variance of individual decision trees by training an ensemble of them on bootstrapped data. You learned that by averaging their predictions, we can create a more robust model that generalizes better.
This lesson builds directly on that foundation to introduce one of the most popular and effective algorithms in machine learning: the Random Forest. We will see how it evolves from the bagged trees concept by adding one simple yet ingenious modification. Our focus will be on understanding what this change is and, more importantly, why it leads to a significant improvement in model performance and generalization.
The Limitation of Bagged Trees: The Correlation Problem
Bagging is a significant improvement over a single decision tree, but it has a subtle weakness. While bootstrapping creates different training sets, leading to different trees, these trees are not always as diverse as we might hope.
Consider a dataset with a very strong predictive feature. Because it's so informative, this feature is likely to be chosen as the best split near the root of the tree. Since most of the bootstrap samples will contain this feature, most of the trees in our bagged ensemble will start by splitting on it. This results in a collection of trees that are structurally similar and therefore highly correlated.
Why is this a problem? The effectiveness of averaging out errors depends on those errors being independent. If our models are correlated, they tend to make the same kinds of mistakes. Averaging their predictions still helps, but the reduction in variance is limited.
The following resource provides an excellent explanation of this very issue, setting the stage for why Random Forest was developed.
Bagging and Random Forest — Applied Machine Learning ...
This text from Michael Pyrcz's online book explains the limitations of standard tree bagging and introduces the Random Forest as the solution.
First, briefly review the section on Tree Bagging to refresh the concept. Then, focus on the "Random Forest" section, specifically the first few paragraphs that explain why correlated trees are a problem and how this attenuates variance reduction. This is the core motivation for the Random Forest algorithm.
The Random Forest Solution: Adding Feature Randomness
To overcome the correlation problem, the Random Forest algorithm introduces a second layer of randomness. In addition to bootstrapping the data, it also randomizes the features used for building each tree.
Here is the key innovation:
When building each tree in the forest, at every split point, the algorithm selects only a random subset of features from the full set of available features. It then finds the best split using only that smaller subset.
This simple tweak has a profound effect. It prevents a single strong predictor from dominating all the trees. Since each tree is forced to consider a different subset of features at each split, the resulting trees are more diverse and less correlated with each other. This is the "random" part of Random Forest.
The StatQuest video below gives a wonderfully clear and visual walkthrough of this two-part random process.
StatQuest: Random Forests Part 1 - Building, Using and Evaluating
This classic video from Josh Starmer breaks down the Random Forest construction process into simple, easy-to-understand steps.
Watch from the beginning of the construction process. Pay close attention to the two key steps: Creating a bootstrap dataset (which you know from our lesson on bagging). Building a tree with a random subset of variables at each step. This is the new, crucial element that defines a Random Forest.
By combining these two sources of randomness—sampling data with replacement (bootstrapping) and sampling features at each split—we get a "forest" of trees that are deliberately different from one another.
This diagram illustrates the complete workflow:
Why Random Forests Improve Generalization
We've established that Random Forest creates a collection of decorrelated trees. But why does this lead to better generalization? It comes back to the bias-variance tradeoff and the power of averaging.
- Individual Trees are High-Variance: As we know, a single, unpruned decision tree has low bias (it can fit the training data very well) but high variance (it's unstable and overfits).
- Averaging Reduces Variance: Averaging the predictions of many models reduces the overall variance of the ensemble.
- Decorrelation Maximizes Variance Reduction: The amount of variance reduction we get from averaging depends on how correlated the models are. The less correlated they are, the more the errors cancel out, and the greater the reduction in variance.
By adding feature randomness, Random Forest actively decorrelates the trees, making the aggregation step much more effective than it is in standard bagging. The final ensemble model retains the low bias of the individual trees but has dramatically lower variance. This combination results in a robust model that generalizes exceptionally well to new, unseen data.
The following resource provides a fantastic, interactive explanation of this phenomenon.
This interactive article from MLU-Explain dives deep into why the structure of a Random Forest leads to better performance.
Please read the following sections: Start with the section on Feature Selection, which reinforces the key innovation we've just discussed. Next, read the section Variance in Composition. This part is crucial as it directly quotes the inventor of Random Forest, Leo Breiman, on the importance of low correlation for good generalization. Finally, review the section Variance in Predictions. The visualization here perfectly illustrates our key point: different trees make different mistakes, and the ensemble performs better by aggregating their diverse "opinions."
In essence, Random Forest turns the high variance of individual decision trees from a bug into a feature. It harnesses this instability by ensuring that each tree is unstable in a different way, then averages out this varied instability to produce a stable, accurate final model.
Conclusion
In this lesson, we've unpacked the Random Forest algorithm and discovered how it builds upon the principles of bagging to create an even more powerful predictive model.
Key Takeaways:
- Evolution from Bagging: Random Forest improves upon bagged trees by adding an extra layer of randomness to actively decorrelate the trees.
- Feature Randomness: The core innovation is that at each split in a tree, the algorithm considers only a random subset of features, not all of them.
- Why It Works: This process ensures the trees in the forest are diverse. When their predictions are aggregated, their individual errors—which are now less correlated—tend to cancel each other out more effectively.
- Improved Generalization: The result is a greater reduction in model variance compared to standard bagging, without a significant increase in bias. This leads to a robust model that generalizes much better to unseen data.
You now understand the "secret sauce" that makes Random Forest one of the most reliable and widely used algorithms in machine learning. In our next lesson, we will put this theory into practice. You'll apply a Random Forest classifier to a real-world dataset using scikit-learn and compare its performance directly against the single decision tree we built previously.