Random Forests: Beyond Single Trees

From Bagging to Random Forests

A Random Forest is essentially a specialized version of Bagging applied to decision trees, with one extra ingredient to improve performance. Let's recap the Bagging process for trees:

1. Bootstrap Sampling: Create many random subsamples of the training data with replacement.
2. Train Trees: Train a separate decision tree on each subsample.
3. Aggregate: Combine the predictions of all trees (e.g., majority vote for classification).

This process helps because the individual trees are trained on different data, making them diverse. However, what if your dataset has one very strong, predictive feature? In standard Bagging, most of the trees would likely select this feature for their first split. This would make the trees highly correlated, and their combined prediction would not be as diverse or robust as we'd like.

Random Forest solves this with a simple trick.

The Key Ingredient: Feature Randomness

The key innovation of Random Forest is to introduce randomness not just in the data sampling, but also in the feature selection process for building each tree.

When building each tree in the forest, at every split point, the algorithm does not consider all available features. Instead, it randomly selects a subset of features and only looks for the best split among that subset.

This has a powerful effect:

• It de-correlates the trees: By forcing some trees to use less-predictive features at certain splits, it prevents all trees from latching onto the same "strongest" features. This ensures the trees in the forest are more independent and diverse.
• It improves generalization: A more diverse "committee" of trees, where each member has a slightly different perspective, makes for a stronger and more generalized final model.

To see this idea explained, let's watch a segment from the StatQuest video we started in the previous lesson.

Now, let's watch another short video that puts it all together—the bootstrapping, the feature randomness, and the final voting—to give you a complete picture of how a Random Forest works.

Training a Random Forest in Python

Now that you understand the "why," let's move to the "how." Training a Random Forest classifier in scikit-learn is straightforward, thanks to the RandomForestClassifier class. It lives in the sklearn.ensemble module, right alongside the BaggingClassifier.

The most important hyperparameter you'll encounter is n_estimators, which simply defines the number of trees to build in the forest. A common starting point is 100.

Here is a complete, self-contained example. We will:

1. Import the necessary libraries and create a synthetic dataset.
2. Split the data into training and testing sets.
3. Initialize RandomForestClassifier with n_estimators=100.
4. Fit the model on the training data and evaluate it on the test data.

from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score




# Create a synthetic dataset for demonstration
X, y = make_classification(n_samples=1000, 
                           n_features=20, 
                           n_informative=5, 
                           n_redundant=0, 
                           random_state=42)




# Split data into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)




# 1. Initialize the Random Forest Classifier
# We specify the number of trees in the forest with n_estimators
rf_classifier = RandomForestClassifier(n_estimators=100, random_state=42)




# 2. Train (fit) the model on the training data
rf_classifier.fit(X_train, y_train)




# 3. Make predictions on the test data
y_pred = rf_classifier.predict(X_test)




# 4. Evaluate the model's accuracy
accuracy = accuracy_score(y_test, y_pred)
print(f"Random Forest Accuracy: {accuracy:.4f}")

This code trains a forest of 100 trees and uses their collective vote to make predictions. For a more detailed walkthrough of implementing a Random Forest on a real-world dataset, you can refer to the following guide.

Why Random Forests are an Improvement

The combination of bootstrap sampling and feature randomness makes Random Forests highly effective and robust.

• Reduces Overfitting: By averaging the predictions of many de-correlated trees, the model's variance is significantly reduced. The individual errors of each tree tend to cancel out, leading to a model that generalizes much better to new, unseen data.
• Handles High-Dimensionality: The random feature selection makes it effective even when you have a very large number of features.
• Good Out-of-the-Box Performance: Random Forests often work well with default hyperparameters, making them a great baseline model to try on a new problem.

The visual below compares the decision boundary of a single decision tree with that of a Random Forest. Notice how the Random Forest boundary is much smoother and less jagged. This is a visual representation of lower variance—it's less sensitive to individual data points and captures the general trend more effectively.

Conclusion

In this lesson, we demystified the Random Forest algorithm, seeing it as a clever and powerful extension of the Bagging technique we learned previously.

Key Takeaways:

• A Random Forest is an ensemble of decision trees that combines bootstrap aggregating (bagging) with feature randomness.
• The key improvement is that at each split in a tree, the algorithm considers only a random subset of features, which de-correlates the trees and makes the ensemble more robust.
• This combination drastically reduces variance and protects against overfitting, making Random Forest one of the most reliable and widely used machine learning models.
• You can easily train a Random Forest using RandomForestClassifier from scikit-learn.

Preview of the Next Lesson

We've now covered decision trees and powerful ensemble methods built upon them. We'll now shift our focus to a completely different approach to classification. In the next module, we will explore instance-based learning with the K-Nearest Neighbors (K-NN) algorithm and dive into probabilistic models with Naive Bayes. These models make predictions based on fundamentally different principles than the rule-based splitting we've seen in trees.