Hello! Welcome back to our course on Altery Advanced Certification.
In our last lesson, we took a practical look at how to split a dataset into training, validation, and holdout sets using the Create Samples tool in Alteryx. We also briefly touched on the different roles these datasets play.
Today, we will dive deeper into the "why" behind this process. The learning outcome for this lesson is to explain the importance of data partitioning for unbiased model validation. We will explore core machine learning concepts like generalization, overfitting, and data leakage to understand why simply splitting your data isn't just a good practice—it's the only way to build a predictive model you can truly trust.
1. The Goal of Predictive Modeling: Generalization
The ultimate goal of building a predictive model is not to perfectly predict the data you already have; it is to create a model that generalizes well. Generalization is the model's ability to make accurate predictions on new, previously unseen data.
Think about it: a model that can tell you with 100% accuracy which of your current customers will churn is interesting, but not very useful. What you really want is a model that can accurately predict which of your future customers are likely to churn, so you can take action.
This is why we need an "honest assessment" of our model's performance. If we build a model and test it on the very same data we used to train it, we get a skewed and overly optimistic result.
To understand this better, let's read a short excerpt that formalizes this idea.
How to Avoid Data Leakage When Imputing for Predictive Modeling
This article from the SAS Communities Library, titled "How to Avoid Data Leakage...", gives an excellent, concise explanation of the goal of predictive modeling and the role of data partitioning.
Please read the section titled Data partitioning. Focus on the sentences that explain the goal of generalization and why assessing a model on its training data provides an "optimistically biased assessment".
As the article states, we need to simulate applying a model built today to new data seen tomorrow. Data partitioning is the mechanism that allows us to do this.
2. The Problems: Overfitting and Data Leakage
If we're not careful, two major problems can prevent us from achieving good generalization and an unbiased assessment: overfitting and data leakage.
Overfitting: Memorizing the Answers
Overfitting occurs when a model learns the training data too well. Instead of learning the general patterns, it also learns the random noise and specific quirks of the training set.
An analogy is a student preparing for an exam.
- A student who generalizes studies the concepts and can answer any question on the topic.
- A student who overfits memorizes the answers to the specific questions in the practice test. They will ace the practice test but will likely fail the real exam, which has different questions.
A model that overfits will have fantastic performance on the training data but will fail when it encounters new data.
Data Leakage: Cheating on the Exam
Data leakage is a more subtle but equally damaging problem. It happens when information from outside the training set "leaks" into the model during the training process. In effect, the model is "cheating" by getting a sneak peek at the data it will be evaluated on.
The SAS article you just read provides a clear definition of this.
How to Avoid Data Leakage When Imputing for Predictive Modeling
Let's return to the same article to get a formal definition of data leakage.
Please read the section titled Data leakage. Focus on the definition and the consequences of leakage, such as getting an "unfair advantage" that won't exist in the real world.
A common way data leakage occurs is by performing data preparation steps (like imputation, which we covered earlier) on the entire dataset before partitioning it. If you calculate the average of a column using all your data and then use that average to fill missing values, your training records have been influenced by information from your validation and test records.
3. The Solution: A Disciplined Partitioning Strategy
To prevent overfitting and data leakage, we use a strict separation of our data into three sets.
This short video provides an excellent, intuitive explanation of why all three sets are necessary for an unbiased result.
Intuition: Training Set vs. Test Set vs. Validation Set
This video from IntuitiveML, "Intuition: Training Set vs. Test Set vs. Validation Set", clearly explains why you can't reuse sets and why the final test set is so crucial.
Watch from the beginning until the timestamp 01:29. Pay close attention to the reasoning for why you cannot use the training set (memorization) or the validation set (model bias) for the final performance report.
Let's summarize the roles in the context of unbiased validation:
-
Training Set: This is where the model learns. We fit the model's parameters using only this data. We expect the performance here to be high, but we know it's optimistically biased.
-
Validation Set: This is where we tune the model and make choices. We might train several different models (or versions of the same model with different settings). We then see which one performs best on the validation data. Because we selected our champion model based on its performance on this specific dataset, we have introduced "model bias," as the video explains. The validation set has done its job, but we can't use it for the final report card because it influenced our decision.
-
Test (Holdout) Set: This is the "final exam." This data has been locked away throughout the entire process of training and selection. It has not been used to fit parameters or to choose the best model. By evaluating our final, chosen model on this completely unseen data, we get the most realistic and unbiased assessment of how it will perform in the real world. This is our "honest assessment."
Conclusion
Today we've moved from the "how" to the crucial "why" of data partitioning. Building a reliable predictive model is as much about the validation process as it is about the algorithm you choose. By rigorously separating your data, you can build models that generalize well and provide real-world value.
Key Takeaways:
- The goal of predictive modeling is generalization—the ability to perform well on new, unseen data.
- Evaluating a model on the data used to train it gives an optimistically biased assessment due to overfitting.
- Data leakage occurs when information from the validation or test sets contaminates the training process, leading to an unfair and unrealistic performance boost.
- A three-part split (Training, Validation, Test) allows for a disciplined workflow:
- Train the model.
- Validate and select the best model.
- Test the final model for an unbiased, honest assessment of its performance.
Preview of the Next Lesson:
Now that we have established the foundational principles for preparing and validating models, we are ready to start building them. In our next lesson, we will begin Module 8 by constructing our first predictive model in Alteryx: a Linear Regression model. We will learn how to configure the tool, interpret its output, and use our partitioned data to evaluate its performance.