Hello! Welcome to the final lesson of our first module, "Time Series Fundamentals."
So far, we have covered the essential groundwork: defining time series, identifying their characteristics, decomposing them, and even transforming them to achieve stationarity. These are all crucial steps in understanding and preparing your data. Now, we'll tackle the last fundamental step before we can begin building models: correctly splitting the data for training and evaluation.
This lesson addresses how to split time series data into training, validation, and test sets while respecting its temporal order. This might sound straightforward, but it's one of the most common pitfalls for those new to time series forecasting.
1. The Critical Flaw of Random Splitting
In many machine learning tasks, like image classification or customer churn prediction, you might be used to randomly shuffling your dataset before splitting it into training and test sets. Scikit-learn's train_test_split function, a workhorse for many data scientists, even has shuffle=True as its default.
For time series data, this approach is fundamentally wrong and must be avoided.
To understand why, let's start by reading about the problems it causes.
Train-Test Split Strategies for Time Series Data
The following resource, 'Train-Test Split Strategies for Time Series Data', clearly explains why random splitting is invalid for time-ordered data.
Please read the introduction and the section 'The Problem with Random Splitting'. Focus on understanding the concepts of Data Leakage and Loss of Temporal Structure.
As the article explains, shuffling time series data allows your model to "peek into the future." During training, it would learn from data points that occur after the data points it's being tested on. This creates an illusion of high performance that will not hold up in a real-world scenario where the future is unknown. This phenomenon is called data leakage.
2. The Correct Approach: Chronological Splitting
To evaluate a forecasting model properly, the splitting strategy must simulate a real-world scenario: we use the past to predict the future. This means our splits must always be chronological.
We typically use a three-way split:
- Training Set: The oldest portion of the data. The model learns the underlying patterns from this set.
- Validation Set: The portion of data immediately following the training set. This set is used to tune model hyperparameters and select the best-performing model architecture.
- Test Set: The most recent portion of the data. This set is held back—or kept "sacred"—until the very end. It provides a final, unbiased assessment of how the selected and tuned model is expected to perform on new, unseen data.
This chronological ordering can be visualized as:
|----------- Training Set -----------|---- Validation Set ----|---- Test Set ----|Earliest Time -----------------------------------------------------> Latest Time
Data Splits in ML: Training, Validation, and Test Sets
This article, 'Data Splits in ML', provides a good overview of the roles of each dataset split. While it covers general machine learning, it has a specific, concise section on time series.
First, read the sections 'Training Set,' 'Validation Set,' and 'Test Set' to solidify your understanding of each part's purpose. Then, read the short section 'Time-Series Splits,' which reinforces the chronological splitting principle.
3. Implementing a Chronological Split in Python
Now let's translate this concept into code. Using pandas, we can easily create these splits based on a specific date or by positional index.
3.1. Creating a Simple Train-Test Split
Let's start with a basic two-way split (training and testing). You can achieve this by defining a cutoff point.
Train-Test Split Strategies for Time Series Data
Let's return to the 'Train-Test Split Strategies' article to see how this is implemented in code.
Read the sections 'Temporal Splitting: Maintaining Order' and 'Implementing a Temporal Split in Pandas'. The code snippets show you how to split a pandas Series using both a date-based cutoff and a position-based cutoff.
The video below offers a quick, practical demonstration of this exact process.
Time Series Forecasting with XGBoost - Use python and machine learning to predict energy consumption
In this video segment from 'Time Series Forecasting with XGBoost' by Rob Mulla, you'll see a live demonstration of splitting a dataset into training and testing sets using a specific date as the cutoff.
Watch from 05:29 to 07:30. Notice how he creates the train and test dataframes by filtering the index based on a cutoff date and then visualizes the split.
3.2. Creating a Train-Validation-Test Split
Expanding this to a three-way split is just as simple. You just define two cutoff points. For example, using a positional split on a dataset with 1000 points, you might do:
- Training: First 700 points (
.iloc[:700]) - Validation: Next 150 points (
.iloc[700:850]) - Test: Final 150 points (
.iloc[850:])
3.3. How to Choose the Split Points?
The size of your validation and test sets is an important choice. Here are some guidelines:
- Forecast Horizon: Your test set should be at least as long as the maximum period you want to forecast. If you need to predict sales for the next 12 months, your test set must contain at least 12 months of data.
- Seasonality: If your data has a yearly seasonal pattern, your test set should ideally cover at least one full year to ensure the model's ability to handle seasonality is properly evaluated.
- Data Availability: For large datasets, a 80-10-10 split (train-validation-test) is common. For smaller datasets, you may need a larger training percentage to ensure the model can learn the patterns effectively.
Test your understanding!
You have been tasked with building a model to forecast daily electricity demand for the next 7 days. You have 5 years of historical daily data. A colleague suggests splitting the data by taking 80% of the rows for training and 20% for testing, after shuffling the entire dataset to ensure the training data is "representative".
Critique this approach and propose a better one.
Show answer
The colleague's approach is incorrect for two main reasons:
- Shuffling: Shuffling the data breaks the temporal order, leading to data leakage. The model would be trained on future data points to predict past ones, resulting in misleadingly high accuracy.
- No Validation Set: Using only a train and test set encourages tuning the model based on its performance on the test set. This means the final performance metric will be biased and overly optimistic.
A much better approach would be:
- Do not shuffle the data. Keep it in chronological order.
- Create a three-way split. For 5 years (~1825 days) of data:
- Test Set: The most recent
Ndays. Since the forecast horizon is 7 days, the test set must be at least 7 days long. To get a more robust evaluation across different seasons, a larger test set, like the last 6 months or 1 year (~180-365 days), would be better. Let's reserve the last year (365 days). - Validation Set: The year of data before the test set (365 days).
- Training Set: The remaining first 3 years of data.
- Test Set: The most recent
This approach respects the temporal order, provides a dedicated set for tuning, and ensures the final test is a fair evaluation on truly unseen future data.
4. A More Robust Method: Time Series Cross-Validation
A single train-validation-test split is good, but its main weakness is that the evaluation is based on just one specific time period. Model performance might be unusually good or bad on that particular slice of data.
A more robust and reliable technique is Time Series Cross-Validation, also known as walk-forward validation or rolling forecast origin.
Cross-Validation for Time Series Forecasting | Python Tutorial
This excellent tutorial from Egor Howell explains why standard cross-validation fails for time series and introduces the correct 'rolling' approach.
Please watch from 03:35 to 10:22. The video covers: The data leakage problem in standard cross-validation. The concept of a 'rolling test' or walk-forward validation. A practical demonstration using scikit-learn's TimeSeriesSplit function. A clear summary of the main rule: your test set must always be forward in time.
The core idea of walk-forward validation is to create several train/test splits that move forward through time. This gives us multiple evaluation scores, providing a more stable and trustworthy estimate of our model's performance.
There are two main variations:
- Rolling/Sliding Window: The training window size is fixed. As we move forward, the window slides, dropping the oldest data.
- Expanding Window: We start with a minimum amount of training data and gradually add more data to it in each step. The training set "expands" over time.
Scikit-learn's TimeSeriesSplit implements the expanding window approach.
Using TimeSeriesSplit is highly recommended when you need to rigorously tune hyperparameters and get a confident estimate of your model's generalization ability.
Conclusion
Congratulations on completing the first module! You now have a solid foundation in the principles of time series analysis. Properly splitting your data is the gatekeeper to reliable model evaluation. Getting this wrong can invalidate all your subsequent work.
Key Takeaways:
- Never shuffle time series data before splitting. It causes data leakage and leads to overly optimistic performance metrics.
- Always perform a chronological split: train on the past, validate on the near future, and test on the distant future.
- Use a train-validation-test split to prevent your model selection and tuning process from being biased by the final test set.
- For a more robust performance estimate, use time series cross-validation (walk-forward validation), which can be easily implemented with tools like scikit-learn's
TimeSeriesSplit.
In the next module, "Data Preprocessing for Time Series," we will build on this foundation. Our first lesson will tackle a very practical problem: how to handle missing values in your time series data.