Filtering Data for Pre-Christmas Analysis

1. The Prerequisite: Working with datetime Objects

Before we can filter by date, we must ensure our dates are stored in a format that Python and pandas understand chronologically. A simple string like "2022-11-25" is just text to the computer; it doesn't inherently know that it comes before "2022-12-01".

The first and most critical step is to convert any date-like columns into datetime objects. The pandas function pd.to_datetime() is the standard tool for this. It's smart enough to recognize most common date formats automatically, but can also be guided to parse more unusual formats, which connects to the pattern-matching concepts from our last lesson.

The video below provides an excellent demonstration of this process.

Once a column is converted, pandas unlocks a special .dt accessor. This accessor lets you easily extract properties like the year, month, day, or day of the week from each datetime object. For example:

# Assuming 'df' is your DataFrame and 'viewing_date' is a string column
df['viewing_date'] = pd.to_datetime(df['viewing_date'])

# Now you can access datetime properties
months = df['viewing_date'].dt.month # This returns a Series of integers (1-12)
years = df['viewing_date'].dt.year   # This returns a Series of years

This .dt accessor is the key that enables all subsequent filtering techniques.

2. Filtering by Date Range

With our date column properly formatted, we can now select the data from November and December. There are two primary and highly effective methods to achieve this.

Method 1: Boolean Masking

This is a flexible and intuitive approach that you've likely encountered in your work with pandas and NumPy. The idea is to create a "mask"—a boolean Series (True/False)—that identifies which rows meet our criteria. We can then use this mask to select the desired rows from the DataFrame.

To define our pre-Christmas period, we need to combine two conditions:

1. The month is November (11) OR the month is December (12).
2. (Optional but recommended) The year is the specific year we are analyzing.

The following video explains the syntax for combining multiple conditions in pandas.

Applying these concepts, we can filter our data. Let's say we want data for Nov/Dec 2022.

# Assume 'df' has a datetime column 'viewing_date'

# Method A: Using the | (OR) operator
is_november = (df['viewing_date'].dt.month == 11)
is_december = (df['viewing_date'].dt.month == 12)
is_2022 = (df['viewing_date'].dt.year == 2022)

pre_christmas_df = df[(is_november | is_december) & is_2022]


# Method B: Using .isin() for a cleaner approach
months_of_interest = [11, 12]
is_pre_christmas_period = df['viewing_date'].dt.month.isin(months_of_interest)

pre_christmas_df = df[is_pre_christmas_period & is_2022]

A very powerful feature of datetime columns is that you can also compare them directly with date strings. This often leads to the most readable code. The following article and image illustrate this perfectly.

This direct comparison is often the preferred method:

# Method C: Direct comparison with date strings (most readable)
start_date = '2022-11-01'
end_date = '2022-12-31'

mask = (df['viewing_date'] >= start_date) & (df['viewing_date'] <= end_date)
pre_christmas_df = df.loc[mask]

import pandas as pd

# Assume df_views is already created and watch_date is a datetime column
start = '2022-11-15'
end = '2022-12-24'

mask = (df_views['watch_date'] >= start) & (df_views['watch_date'] <= end)
filtered_df = df_views.loc[mask]

# You could also write it in a single line:
# filtered_df = df_views.loc[(df_views['watch_date'] >= '2022-11-15') & (df_views['watch_date'] <= '2022-12-24')]

Method 2: Using a DatetimeIndex

For datasets where time is the primary axis of analysis, pandas offers a more powerful and efficient structure: the DatetimeIndex. By setting your date column as the DataFrame's index, you can select data using intuitive slicing, similar to how you would slice a list in Python.

This method has two steps:

1. Set the datetime column as the index using df.set_index('your_date_column', inplace=True).
2. Select date ranges using df.loc[].

The same video from Corey Schafer that we started with also covers this elegant technique.

Here is how you would apply this to our problem:

# Assume df is your DataFrame and 'viewing_date' is a datetime column
df_indexed = df.set_index('viewing_date')

# Now, you can slice directly with strings!
# This selects all data from November and December 2022
pre_christmas_df = df_indexed.loc['2022-11':'2022-12']

This method is not only highly readable but can also be more performant on very large time series datasets.

Conclusion

You have now learned the essential techniques for zeroing in on a specific time window within your data. This filtering is what transforms a general dataset into one that is tailored to answer a specific, time-sensitive question.

Key Takeaways:

• Always convert to datetime: Before any time-based filtering, ensure your date column is converted using pd.to_datetime().
• Boolean masking is flexible: Creating masks using logical operators (&, |) and direct date comparisons gives you fine-grained control over selection. The .isin() method is particularly useful for selecting multiple months.
• DatetimeIndex is efficient and elegant: For time-centric data, setting a DatetimeIndex allows for clean, intuitive slicing and can improve performance.

In our journey so far, we have sourced, scraped, cleaned, standardized, and now filtered our data. The pieces are ready.

In the next lesson, we will perform the final step of our data preparation: Consolidate cleaned data into a single tidy dataset with columns for country, movie title, viewing metrics, and date. This will create the master dataset upon which we will build all our analyses and visualizations.