Data Cleaning for Movie Titles and Countries

1. Foundational Cleaning with pandas

Your experience as an ML engineer means you're likely familiar with pandas. For this task, we'll leverage its powerful and efficient vectorized string methods, which operate on entire columns (Series) at once. These are the first line of defense for most common data cleaning tasks.

The most common inconsistencies you'll face are:

• Case differences: "France" vs. "france"
• Leading/trailing whitespace: " It's a Wonderful Life " vs. "It's a Wonderful Life"
• Unwanted characters: "Die Hard™" vs. "Die Hard"

Pandas provides a .str accessor on Series objects that makes fixing these issues straightforward.

The key takeaway is that you can chain these methods together for a clean and highly readable workflow. For example, to clean a country column in a DataFrame called df:

# A common cleaning pipeline
df['country_cleaned'] = df['country'].str.lower().str.strip()

Similarly, for movie titles, you might remove punctuation or specific substrings.

# Example for movie titles
df['title_cleaned'] = df['title'].str.replace('[^\w\s]', '', regex=True).str.lower().str.strip()

This example uses a simple regex [^\w\s] to remove any character that is not a word character or whitespace. We'll dive deeper into regex next.

This first pass handles the easy cases and provides a consistent baseline for the more complex steps that follow.

2. Pattern-Based Extraction with Regular Expressions (Regex)

Sometimes, you don't want to just remove characters; you want to extract structured information. A classic example is a movie title that includes the year, like "A Christmas Story (1983)". For your analysis, you'll likely want to separate the title ("A Christmas Story") from the year ("1983"). Regular Expressions, or regex, are the perfect tool for this kind of pattern matching and extraction.

The pandas .str.extract() method allows you to apply a regex pattern with capturing groups (...) to a Series. Each capturing group extracts its matched content into a new column.

The video below gives a practical tutorial on building regex patterns and using capturing groups in Python.

Let's apply this to our movie title problem. Suppose we have a column of titles like "Movie Title (YYYY)". We can define a regex pattern with two capturing groups: one for the title and one for the year.

# Assume df['raw_title'] contains titles like "Klaus (2019)"
# Pattern:
# (.*?)      - Group 1: Match any character (non-greedy) for the title
# \s*        - Match any whitespace
# \((\d{4})\) - Group 2: Match a literal '(', then 4 digits, then a literal ')'
pattern = r'(.*?)\s*\((\d{4})\)'

# Use .str.extract to create new columns
df[['clean_title', 'year']] = df['raw_title'].str.extract(pattern)

# Now, df['clean_title'] would have "Klaus" and df['year'] would have "2019"

import re

movie = "Die Hard II"
# Pattern: (anything)(whitespace)(one or more capital letters from I, V, X) at the end of the string
pattern = r'^(.*)\s+([IVXLCDM]+)$'

match = re.search(pattern, movie)

if match:
    title = match.group(1).strip() # "Die Hard"
    numeral = match.group(2)       # "II"
    print(f"Title: {title}, Numeral: {numeral}")

3. Handling Ambiguity with Fuzzy String Matching

You've cleaned up formatting and extracted structural patterns. What about the hardest cases?

• Typos: "The Grinch Who Stole Christms"
• Abbreviations: "USA", "U.S.A.", "United States of America"
• Word order variations: "The Polar Express" vs. "Polar Express, The"

This is where rule-based methods like .replace() and regex fail. We need a method that can measure the similarity between strings. This is called fuzzy string matching. The most common technique is based on the Levenshtein distance, which counts the minimum number of single-character edits (insertions, deletions, or substitutions) required to change one string into the other.

A fantastic Python library for this is thefuzz.

The most powerful application for our project is to create a "canonical" list of names (e.g., a definitive list of countries or a list of top 25 Christmas movies) and then map our messy data to that clean list.

The thefuzz.process.extractOne function is perfect for this. For a given messy string, it finds the best match from a list of choices. The following reading shows exactly how to integrate this into a pandas workflow to clean up country names—a task directly relevant to our project.

Here's how you'd implement this strategy:

1. Create a canonical list. This is your "source of truth".

   canonical_countries = ['United States of America', 'United Kingdom', 'Canada', 'Germany', 'France']
   

2. Use apply and process.extractOne to map messy data.

   from thefuzz import process
   
   # Assume df['messy_country'] has values like "USA", "UK", "Germany.", "Canda"
   def standardize_country(name):
       # Find the best match from the canonical list
       # It returns a tuple (match, score), so we take the first element [0]
       # We can also add a score_cutoff to avoid bad matches
       match, score = process.extractOne(name, canonical_countries)
       if score >= 80: # Set a confidence threshold
           return match
       else:
           return None # Or return the original name, or flag for review
   
   df['standardized_country'] = df['messy_country'].apply(standardize_country)
   

This approach combines the power of fuzzy matching with the scalability of pandas, allowing you to clean thousands of rows of inconsistent data with a few lines of code. For movie titles, where word order can vary (token_sort_ratio) or subtitles may be present (token_set_ratio), choosing the right scorer in extractOne is key.

Conclusion

You now have a comprehensive, three-tiered toolkit for cleaning and standardizing textual data. This is often one of the most time-consuming parts of a data project, but mastering these techniques will make your work significantly more efficient and your results more reliable.

Key Takeaways:

• Always start with basic pandas string methods (.str.lower(), .str.strip(), .str.replace()) for quick, high-impact fixes.
• Use Regular Expressions with pandas.str.extract() to parse strings that have a consistent, repeating structure, like a movie title followed by a year in parentheses.
• For complex variations like typos, abbreviations, or different phrasings, use fuzzy string matching with thefuzz to map messy data to a pre-defined canonical list.

With our data sources scraped and our key categorical fields (country and movie title) cleaned and standardized, we are now in an excellent position to bring everything together.

In the next lesson, we will focus on the final step of our data preparation module: Consolidate cleaned data into a single tidy dataset with columns for country, movie title, viewing metrics, and date.