Categorical Data Encoding

1. Understanding Categorical Data

First, let's clarify what we're working with. Categorical data represents discrete, named groups. It's crucial to distinguish between its two main types, as this distinction dictates which encoding strategy you should use.

• Nominal Data: The categories have no intrinsic order or ranking. For example, the township column in a housing dataset ('Monroeville', 'West Windsor', 'Robinsville') is nominal. There's no inherent order suggesting one town is "greater" or "less than" another.
• Ordinal Data: The categories have a meaningful order or rank. For example, a feature for education level ('High School', 'Bachelors', 'Masters', 'PhD') is ordinal because there's a clear progression.

To dive deeper into this distinction and see why simply assigning numbers (e.g., Monroeville=0, West Windsor=1) is problematic for nominal data, please watch the beginning of the following video.

This tutorial from DataCamp also provides clear, concise definitions of nominal and ordinal data that you can use as a reference.

2. Label Encoding for Ordinal Data

As you just saw, naively assigning numbers to nominal categories can be a bad idea. However, what if the data is ordinal? In that case, converting categories to integers is the right approach. This technique is called Label Encoding or Ordinal Encoding.

For example, if you have a feature size with categories ['small', 'medium', 'large'], encoding them as [0, 1, 2] preserves the inherent order.

Scikit-learn provides the OrdinalEncoder for this purpose. It maps each category to an integer. By default, it determines the order alphabetically, but you can—and should—provide the correct order for your ordinal features to ensure the encoding is meaningful.

Let's look at how this works in practice.

The key takeaway is that Label/Ordinal Encoding should primarily be used for features where the categories have a clear, logical ranking. Using it on nominal data can cause a model to learn a false relationship (e.g., that 'Blue' is greater than 'Green') and make incorrect predictions.

3. One-Hot Encoding for Nominal Data

So, how do we handle nominal data? The most common and effective technique is One-Hot Encoding.

The idea is to create a new binary (0 or 1) column for each unique category in the original feature. For each row, a '1' is placed in the column corresponding to its category, and '0's are placed in all other new columns.

This avoids the problem of artificial ordering because no single column contains a range of values. Each category is represented by its own on/off switch.

For a crystal-clear conceptual explanation, the following StatQuest video is excellent.

Practical Implementation

There are two main ways to perform one-hot encoding in Python.

A. Using Pandas get_dummies()

This is the quickest way to perform one-hot encoding. It's a simple function that takes a DataFrame or Series and returns a new DataFrame with the encoded columns.

The Dummy Variable Trap

When you one-hot encode a feature with k categories, you get k new columns. However, one of these columns is perfectly predictable from the others. For example, if you know a person is not from 'West Windsor' and not from 'Robinsville', they must be from 'Monroeville'. This redundancy is called multicollinearity and can be problematic for some models, particularly linear ones.

To avoid this "dummy variable trap," a common practice is to drop one of the one-hot encoded columns. The model can infer the dropped category when all other binary columns are 0.

B. Using Scikit-learn OneHotEncoder

While get_dummies is convenient for quick analysis, OneHotEncoder from scikit-learn is the preferred method within a machine learning pipeline.

Why? The key reason is consistency. Just like with StandardScaler from our last lesson, you need to fit the encoder only on your training data and then transform both your training and test sets. OneHotEncoder allows you to do this, preventing data leakage. It learns the categories from the training set and can then handle new data consistently. It can also be configured to gracefully handle categories that appear in the test set but were not in the training set.

This makes OneHotEncoder much more robust for building and deploying models.

To automatically avoid the dummy variable trap with OneHotEncoder, you can initialize it like this: OneHotEncoder(drop='first').

4. Choosing the Right Encoding Strategy

You now have two main tools in your toolbox. Here’s a summary of when to use each:

This article provides an excellent summary of these points.

Conclusion

Congratulations! You've just learned another essential data preprocessing skill. You now know how to convert non-numeric categorical data into a format that machine learning models can understand, a process you'll perform in nearly every project.

Key Takeaways:

• Machine learning models require all input features to be numeric.
• Label/Ordinal Encoding converts categories into integers. It's best suited for ordinal data where a natural order exists.
• One-Hot Encoding creates new binary columns for each category. It's the standard choice for nominal data to avoid imposing a false order.
• Be mindful of the dummy variable trap (multicollinearity) when using one-hot encoding and use parameters like drop_first=True (pandas) or drop='first' (scikit-learn) to mitigate it.
• For building robust ML pipelines, prefer scikit-learn's OneHotEncoder and OrdinalEncoder over pandas' get_dummies because they can be saved and applied consistently across training and test data.

Preview of the Next Lesson:

We have now covered preprocessing for both numerical and categorical features. However, sometimes the distribution of a numerical feature itself can be problematic. In our next lesson, we will explore how to handle skewed data distributions using mathematical transformations like the log transform, which can help certain models perform better.