Skip to main content
Back to path

Welcome to a Grasp lesson

Create a free account to keep studying and create your own study path.

Lesson 11

Categorical Encoding Techniques

Recommended resources

Welcome to our next lesson in data preprocessing. In our previous session, we focused on handling numerical features by applying normalization and standardization, ensuring that our models treat all features fairly regardless of their original scale. We also reinforced the critical workflow of splitting data before any transformation to prevent data leakage.

Today, we turn our attention to another common data type: categorical variables. Machine learning algorithms are fundamentally mathematical and expect numerical inputs. This lesson will equip you to translate non-numeric categories—like "color," "country," or "product type"—into a numerical format that models can understand. We will explore the three most fundamental techniques: one-hot, label, and ordinal encoding. Most importantly, you will learn the criteria for choosing the right method for the right situation, a crucial skill in building effective models.

What is Categorical Data?

First, let's clarify what we mean by categorical data. These are variables that contain label values rather than numerical values, drawn from a fixed set of categories. It's essential to distinguish between two types, as this distinction dictates our encoding strategy.

  • Nominal Data: The categories have no intrinsic order or ranking. For example, country ('USA', 'India', 'Canada') or color ('Red', 'Green', 'Blue'). 'Canada' is not "greater than" or "less than" 'USA'; they are simply different.
  • Ordinal Data: The categories have a meaningful, inherent order. For example, size ('Small', 'Medium', 'Large') or customer_satisfaction ('Poor', 'Average', 'Good', 'Excellent'). Here, 'Large' is clearly greater than 'Small'.

The following video provides a clear and concise overview of these two types.

Encoding Categorical Data | Machine Learning Fundamentals

This video from Kody Simpson's "Machine Learning Fundamentals" series introduces categorical data and clearly explains the difference between nominal and ordinal types.

Watch the segment from the beginning to understand the core definitions and see concrete examples of nominal versus ordinal data.

Most machine learning libraries, including scikit-learn, require these string labels to be converted into numbers. Let's explore how to do this correctly.

Encoding Strategy 1: Ordinal and Label Encoding

The most straightforward approach is to assign a unique integer to each category. This is known as ordinal encoding when applied to input features (X) and label encoding when applied to the target variable (y). For example, for a size feature with categories ['Small', 'Medium', 'Large'], we could map them to [0, 1, 2].

This method is perfectly suited for ordinal data, where the numerical order reflects the intrinsic order of the categories. The model can then potentially learn a linear relationship, such as "as the 'size' number increases, the price increases."

However, applying this to nominal data creates a serious problem. If we encode a color feature as 'Red': 0, 'Green': 1, 'Blue': 2, the model might incorrectly infer an ordered relationship, such as Blue > Green > Red. This arbitrary order can mislead the model and lead to poor performance.

The legendary StatQuest channel explains this pitfall very clearly.

One-Hot, Label, Target and K-Fold Target Encoding, Clearly Explained!!!

In this StatQuest video, Josh Starmer explains the concept of label encoding and why the arbitrary numbers it assigns can cause problems for some machine learning models.

Watch the segment on label encoding. Pay close attention to how a decision tree might be forced into making illogical splits because of the artificial order introduced.

In scikit-learn, OrdinalEncoder is used for features. It's crucial to use the categories parameter to explicitly define the order, ensuring the encoding reflects the true relationship in your data. For the target variable, LabelEncoder is used, which is simpler as it works on a 1D array.

Ordinal and One-Hot Encodings for Categorical Data

This article from Machine Learning Mastery provides a solid foundation on encoding, starting with the types of categorical data and then diving into Ordinal Encoding.

Read the following sections: Start with the definitions to solidify your understanding of the two types of categorical data. Then, read the section on Ordinal Encoding for a clear explanation of the technique. Finally, look at the code example demonstrating the OrdinalEncoder. Notice how the default behavior assigns integers alphabetically. This reinforces why manually specifying the order via the categories argument is so important for true ordinal data. The example is under the heading OrdinalEncoder example.

Encoding Strategy 2: One-Hot Encoding

So, how do we correctly handle nominal data without introducing a false order? The answer is one-hot encoding.

This technique transforms a single categorical column with k unique categories into k new binary columns (features). For each row, the column corresponding to its original category is set to 1, and all other new columns are set to 0.

This image shows the difference between Label Encoding and One-Hot Encoding. Label Encoding assigns a single integer to each island, implying an order. One-Hot Encoding creates a new column for each island, representing the category with a 1 or 0 without any implied order.

This removes the issue of implied order, as the model now sees a set of independent binary features.

The Dummy Variable Trap
A subtle but important issue with one-hot encoding is that it introduces multicollinearity. The k new columns are perfectly correlated; if you know the values of k-1 columns, you can perfectly determine the value of the k-th column (if the first k-1 are all 0, the last one must be 1). For some models, particularly linear models, this can cause issues.

The solution is to drop one of the new columns. This is often called creating dummy variables. The dropped category becomes the "baseline" reference, represented by all zeros in the remaining columns. In scikit-learn's OneHotEncoder, you can achieve this by setting the drop='first' parameter.

Ordinal and One-Hot Encodings for Categorical Data

This article also provides a great explanation of One-Hot and Dummy Variable Encoding.

Read these two sections: The definition of one-hot encoding. The explanation of the dummy variable trap and why it's important for models like linear regression.

One-hot encoding is the standard choice for nominal data. Let's look at its implementation.

Encoding Categorical Data | Machine Learning Fundamentals

This video provides a practical demonstration of one-hot encoding using both pandas and scikit-learn.

Watch these two key parts: First, see how to implement it quickly with pandas using pd.get_dummies. This is a very convenient method for exploratory analysis. Next, watch the walkthrough of scikit-learn's OneHotEncoder. Pay attention to how the drop='first' parameter is used to create dummy variables and avoid multicollinearity.

A Decision Framework for Encoding

Choosing the correct strategy is crucial. Here is a simple framework:

  1. Is the variable Ordinal (has a meaningful order)?
    • Yes: Use sklearn.preprocessing.OrdinalEncoder. Always specify the correct order using the categories parameter.
  2. Is the variable Nominal (no inherent order)?
    • Yes: Use sklearn.preprocessing.OneHotEncoder.
    • If using a linear model (like Linear/Logistic Regression), set drop='first' to avoid multicollinearity.
    • If using a tree-based model (like Decision Trees, Random Forests), keeping all columns (drop=None) is often fine and can even make the tree structure more interpretable.
  3. Is it the target variable y for a classification problem?
    • Yes: Use sklearn.preprocessing.LabelEncoder. Since the model's goal is to predict these class labels, not interpret them as ordered features, direct integer mapping is standard practice.

This guide from the scikit-learn MOOC provides an excellent rule of thumb.

Encoding of categorical variables — Scikit-learn course

This section from a scikit-learn course offers a concise summary of when to choose each encoding strategy based on the model you intend to use.

Read the section Choosing an encoding strategy. The note distinguishing between linear models and tree-based models is a particularly valuable takeaway for practical applications.

Efficiently Applying Encoders with ColumnTransformer

In any real-world dataset, you'll have a mix of numerical, ordinal, and nominal columns. Applying different preprocessing steps (like scaling and encoding) to different columns can get messy.

Scikit-learn provides an elegant solution: the ColumnTransformer. This powerful tool allows you to build a single preprocessing object that applies specific transformers to specific columns. Given your background in software development, you'll appreciate how this encapsulates complex logic into a clean, reusable, and deployable pipeline component.

The following segment of the Kody Simpson video gives a perfect demonstration of how to combine OneHotEncoder and OrdinalEncoder into a single step using ColumnTransformer.

Encoding Categorical Data | Machine Learning Fundamentals

This final section of the video demonstrates the power and convenience of the ColumnTransformer.

Watch the segment on the ColumnTransformer. Observe how it allows you to define a list of transformations, each targeting specific columns, and apply them all at once.

Beyond the Basics: Other Encoding Methods

While one-hot, ordinal, and label encoding are the foundational techniques you must master, the world of encoding is vast. For features with very high cardinality (many unique categories), like 'zip_code', one-hot encoding can lead to a massive number of new features (the "curse of dimensionality" which we will discuss later).

In these cases, more advanced methods are used. The image below shows a few others, and one of the most powerful is Target Encoding. This involves replacing each category with the mean of the target variable for that category. It's a powerful technique but comes with a high risk of data leakage if not implemented carefully. We won't delve into it now, but it's good to be aware of it as a next step in your learning journey.

An overview of various encoding methods. In this lesson, we focused on One-Hot, Label, and Ordinal encoding, which form the bedrock of categorical data preparation.

Conclusion

In this lesson, we've tackled the essential task of converting categorical data into a numerical format suitable for machine learning. You've learned not just the "how" but, critically, the "when" and "why" for each of the core encoding techniques.

Key Takeaways:

  • Nominal vs. Ordinal: The distinction between unordered (nominal) and ordered (ordinal) data is the primary factor in choosing an encoding strategy.
  • Ordinal Encoding (OrdinalEncoder): Use for ordinal features. It maps categories to integers, preserving their natural order. Be sure to specify the order explicitly.
  • One-Hot Encoding (OneHotEncoder): The standard for nominal features. It creates new binary columns, avoiding any false sense of order. Use drop='first' to create dummy variables and mitigate multicollinearity for linear models.
  • Label Encoding (LabelEncoder): The standard for encoding the target variable (y) in classification tasks.
  • ColumnTransformer: An indispensable tool for applying different preprocessing steps to different columns in a clean and efficient pipeline.

Now that we have learned how to transform all our features into numerical data (through scaling for numbers and encoding for categories), we are ready for the next step in feature engineering: feature selection. In our next lesson, we will explore how to identify and remove features that have little or no predictive power, starting with a simple yet effective technique called variance thresholding.