Categorical Encoding: One-Hot vs. Label

1. Understanding Categorical Variables and Encoding Strategies

First, let's establish the different types of categorical data we'll encounter. This distinction is the most important factor in choosing an encoding method.

• Nominal Variables: These are categories without any intrinsic order. For example, country ('USA', 'France', 'Japan') or color ('Red', 'Blue').
• Ordinal Variables: These are categories that have a meaningful order. For example, education level ('High School', 'Bachelors', 'Masters') or customer satisfaction ('Low', 'Medium', 'High').

The goal of encoding is to convert these text labels into numbers. The two primary techniques we will cover are Label Encoding (or more appropriately, Ordinal Encoding) and One-Hot Encoding.

To get a clear initial understanding of these concepts, let's watch a short segment of a video.

As the video explained, simply assigning sequential numbers to a nominal variable like 'township' can mislead the model into thinking there's an order or a mathematical relationship between them (e.g., town_A + town_B = town_C), which is incorrect.

This brings us to our two main encoding strategies. This image provides a perfect visual summary of how they transform the data differently.

Now, let's clarify a common point of confusion: the difference between LabelEncoder and OrdinalEncoder in scikit-learn.

The key takeaway is:

• OrdinalEncoder is designed for encoding features (X).
• LabelEncoder is designed for encoding the target variable (y).

Although you might see older code or tutorials using LabelEncoder on features, it's best practice to use OrdinalEncoder for this task. We'll stick to this convention.

2. Ordinal Encoding: For Categories with Order

When your data is ordinal, like ['small', 'medium', 'large'], you want to convert it to numbers in a way that preserves this order, such as [0, 1, 2]. This allows the model to learn relationships like "as size increases, price increases." This is where OrdinalEncoder is the perfect tool.

How it Works and When to Use It

You use OrdinalEncoder when the categorical feature has a clear, inherent order. By encoding this order into a single numerical feature, you provide a strong signal to your model without adding extra complexity.

Let's read about its implementation and the importance of specifying the order.

Here's a practical Python example where we correctly specify the order for a size feature.

import pandas as pd
from sklearn.preprocessing import OrdinalEncoder




# Sample data with an ordinal feature
df = pd.DataFrame({
    'item': ['shirt', 'pants', 'shirt', 'hat'],
    'size': ['M', 'L', 'S', 'L']
})




# Define the correct order of categories
size_order = ['S', 'M', 'L']




# Create the encoder with the specified order
# The list of lists corresponds to the columns you are encoding
ordinal_encoder = OrdinalEncoder(categories=[size_order])




# Apply the encoder
df['size_encoded'] = ordinal_encoder.fit_transform(df[['size']])

print(df)



#    item size  size_encoded
# 0  shirt    M           1.0
# 1  pants    L           2.0
# 2  shirt    S           0.0
# 3    hat    L           2.0

Notice that 'S' is mapped to 0, 'M' to 1, and 'L' to 2, exactly as we intended. If we had let the encoder decide, it would have used alphabetical order ('L', 'M', 'S'), which is incorrect.

3. One-Hot Encoding: For Categories Without Order

When your data is nominal (e.g., city names, car brands), there is no order to preserve. Using ordinal encoding would introduce a false ranking. The solution is One-Hot Encoding.

How it Works and When to Use It

One-Hot Encoding transforms each category into a new column and uses a 1 or 0 to indicate whether the category is present for a given row. This avoids the ordinality problem entirely.

There are two common ways to implement this in Python: pandas.get_dummies and scikit-learn's OneHotEncoder.

While pd.get_dummies is great for quick analysis, sklearn.OneHotEncoder is preferred for building machine learning models because it can be included in a preprocessing pipeline, which helps prevent data leakage (a concept we touched on in the last lesson).

Here is a consolidated code example:

import pandas as pd
from sklearn.preprocessing import OneHotEncoder




# Sample data with a nominal feature
df = pd.DataFrame({
    'product_id': [101, 102, 103, 104],
    'color': ['red', 'blue', 'green', 'blue']
})




# --- Method 1: pandas.get_dummies() ---
df_dummies = pd.get_dummies(df, columns=['color'], prefix='color', dtype=int)
print("--- Using pandas.get_dummies() ---")
print(df_dummies)




# --- Method 2: sklearn.OneHotEncoder ---
# Create an instance of the encoder
ohe = OneHotEncoder(sparse_output=False, handle_unknown='ignore') # sparse_output=False returns a numpy array




# Fit and transform the 'color' column
encoded_features = ohe.fit_transform(df[['color']])




# Create a new DataFrame with the encoded features
encoded_df = pd.DataFrame(encoded_features, columns=ohe.get_feature_names_out(['color']))




# Concatenate with the original DataFrame (and drop the original column)
df_ohe = pd.concat([df.drop('color', axis=1), encoded_df], axis=1)
print("\n--- Using sklearn.OneHotEncoder ---")
print(df_ohe)

Notice I included handle_unknown='ignore'. This is a very useful parameter that tells the encoder to simply create a row of all zeros if it encounters a category in new data that it didn't see during training. This prevents errors when you deploy your model.

4. Trade-offs and Making the Right Choice

You now know how to implement both methods. The most important skill is knowing when to use each one. This involves considering the trade-offs.

The Dummy Variable Trap

One key trade-off with one-hot encoding is the Dummy Variable Trap. When you one-hot encode a feature with k categories, you create k new columns. These columns are perfectly multicollinear, meaning one can be predicted from the others (e.g., if a person is not from 'USA' and not from 'Canada', they must be from 'Mexico' if those are the only three options). This can cause issues for some models, like linear regression.

The common solution is to drop one of the one-hot encoded columns.

You can achieve this in pd.get_dummies with the drop_first=True argument, or in OneHotEncoder with drop='first'.

Summary of Trade-offs

Let's crystallize the decision-making process.

Here is a summary of the key decision points:

Conclusion

Excellent work! You've just mastered a fundamental data preprocessing skill. Being able to correctly convert categorical data into a numerical format is essential for building effective machine learning models.

Key Takeaways:

• The choice between encoding methods depends primarily on whether the data is ordinal (has order) or nominal (no order).
• Use OrdinalEncoder for ordinal data, making sure to specify the correct order to preserve its meaning.
• Use OneHotEncoder for nominal data to avoid creating a false sense of order. This is the safest default for most non-ordinal data.
• Be mindful of the trade-offs:
  • Ordinal Encoding is simple but can mislead a model if used on nominal data.
  • One-Hot Encoding is robust but can increase dimensionality significantly (the "curse of dimensionality") and requires handling the dummy variable trap for certain models.

Preview of the next lesson:

Our dataset is now fully numerical! However, we might have features on very different scales (e.g., an age column from 18-80 and a salary column from 30,000-200,000). Many algorithms perform better when features are on a similar scale. In our next lesson, we will tackle this by learning to apply feature scaling techniques like standardization and normalization.