Encoding Categorical and Temporal Data

A Roadmap for Encoding Metadata

Before we can feed metadata into a neural network, we must convert it into numbers. However, the way we convert it matters immensely. Different types of data require different encoding strategies to preserve their meaning.

The diagram below provides an excellent overview of various categorical data types and their recommended encoding methods. We will be focusing on Nominal and Cyclic data today.

Part 1: Encoding Nominal Categorical Data

Your project metadata includes features like school type (e.g., 'government', 'aided') or location (e.g., 'urban', 'rural'). These are examples of nominal categorical data: they are distinct labels with no inherent order. 'Government' is not "greater" or "less than" 'aided'; they are simply different categories.

A naive approach would be to assign integers: government=0, aided=1. However, this introduces an artificial order that can mislead a model into thinking there is a mathematical relationship between the categories. The standard and most effective solution is One-Hot Encoding. This technique converts a single column of categories into multiple new binary (0 or 1) columns, one for each unique category.

For example, a school_type column would become:

This representation is unambiguous and does not imply any ordinal relationship. The following video provides a clear explanation of different data types and demonstrates how to perform one-hot encoding.

For your project, using pandas.get_dummies is the most straightforward way to handle nominal features. Given a DataFrame df with a column 'school_type', the implementation is as simple as:

import pandas as pd




# Sample data mimicking your project's metadata
data = {'school_type': ['government', 'aided', 'government', 'private'],
        'region': ['rural', 'urban', 'urban', 'rural']}
df = pd.DataFrame(data)




# Apply one-hot encoding to specified columns
df_encoded = pd.get_dummies(df, columns=['school_type', 'region'], prefix=['school', 'reg'])

print(df_encoded.head())



# Output:
#    school_government  school_private  school_aided  reg_rural  reg_urban
# 0                  1               0             0          1          0
# 1                  0               0             1          0          1
# 2                  1               0             0          0          1
# 3                  0               1             0          1          0

Part 2: Encoding Temporal (Cyclic) Data

Your project also involves temporal features, like the season or month a complaint was filed. These features are cyclic. For example, December is followed by January, and Sunday is followed by Monday. If we encode months linearly (Jan=1, Feb=2, ..., Dec=12), our model will think that December (12) and January (1) are very far apart, when in reality they are adjacent.

To solve this, we use Cyclical Encoding. The core idea is to map the cyclical feature onto a circle using sine and cosine transformations. This way, the end of the cycle connects smoothly back to the beginning.

The article "Cyclic Encoding in Machine Learning" explains this concept brilliantly.

As the article explains, for a variable v with a maximum value max_v (e.g., 12 for months), we create two new features:
$v_{\text{sin}}=\sin \left(2\pi \frac{v}{\text{max\_v}}\right)$
$v_{\text{cos}}=\cos \left(2\pi \frac{v}{\text{max\_v}}\right)$

This transformation maps each point in the cycle to a unique (x, y) coordinate on a unit circle, perfectly preserving the cyclical nature of the data. The image below shows what this looks like for 'month' and 'day' features. Notice how the values form smooth, continuous waves.

Here's how you can implement this for a 'month' feature in Python. First, you'll need to extract the month from a date string. Python's built-in datetime library is perfect for this.

Once you have the numerical month, you can apply the cyclical encoding:

import numpy as np
import pandas as pd
from datetime import datetime




# Sample data with timestamps
data = {'timestamp': ['2023-01-15', '2023-07-20', '2023-12-25']}
df = pd.DataFrame(data)




# 1. Convert string to datetime objects
df['datetime'] = pd.to_datetime(df['timestamp'])




# 2. Extract the month (1-12)
df['month'] = df['datetime'].dt.month




# 3. Apply cyclical encoding
max_val = 12  # Maximum value for month is 12
df['month_sin'] = np.sin(2 * np.pi * df['month'] / max_val)
df['month_cos'] = np.cos(2 * np.pi * df['month'] / max_val)




# Note: We drop the original 'month' column as it's now redundant
print(df[['timestamp', 'month_sin', 'month_cos']].head())



# Output:
#     timestamp  month_sin  month_cos
# 0  2023-01-15   0.500000   0.866025  (January)
# 1  2023-07-20  -0.500000  -0.866025  (July)
# 2  2023-12-25   0.000000   1.000000  (December, note sin is ~0, cos is 1)

A Practical Function for Your Pipeline

To make this easy to integrate into your project, you can create a single function that processes all your metadata at once. This function can take a Pandas DataFrame, identify which columns are categorical and which are temporal, and apply the correct encoding.

import pandas as pd
import numpy as np

def encode_metadata(df: pd.DataFrame, categorical_cols: list, temporal_cols: dict) -> pd.DataFrame:
    """
    Encodes categorical and temporal metadata columns in a DataFrame.

    Args:
        df (pd.DataFrame): The input DataFrame.
        categorical_cols (list): A list of column names to be one-hot encoded.
        temporal_cols (dict): A dictionary where keys are temporal columns (e.g., 'month')
                              and values are their max period (e.g., 12).

    Returns:
        pd.DataFrame: A new DataFrame with encoded features.
    """



    # Create a copy to avoid modifying the original DataFrame
    df_encoded = df.copy()




    # --- Temporal Encoding ---
    # First, ensure temporal columns are derived if needed (e.g., from a 'timestamp' column)
    if 'timestamp' in df_encoded.columns:
        dt_series = pd.to_datetime(df_encoded['timestamp'])
        if 'month' in temporal_cols:
            df_encoded['month'] = dt_series.dt.month
        if 'day_of_week' in temporal_cols:
            df_encoded['day_of_week'] = dt_series.dt.dayofweek # Monday=0, Sunday=6
    
    for col, max_val in temporal_cols.items():
        df_encoded[f'{col}_sin'] = np.sin(2 * np.pi * df_encoded[col] / max_val)
        df_encoded[f'{col}_cos'] = np.cos(2 * np.pi * df_encoded[col] / max_val)
    



    # --- Categorical Encoding ---
    df_encoded = pd.get_dummies(df_encoded, columns=categorical_cols, prefix_sep='_')




    # --- Clean up ---
    # Drop original columns that have been encoded
    cols_to_drop = list(temporal_cols.keys())
    if 'timestamp' in df_encoded.columns:
        cols_to_drop.append('timestamp')
        
    df_encoded = df_encoded.drop(columns=cols_to_drop, errors='ignore')

    return df_encoded




# --- Example Usage ---
if __name__ == "__main__":



    # Your real data would come from your scraped datasets
    sample_data = {
        'timestamp': ['2023-01-15', '2023-05-10', '2023-12-30'],
        'school_type': ['government', 'aided', 'government'],
        'region': ['rural', 'urban', 'rural']
    }
    metadata_df = pd.DataFrame(sample_data)

    final_features = encode_metadata(
        df=metadata_df,
        categorical_cols=['school_type', 'region'],
        temporal_cols={'month': 12}
    )
    
    print("--- Final Encoded Metadata Features ---")
    print(final_features.head())

Conclusion

You have now learned how to transform categorical and temporal metadata into numerical features suitable for a machine learning model. This is a vital skill in any data science project, as real-world data is rarely in a ready-to-use format.

Here are the key takeaways from this lesson:

• Nominal categorical data (like school type) should be encoded using one-hot encoding to avoid creating a false sense of order. pandas.get_dummies is an excellent tool for this.
• Cyclic temporal data (like month or day of week) requires cyclical encoding with sine and cosine transformations to preserve the "wrap-around" nature of time.
• These encoded features are now ready to be concatenated with the image and text vectors you prepared earlier.

In our next lesson, we will finally bring all these different feature vectors together. You will learn about multimodal fusion architectures, specifically a "late fusion" approach, and build a model that combines image, text, and metadata insights to make a single, powerful prediction.