Text Data Exploration for Theme and Category Identification

1. Preprocessing and Basic Feature Extraction

Raw text from the internet is messy. It contains punctuation, inconsistent casing, numbers, and other "noise" that can interfere with analysis. Our first step is to clean and standardize this text. Following that, we can extract some simple but powerful numerical features from each complaint.

The video below from the KGP Talkie channel provides an excellent, hands-on demonstration of these initial steps using a dataset of e-commerce reviews, which serves as a great parallel to your complaint data.

Let's apply these ideas. First, you'll need to load your text data. Assuming you have a CSV file of complaints from Reddit located at data/01_raw/text_complaints/reddit_complaints.csv, you can load it like this:

import pandas as pd
import re
from textblob import TextBlob

# Load your data
df = pd.read_csv('data/01_raw/text_complaints/reddit_complaints.csv')

# Let's assume the complaints are in a column named 'body'
# Drop rows where the complaint text is missing
df.dropna(subset=['body'], inplace=True)

# --- Text Cleaning ---
# Convert to lowercase
df['clean_text'] = df['body'].str.lower()
# Remove punctuation
df['clean_text'] = df['clean_text'].apply(lambda x: re.sub(r'[^\w\s]', '', x))

# --- Feature Extraction ---
# 1. Polarity (Sentiment)
df['polarity'] = df['clean_text'].apply(lambda x: TextBlob(x).sentiment.polarity)

# 2. Complaint Length (number of characters)
df['text_len'] = df['clean_text'].str.len()

# 3. Word Count
df['word_count'] = df['clean_text'].apply(lambda x: len(x.split()))

print(df[['body', 'clean_text', 'polarity', 'text_len', 'word_count']].head())

Running this code will give you a new DataFrame with columns representing the sentiment and structure of each complaint. Analyzing the distributions of these features (e.g., by plotting histograms as shown in the video from 31:04 - 35:41) can already give you insights. Are most complaints negative? Are they typically long or short?

2. Identifying Themes with N-Gram Analysis

Now that we have cleaned text, we can dig deeper to find common themes. A powerful and straightforward way to do this is through n-gram analysis.

• Unigrams are single words (e.g., "food", "stale", "insect").
• Bigrams are pairs of words (e.g., "stale food", "dirty water").
• Trigrams are triplets of words (e.g., "insects in meal").

By counting the most frequent n-grams, we can quickly identify the most talked-about issues. A crucial step here is to remove stop words—common words like "the," "is," "and," "in"—which add little semantic value. Removing them helps us focus on the words that truly define the topic.

The KGP Talkie video again provides an excellent guide on how to implement this using scikit-learn's CountVectorizer.

Here is a function that encapsulates this logic to find the top 20 bigrams in your complaint data.

from sklearn.feature_extraction.text import CountVectorizer
import numpy as np

def get_top_n_bigrams(corpus, n=20):
    """
    Finds the top N most frequent bigrams in a corpus of text.
    """
    # Use CountVectorizer to count word frequencies, specifying bigrams and removing English stop words
    vec = CountVectorizer(ngram_range=(2, 2), stop_words='english').fit(corpus)
    bag_of_words = vec.transform(corpus)
    sum_words = bag_of_words.sum(axis=0) 
    words_freq = [(word, sum_words[0, idx]) for word, idx in vec.vocabulary_.items()]
    words_freq = sorted(words_freq, key = lambda x: x[1], reverse=True)
    return words_freq[:n]

# Get the top 20 bigrams from our cleaned text
top_bigrams = get_top_n_bigrams(df['clean_text'])

# Create a DataFrame for easy viewing
bigram_df = pd.DataFrame(top_bigrams, columns=['Bigram', 'Frequency'])
print(bigram_df)

# You can plot this for a nice visualization
bigram_df.set_index('Bigram').plot(kind='barh', figsize=(10, 8), legend=False).invert_yaxis()

When you run this, you will likely see phrases directly related to meal quality issues, such as "food quality," "dal rice," "not cooked," or "small quantity." These are your first data-driven clues for defining complaint categories.

3. Discovering Latent Topics with LDA

While n-grams find explicit, frequently co-occurring words, Topic Modeling is an unsupervised machine learning technique that goes a step further by discovering latent (hidden) themes. Latent Dirichlet Allocation (LDA) is a popular algorithm for this.

LDA works by assuming that each document is a mixture of topics, and each topic is a mixture of words. By analyzing the word co-occurrence patterns across all complaints, it can reverse-engineer what these topics might be. For example, it might learn that words like "insect," "fly," "dirty," and "unhygienic" tend to appear together, and group them into a "Hygiene Issues" topic.

This video from David Langer provides a clear, high-level explanation and implementation of LDA.

Let's apply a simple LDA model to your data. We'll use CountVectorizer to prepare the data and LatentDirichletAllocation to find 5 topics.

from sklearn.decomposition import LatentDirichletAllocation

def display_topics(model, feature_names, no_top_words):
    for topic_idx, topic in enumerate(model.components_):
        print(f"Topic {topic_idx}:")
        print(" ".join([feature_names[i]
                        for i in topic.argsort()[:-no_top_words - 1:-1]]))

# Vectorize the text data, removing stop words
# We limit to the top 1000 most frequent words to keep the model focused
vectorizer = CountVectorizer(max_df=0.95, min_df=2, max_features=1000, stop_words='english')
tf = vectorizer.fit_transform(df['clean_text'])
tf_feature_names = vectorizer.get_feature_names_out()

# Run LDA
num_topics = 5
lda = LatentDirichletAllocation(n_components=num_topics, max_iter=5, learning_method='online', random_state=42).fit(tf)

# Display the topics
no_top_words = 10
display_topics(lda, tf_feature_names, no_top_words)

The output will look something like this (your results will vary):

Topic 0:
school meal food children quality day state government district poor
Topic 1:
rice dal curry vegetable watery thin portion small quantity plate
Topic 2:
egg banana fruit milk protein served provide menu child
Topic 3:
complaint action taken filed rti no response official ignored
Topic 4:
insect worm fly found stale rotten smell bad spoiled

From this, you can infer the topics:

• Topic 0: General discussion about the meal scheme.
• Topic 1: Issues with Quantity and Quality of staple foods.
• Topic 2: Issues related to Nutritional Items (or lack thereof).
• Topic 3: Issues related to the Complaint Process itself.
• Topic 4: Critical Hygiene and Safety Violations.

These discovered topics provide an excellent, data-driven starting point for defining the labels for your supervised complaint classification model.

Conclusion

In this lesson, you've moved from having a collection of raw text files to having a deep, analytical understanding of their content. You've learned to clean text data, extract quantitative features, identify the most common complaint phrases using n-gram analysis, and discover abstract complaint categories with topic modeling.

Key Takeaways:

• Text preprocessing (lowercasing, removing punctuation, removing stop words) is a mandatory first step for meaningful analysis.
• N-gram analysis is excellent for finding frequent, explicit keywords and phrases that point to common issues.
• Topic modeling with LDA is a powerful unsupervised technique to discover latent themes and help structure your classification problem.
• The themes and categories you've identified today will directly inform the annotation schema and target labels for the BERT model you will build later in the course.

In our next lesson, we will shift our focus from text to images. You will learn how to perform an initial EDA on your image data, including analyzing metadata and visually inspecting samples to inform your preprocessing and annotation strategy for the meal quality CNN.