Welcome back. In our previous lesson, we built a taxonomy of the common types of noise and irrelevance you can expect to find in your scraped data. You learned to distinguish the high-value "signal" from the low-value "noise" by categorizing text data by its source, sentiment, and content, and by identifying common visual issues in images that create spurious correlations.
This lesson is the practical continuation of that work. We will now move from identifying these problems to acting on them. Our goal is to define and apply specific, programmable filtering criteria to automatically remove low-quality and off-topic samples from your dataset. This is a critical step in any real-world machine learning project, as it ensures that your valuable time and computational resources are spent on data that can actually help your model learn.
We'll operate on the core principle of "garbage in, garbage out." A sophisticated model trained on noisy, irrelevant data will produce unreliable results. By cleaning your data first, you create a stronger foundation for everything that follows.
By the end of this lesson, you will have a clear, step-by-step strategy for filtering both your text and image datasets, turning a messy collection of raw data into a refined set ready for annotation and modeling.
A Filtering Funnel for Text Data
Cleaning text data is best approached as a multi-stage process, like a funnel that progressively refines the data. We'll start with broad, structural cleaning to remove obvious noise, and then apply more nuanced, semantic filters to isolate the most relevant content.
The following flowchart provides a great overview of the journey from raw text to processed features, which we will be following conceptually.
Stage 1: Foundational Text Cleaning
Before we can assess the relevance of a piece of text, we need to clean up its structure. Social media text is notoriously messy, filled with characters and formatting that are meaningless for analysis. The following video provides a practical tutorial on these foundational cleaning steps using Python libraries you are familiar with, like Pandas and NLTK.
How to Prepare Text for NLP and Data Analysis (Tutorial)
This tutorial from Gary Eckstein's channel provides a clear, code-oriented walkthrough of the most common text preparation steps. It will give you a concrete sense of how these filtering criteria are implemented.
Watch the following segments to see how to apply each cleaning technique: Convert to lowercase: This is a crucial first step for standardization. It ensures that words like "Meal," "meal," and "MEAL" are all treated as the same token by the model. Remove extra whitespace: This tidies up the text and removes formatting artifacts. Remove stopwords: The video explains removing both a standard list of stopwords (like "the," "a," "is") and, importantly, a custom list. For your project, a custom stopword list is essential. You could add words related to politics, news media, or general policy debates to help filter out second-hand commentary. Remove punctuation: Punctuation marks generally don't add semantic value for classification models and can be removed. Remove short words: A simple but effective heuristic for removing noise. Don't worry about the tokenization and lemmatization steps for now; we'll revisit those when we prepare data for the BERT model. The key takeaway is this sequence of rule-based cleaning operations.
In addition to these steps, you would also apply regular expressions to remove URLs, email addresses, hashtags, and user mentions (@username), which are common artifacts in scraped data.
Stage 2: Semantic Filtering for Relevance and Quality
Once the basic structure is clean, we can apply more intelligent filters to assess the content of the text. In the last lesson, we discussed filtering by source, sentiment, and content type. The following paper formalizes this by proposing four criteria for identifying high-quality text: relevance, informativeness, readability, and objectivity.
[PDF] Enhancing NLP Model Performance Through Data Filtering - EECS
This research paper provides a clear, structured methodology for filtering text data. It moves beyond simple cleaning and introduces metrics to quantify the quality and relevance of text.
Please read the following sections: In the Introduction, read the paragraph that introduces the four criteria: relevance, informativeness, readability, and objectivity. This sets the stage for our filtering strategy. In the Methodology section, read subsections 3.1 through 3.4. For each criterion—Relevance, Informativeness, Readability, and Objectivity—focus on the intuition behind it and the proposed method for calculating a score (e.g., TF-IDF for relevance, Flesch Reading Ease for readability, TextBlob for objectivity). Finally, read the Discussion section. Pay attention to the idea that dynamic thresholds might be better and that you can use a weighted combination of these scores. This highlights the iterative nature of data cleaning.
Let's translate this framework into concrete filtering rules for your mid-day meal project:
-
Informativeness:
- Criterion: The post must contain a minimum amount of information.
- Application: Filter out posts with fewer than, say, 10 words. This removes very low-effort or irrelevant comments like "lol" or "agreed."
-
Relevance:
- Criterion: The post should be a first-hand account, not a general discussion.
- Application: You can create two keyword lists.
- Inclusion List: Words indicating personal experience (e.g., "my son," "our school," "today's meal," "the rice served"). Keep posts that contain these.
- Exclusion List: Words indicating policy or news commentary (e.g., "government," "budget," "policy," "Supreme Court," "news report"). Discard posts that contain these.
- This is a practical heuristic that approximates the paper's more complex TF-IDF relevance score.
-
Objectivity/Sentiment:
- Criterion: The post should express a negative sentiment, as this is the strongest signal for a complaint.
- Application: Use a library like TextBlob (as mentioned in the paper) to calculate a sentiment polarity score for each post. Set a threshold to keep only posts with a negative score (e.g., polarity < -0.1).
By applying these rules sequentially, you create a powerful filtering pipeline that significantly increases the density of high-quality, relevant complaints in your dataset.
Defining Filtering Criteria for Image Data
Filtering images follows a similar logic: we want to remove samples that are technically poor or contextually irrelevant. The goal is to eliminate images from which a model (or even a human) cannot reliably extract information about the meal.
A study that created a large, benchmark dataset of food photos provides an excellent real-world example of how to think about image quality control.
This paper details the creation of a food photo dataset for dietary assessment. Their rigorous process for ensuring data quality provides a fantastic blueprint for your own project.
As you read, focus on the practical steps the researchers took to control image quality: In the Materials and Methods section, read about the instructions given to participants for taking photos. Note the specific guidance on distance and angle and the use of a sizing marker. This shows what an "ideal" data collection process looks like. Next, review the data completeness checks. The study coordinators would ask participants to revise their submissions if there were missing photos or if the automated cropping cut off important parts. This is a form of manual filtering. Finally, in the Results section, look at the description of "problematic food photos." The authors explicitly state that photos with dark backgrounds or non-food items could not be processed. This directly informs the kinds of automated filters we can build.
Inspired by this paper and the concepts from our last lesson, we can define a set of programmable filters for your image data. Since you are studying OpenCV in your other course, you'll recognize that these criteria can be implemented using standard image processing functions.
-
Filter 1: Image Dimensions
- Criterion: The image must be large enough to contain detail.
- Application: Define a minimum resolution (e.g., 256x256 pixels). Any image smaller than this is likely an icon or thumbnail and should be discarded. You can also filter images with extreme aspect ratios (e.g., wider than 3:1 or taller than 1:3), which are often banners or heavily cropped screenshots.
-
Filter 2: Blur
- Criterion: The image must be reasonably in focus.
- Application: Calculate a blurriness score. A common method is to compute the variance of the Laplacian of the image. A low variance indicates a lack of sharp edges, meaning the image is likely blurry. You can set a threshold to discard images below a certain score.
-
Filter 3: Brightness
- Criterion: The image must not be too dark or too bright (washed out).
- Application: Convert the image to grayscale and calculate the mean pixel intensity. This gives a single value for the image's overall brightness. Define a valid range (e.g., between 50 and 200 on a 0-255 scale) and discard images outside this range.
-
Filter 4: Low Information
- Criterion: The image should not be a uniform block of color.
- Application: Calculate the standard deviation of the pixel intensities. An extremely low standard deviation suggests the image is mostly one color (e.g., a photo of a wall, or a user's thumb over the lens). Set a threshold to discard these.
Applying these four filters will automatically remove a significant portion of the "rogues' gallery" of bad images we identified in the previous lesson, leaving you with a much cleaner set of candidate images for labeling.
Conclusion
In this lesson, you have learned how to translate the abstract concepts of noise and irrelevance into concrete, applicable filtering criteria for your text and image data. This is a pragmatic and essential part of the data preparation pipeline that prevents model failures and wasted effort down the line.
Key Takeaways:
- Filtering as a Funnel: Data cleaning is a multi-stage process that starts with broad, structural cleaning and moves to more specific, semantic filtering.
- Text Filtering Criteria: You can filter text programmatically based on foundational properties (e.g., length, case, stopwords) and semantic qualities (e.g., relevance keywords, sentiment scores).
- Image Filtering Criteria: You can use image processing metrics to automatically assess and filter images based on properties like size, blurriness, brightness, and information content.
- Iterative Refinement: The thresholds for these filters are not set in stone. The best practice is to start with a reasonable set of values, inspect the data that is kept and discarded, and iteratively adjust your criteria to achieve the right balance.
You now have a robust strategy for cleaning your raw data. With this cleaner dataset, the next logical question is: how do we get reliable labels for it, especially when manual annotation is slow and expensive? In our next lesson, we will explore the concept of weak supervision, where we'll discuss how to use heuristics and other methods to generate noisy labels at scale, providing a powerful way to bootstrap your model's training process.