Welcome to the final lesson of our first module. Over the past four lessons, we have established a strong foundation by dissecting the PM POSHAN scheme, identifying common violations, and understanding the critical gaps in current monitoring. In our last session, you mapped out how different data types—images, text, and metadata—act as distinct but complementary sources of evidence, or "signals," for detecting these violations.
Today, we will build directly on that map to achieve your final learning outcome for this module: to outline a data collection plan, specifying target sources, acquisition methods, and ethical considerations. This is the crucial bridge between understanding a problem and gathering the raw materials to solve it. We'll move from the abstract "we need text data" to the concrete "we will scrape specific news sites and use the Reddit API for these subreddits." This plan will serve as your blueprint for the data acquisition phase of your project.
From Data Needs to a Data Plan
A well-structured data collection plan is essential for any serious data science project. It forces you to think critically about feasibility, resources, and ethics before you write a single line of code. The Data Science for Social Good initiative at Carnegie Mellon University provides an excellent framework for this process, which we will adapt for your project.
Data Science Project Scoping Guide
This guide provides a professional framework for planning data science projects. We will focus on its guidance for data strategy and ethical foresight. It will help structure your thinking about not just what data you need, but how to responsibly acquire and manage it.
Please read three key parts of this guide: First, read the overview of the scoping process in the introduction, specifically the list from Step 0 to Additional Considerations. This reinforces the structured thinking we've been applying. Next, read Step 3: What data do you have and what data do you need? in its entirety. As you read, pay close attention to the checklist of data source characteristics, such as granularity, frequency, history, and others. Also, note the advice on matching data to actions and augmenting with external public data. Finally, carefully review the section on Ethical Issues. Focus on the motivating questions under Privacy, Transparency, Discrimination, and Social License. These questions are not just a checklist; they are prompts for deep reflection that must be embedded in your project from day one.
The principles in this guide transform data collection from a simple technical task into a strategic activity. Your plan must consider the granularity, frequency, and history of the data needed and, most importantly, be governed by a strong ethical framework.
Identifying Sources and Acquisition Methods
Now, let's translate the theoretical framework into a concrete plan for your project. We'll specify the "where" (target sources) and the "how" (acquisition methods) for each data modality. Given your background in web development and your current self-study in Python libraries, you are well-equipped to understand and later implement these methods.
1. Textual Data (Complaints, Reports, News)
-
Target Sources:
- Social Media: Platforms where public discussion occurs. Key targets would be Twitter/X (searching for hashtags like #MidDayMeal) and Reddit (specifically communities like r/india, r/mumbai, etc.).
- News Media: Online portals of national and local newspapers (e.g., The Hindu, Times of India, Dainik Bhaskar). Local news is often a rich source of specific incidents.
- Activist & NGO Reports: Publications from organizations working on food security and child rights. These are often in PDF format on their websites.
- Official Sources: Government websites sometimes publish reports or press releases, which can provide context or official data.
-
Acquisition Methods:
- APIs: For sources like Reddit and Twitter, using their official APIs (via libraries like PRAW for Reddit) is the most reliable and respectful method. It allows you to query for data in a structured way.
- Web Scraping: For news sites and blogs that don't have APIs, you'll need to write scripts. Your self-study on
requestsandBeautifulSoupwill be directly applicable here. This involves fetching the HTML of a webpage and parsing it to extract the article text, author, and publication date. - Manual Download: For PDF reports, the most practical approach is often manual download and text extraction.
2. Image Data (Meal Photos)
- Target Sources: Images are almost always embedded within the text sources mentioned above. A news article about a contamination incident will likely have a photo. A tweet complaining about a small portion might include an image.
- Acquisition Methods: Your web scraping scripts must be designed to not only extract text but also identify and download the associated images (
<img>tags). A key challenge will be ensuring you correctly link each image to its original context (the article or post it came from), as the surrounding text is crucial for interpretation.
3. Metadata (Contextual Information)
- Source & Acquisition: Metadata is rarely a distinct data source to be collected on its own. Instead, it is generated during the collection of text and images.
- Technical Metadata: The URL, date of scraping, and source (e.g., 'The Hindu', 'r/india') are captured by your collection script.
- Explicit Metadata: The publication date, author, and location mentioned in an article must be parsed from the text.
- Implicit Metadata: Information like the school's location (rural/urban) or management type (government/NGO) might not be in the source data. You may need to augment your dataset later by cross-referencing it with external public data, such as school directories or census information—a practice highlighted in the scoping guide you just read.
The Indispensable Ethical and Legal Layer
Acquiring data, especially from public sources and about sensitive topics, is not a purely technical exercise. It is governed by law, ethics, and social norms. For a project with social impact, getting this right is non-negotiable.
A paper on the PM POSHAN scheme provides critical, India-specific legal context for your project.
[PDF] PM POSHAN: ENSURING SOCIAL & ECONOMIC JUSTICE FOR ...
This article discusses the use of technology in the PM POSHAN scheme and directly addresses the legal framework for data handling in India.
Please read the section that begins with a discussion of leveraging technology and the POSHAN Tracker App. Focus on the paragraph that explicitly mentions data privacy. Pay attention to the reference to the DPDP Act and the reasoning for why handling data within government school systems can help ensure accountability.
This reference to the Digital Personal Data Protection (DPDP) Act 2023 is vital. It means your data collection and storage strategy must comply with Indian law, particularly concerning personal and sensitive data.
The legal landscape around web scraping, a core acquisition method for you, can be murky. The following video provides a concise and developer-focused overview of the key legal precedents in the U.S., which have influenced digital privacy discussions globally.
Am I going to jail for web scraping?
This video from the Fireship channel, "Am I going to jail for web scraping?", discusses the legality and ethics of web scraping from a programmer's perspective. It will give you a practical understanding of the risks involved.
Watch the entire video, but pay special attention to two parts: The discussion of key court cases like Craigslist vs. Three Taps and LinkedIn vs. hiQ, which established important precedents. The final summary of the risks, which distinguishes between criminal charges and civil lawsuits, around the concluding thoughts.
The key takeaways from these legal and ethical considerations are:
- Public vs. Private: Focus on publicly available data. Do not attempt to bypass login walls or access private accounts.
- Terms of Service: Always review a website's
robots.txtfile and its Terms of Service. While not always legally binding in the same way as a statute, violating them is what can lead to getting your IP blocked or receiving a cease-and-desist letter. - Respect "No": The legal precedent is clear: if a company sends you a formal request to stop scraping (a cease-and-desist letter), you must stop. Continuing after being explicitly told "no" is what crosses the line into potential CFAA violations.
- PII (Personally Identifiable Information): Your project deals with children. You must be extremely careful. Even if you scrape an image of a child from a public news article, that image is sensitive. Your plan must include a strategy for anonymization or de-identification wherever possible. Do not store names of individuals, especially children, unless absolutely necessary and legally permissible.
- Social License: Think about the headline. "Data Scientist Uses AI to Improve School Meals" is a positive story. "Researcher Scrapes Photos of Children for Secret Database" is not. Your methods must be transparent and defensible.
Your Task: Create the Data Collection Plan
Now it's time to consolidate this into a formal plan. Create a document with a table for each data modality (Text, Image). Use the following structure. This will be the key deliverable for this module.
| Field | Description & Example |
|---|---|
| Information to Capture | What key information are you seeking? (e.g., For Text: Type of complaint, severity indicators, location.) |
| Target Sources | List specific URLs or platforms. (e.g., r/india, https://timesofindia.indiatimes.com/, https://twitter.com/search?q=%23MidDayMeal) |
| Acquisition Method | The technical method to be used. (e.g., PRAW (Python Reddit API Wrapper), BeautifulSoup/Requests script.) |
| Data Fields to Extract | The specific data points you will save. (e.g., post_title, body_text, post_date, url, image_url) |
| Frequency of Collection | How often will you run your collection script? (e.g., Daily, Weekly.) |
| Ethical & Legal Checklist | - Public data only? (Y/N) - Terms of Service checked? (Y/N) - PII present? (Y/N) - If PII, anonymization plan: [Describe plan] - Compliant with DPDP Act? (Y/N) |
Completing this table is not just an exercise; it creates an actionable blueprint that demonstrates you have thought through the practical and ethical dimensions of data collection.
Conclusion
In this lesson, you've learned how to construct a comprehensive data collection plan. We've moved beyond the "what" to define the "where" and "how" of data acquisition for your project.
Key Takeaways:
- A data collection plan requires defining target sources, acquisition methods, and key data fields for each modality.
- Acquisition methods range from using structured APIs, to less-structured web scraping, to manual downloads.
- Metadata is often derived during the collection process and can be augmented with external public datasets.
- Ethical and legal considerations are paramount. You must be aware of local laws like India's DPDP Act and the general legal principles surrounding web scraping and data privacy.
- Your plan must be governed by the principle of "social license," ensuring your methods are transparent, defensible, and serve the project's positive goals.
This lesson concludes our first module. You have successfully journeyed from a broad project idea to a deep understanding of the problem domain and a concrete plan for data acquisition.
In our next lesson, we will begin Module 2: Translating Problems into ML Tasks. Your first task will be to take the violations we've discussed and the data plan you've just created, and from them, formulate a precise, measurable definition of a "violation" that your machine learning models can actually learn to predict.