Skip to main content
Back to path

Welcome to a Grasp lesson

Create a free account to keep studying and create your own study path.

Lesson 13

Crafting Clear Annotation Guidelines

Recommended resources

Welcome to the next lesson in our journey to build a robust meal quality monitoring system. In our last two sessions, we designed the core data blueprints for your project: a structured schema for annotating meal images and a corresponding schema for classifying text complaints. You now have a clear plan for what information to extract from each data source.

However, a schema is just a skeleton. To bring it to life, we need to ensure that everyone who uses it—whether it's you, a colleague, or a hired annotator—interprets each label in exactly the same way. This is where annotation guidelines come in. They are the detailed, unambiguous instructions that transform subjective judgment into consistent, high-quality data. In machine learning, this high-quality, consistently labeled data is known as "ground truth," and it is the absolute foundation upon which your models will be built.

In this lesson, you will learn how to write clear, comprehensive labeling guidelines for both the image and text schemas we've developed. We will ground our definitions in the official PM POSHAN rules and explore best practices for eliminating ambiguity, ensuring that the data you create is a reliable reflection of reality.

The Purpose of Ground Truth and Guidelines

Before we dive into writing, let's solidify our understanding of why this step is so critical. An AI model is only as good as the data it's trained on. If your labels are inconsistent or incorrect, your model will learn the wrong patterns, leading to unreliable predictions—a concept often summarized as "garbage in, garbage out."

Ground Truth: The Foundation of Accurate AI & Machine Learning Models

This video from IBM Technology provides a concise and clear explanation of "ground truth" and the common challenges in creating it.

Please watch the following segments: Start with the initial definition of ground truth and its purpose. Understand its role in supervised learning from this section. Notice the emphasis on how incorrect labels lead to false predictions. Pay close attention to the common challenges, especially ambiguity and complexity, as these are the primary problems that good guidelines solve. Finally, see how a good labeling strategy with standardized guidelines is the key solution.

As the video highlights, creating high-quality ground truth requires a deliberate strategy. A well-written guideline document is the most important part of that strategy.

The Anatomy of an Effective Guideline Document

What does a good set of guidelines look like? It's more than just a list of definitions. It's a comprehensive document designed to onboard a non-expert, align their judgment with the project's goals, and equip them to handle tricky cases.

How to Write Data Labeling/Annotation Guidelines - Eugene Yan

This article by Eugene Yan breaks down the structure of professional-grade annotation guidelines, using examples from Google and Bing. It provides an excellent template for us to follow.

Please read through the article, focusing on the key sections that answer these questions: Why is the task important? Read this section to understand the importance of motivating your annotators. What is the task? Review the introduction to this section. What do the terms mean? See why defining terms is crucial for consistency. How should annotators decide? This is the most critical part. Read the section that starts with the "meat of the guidelines". Note the emphasis on using examples and decision trees. How should the task be performed? Review the advice on logistics and what to do when an annotator is unsure.

Based on these principles, we will now draft the guidelines for your project. We'll structure them as a formal document that you can adapt and expand upon.

Annotation Guidelines v1.0: Mid-Day Meal Quality

1. Introduction: Why This Task is Important

Goal: To improve the health and nutrition of millions of children in India's Mid-Day Meal Scheme (PM POSHAN).

Your work as an annotator is the first and most critical step in building an AI system that can automatically detect potential problems with meals served in schools. By carefully labeling images and text complaints, you are creating the "ground truth" data that will teach our models to spot issues like insufficient food, missing nutritional components, and unhygienic conditions. The quality of your work directly impacts the model's accuracy and, ultimately, its ability to help ensure children receive safe, nutritious meals.

2. The Tasks

You will be presented with two types of data:

  • Image Annotation: You will see a photo of a meal served to a child. You will answer a series of questions about the meal's quantity, variety, and quality based only on what is visible in the image.
  • Text Annotation: You will read a complaint or comment related to the meal scheme. You will identify the specific type of violation being reported and assess its severity.

3. Part 1: Image Annotation Guidelines

For each image, you will answer the questions laid out in our schema. Here are the detailed definitions and decision rules. Our definitions are based on the official PM POSHAN scheme guidelines.

[PDF] guidelines of the PM Poshan scheme - Ministry of Education

To ground our labels in official policy, we'll refer to the government's own standards. This ensures our model is aligned with the real-world rules it's meant to monitor.

Please review the food and nutrition norms specified in the guidelines. In the PDF, find section "1.7 Food and Nutrition norms". Pay close attention to the table that details the required grams of food grains, pulses, and vegetables for primary and upper primary students. Note both the nutritional norms (calories, protein) and the corresponding food norms (in grams). We will translate these into visual cues for our annotators.

3.1 Quantity

This assesses if the amount of food appears sufficient. Since we cannot weigh the food, we use visual heuristics based on a standard-sized plate.

  • Label: quantity_sufficient

    • Definition: The meal appears to provide a reasonable amount of food for a child's lunch.
    • Rule of Thumb: The main compartment(s) of the plate are at least half to two-thirds full. The portion size looks substantial.
    • Example: A plate with a large serving of rice, a full bowl of dal, and a side of vegetables.

  • Label: quantity_insufficient

    • Definition: The meal appears visibly sparse or inadequate.
    • Rule of Thumb: The main compartment(s) of the plate are less than half full. You can see large empty areas on the plate that should contain food.
    • Example: A plate with only a small scoop of rice and a splash of watery dal.

3.2 Variety

This checks if the core nutritional components mandated by the scheme are present.

  • Label: protein_present / protein_absent

    • Definition: The PM POSHAN guidelines require a protein source, typically pulses (dal, legumes, etc.), with every meal.
    • Rule of Thumb: Label as protein_present if you can clearly identify a serving of dal, chickpeas, kidney beans, or another legume. If the meal is only rice/roti and a watery vegetable curry with no visible pulses, label it as protein_absent.
    • When in doubt: If you see a curry but cannot tell if it contains pulses, label it protein_absent and add a note. We require visible evidence.

  • Label: vegetable_present / vegetable_absent

    • Definition: The guidelines require a serving of vegetables.
    • Rule of Thumb: Label as vegetable_present if you can identify cooked vegetables (e.g., in a sabzi, mixed into rice) or a fresh salad. If the meal consists only of rice and plain dal, label it vegetable_absent.

3.3 Quality

This assesses the visible quality and hygiene of the meal.

This image shows how different annotation tasks capture increasing levels of detail. Our task uses classification-style labels (e.g., 'contamination present/absent') to identify key meal attributes efficiently.

  • Label: visible_contamination

    • Definition: The food contains foreign objects that should not be there.
    • Rule of Thumb: Apply this label if you see insects, hair, dirt, stones, plastic, or any other non-food item in the meal or on the plate.
    • Do not use for: Food that is merely poorly prepared (e.g., burnt). This has its own label.

  • Label: poor_preparation

    • Definition: The food appears unappetizing due to issues in the cooking process.
    • Rule of Thumb: Apply this label if the food is visibly burnt, undercooked, excessively oily, or has an unappealing texture (e.g., dal that looks like water, vegetables cooked into a homogenous mush).
    • Example: Blackened roti, rice grains that are hard and separate, a curry with a thick layer of oil on top.

4. Part 2: Text Annotation Guidelines

For each text complaint, you will identify all violations mentioned and assign a severity level to each. A single complaint can have multiple violations.

4.1 Violation Types

These are the "aspects" or topics of the complaint.

  • quantity_insufficient: Any mention of not enough food, small portions, or children remaining hungry.
  • quality_poor_preparation: Complaints about food being burnt, undercooked, watery, oily, etc.
  • quality_bad_taste_smell: Complaints specifically about bad taste or foul odor.
  • variety_missing_component: Explicitly states a required food item was not served (e.g., "no dal was given today," "they only served rice").
  • hygiene_contamination: Reports of foreign objects (insects, hair, dirt) or generally unhygienic conditions.
  • health_safety_sickness: Any mention of illness, food poisoning, vomiting, or other health issues linked to the meal. This is a high-priority label.
  • staff_misconduct: Complaints about the behavior of school staff, cooks, or servers (e.g., rudeness, discrimination, not serving properly).
  • infrastructure_issue: Complaints about the physical environment (e.g., "the dining area is filthy," "no clean water for washing hands").

4.2 Severity Levels

This captures the intensity of the complaint. We use a four-level scale. The key is to look at the language used, not just the topic.

  • Level 1: no_explicit_reproach

    • Definition: A problem is stated or hinted at, but no one is directly blamed. The tone is often neutral or questioning.
    • Keywords: "It seems...", "Was the rice supposed to be...?", "My son said the food was..."
    • Example: "The dal was very thin today."

  • Level 2: disapproval

    • Definition: The writer expresses personal unhappiness, dissatisfaction, or annoyance. The focus is on their negative feeling.
    • Keywords: "Unhappy," "disappointed," "sad to see," "not good."
    • Example: "I am very disappointed with the quality of the meal served."

  • Level 3: accusation

    • Definition: The writer explicitly states that someone or an entity is at fault for a reprehensible action. It's a direct statement of wrongdoing.
    • Keywords: "You did...", "The school served...", "They are not following the rules."
    • Example: "The school served undercooked rice to the children."

  • Level 4: blame

    • Definition: The most severe level. The writer not only accuses but also holds the party responsible for negative consequences. It connects the fault to a harmful outcome.
    • Keywords: "Because of your...", "Your negligence caused...", "Thanks to you, my child is..."
    • Example: "Because you served contaminated food, my child is now in the hospital."

Decision Rule for Severity: When a complaint contains multiple sentences with different tones, assign the severity based on the single most severe statement in the text.

5. General Instructions

  • Use your best judgment: The guidelines cover most cases, but you may encounter new or ambiguous situations. Use your judgment and leave a comment in the annotator_notes field explaining your reasoning.
  • When in doubt, ask: If you are consistently unsure about a specific type of case, please flag it for review. Your questions help us improve these guidelines.
  • Do not make assumptions: Base your labels only on the evidence present in the image or text. Do not infer information that is not there.

Conclusion

You have now created the first draft of a comprehensive annotation guideline document. This document is the cornerstone of your data quality strategy. By clearly defining each label and providing rules for decision-making—grounded in official government standards—you have laid the groundwork for creating a consistent, reliable, and powerful dataset.

Key Takeaways:

  • Guidelines are Non-Negotiable: A schema without guidelines is open to interpretation, leading to noisy labels and a poorly performing model.
  • Structure is Key: A good guideline document motivates the annotator, defines the task and terms, and provides clear decision-making logic with examples.
  • Ground Truth in Reality: Anchoring your label definitions in official documents (like the PM POSHAN guidelines) ensures your model is optimized for the real-world problem you're trying to solve.
  • Anticipate Ambiguity: The best guidelines proactively address edge cases and provide clear instructions for what to do when a situation is unclear.

This document is a living artifact. It is not meant to be perfect on the first try. In our next lesson, we will discuss how to put it to the test. We'll cover the concept of inter-annotator reliability and plan a small pilot annotation exercise to identify weaknesses in these guidelines and refine them based on real-world data.