Designing Annotation Schemas for Complaint Text

Deconstructing a Complaint: Aspect and Severity

Just as a meal image can have multiple attributes, a single text complaint can contain several distinct issues. A parent might write, "The dal was watery and my child got sick," pointing to two different problems: poor food quality and a potential health safety issue. To capture this richness, we need to break down complaints along two key dimensions:

1. Violation Type (Aspect): What is the complaint about? This is the core topic or subject of the grievance.
2. Severity Level: How serious is the complaint? This captures the intensity and emotional weight of the language used.

This approach, known as aspect-based analysis, is a common technique in Natural Language Processing (NLP) for gaining fine-grained insights from user feedback.

Defining Violation Types (Aspects)

For your project, "aspects" are the specific meal quality violations. We can start by revisiting the categories we identified in Module 1 and the PM POSHAN guidelines. A good set of initial violation types could include:

• quantity_insufficient
• quality_poor_preparation (e.g., watery, burnt, undercooked)
• quality_bad_taste_smell
• variety_missing_component (e.g., "no pulses were served")
• hygiene_contamination (e.g., insects, hair, dirt)
• health_safety_sickness (e.g., reports of food poisoning)
• staff_misconduct (e.g., rudeness, discrimination)
• infrastructure_issue (e.g., unclean dining area)

This level of granularity is crucial. As we discussed in the last lesson, it's always better to start with specific labels. You can combine quality_poor_preparation and quality_bad_taste_smell into a general quality_issue label later, but you cannot split a general label into specifics without re-annotating all your data.

Defining Severity Levels

Not all complaints carry the same weight. A comment like "The rice could be better" is far less urgent than "My child found glass in their food." Instead of using a simple low/medium/high scale, which can be very subjective, we can ground our severity levels in linguistic theory.

A standard framework in pragmatics, proposed by Anna Trosborg, categorizes complaints into four levels based on the directness and face-threat of the language. This gives us a robust, defensible set of labels.

Let's adapt Trosborg's four categories for your Mid-Day Meal project:

1. No Explicit Reproach: The user expresses a problem without directly blaming anyone. This is often a hint or a mild statement of a negative situation.

   • Example: "Is dal supposed to be this watery?"
   • Example: "My son didn't get his meal today."

2. Disapproval: The user expresses clear dissatisfaction, annoyance, or negative feelings about a situation. The responsibility is often implied but not directly stated as an accusation.

   • Example: "Another disappointing meal. The vegetables were completely mushy."
   • Example: "I'm really unhappy with the quality of food being served."

3. Accusation: The user explicitly asserts that someone or something is responsible for a reprehensible action. This moves from expressing personal feeling (disapproval) to stating a fault.

   • Example: "You served undercooked rice to the children."
   • Example: "The school staff is not following the menu."

4. Blame: This is the most severe level. The user not only accuses but also presupposes the complainee's guilt and holds them responsible for the negative consequences.

   • Example: "Thanks to your expired food, my daughter is now sick."
   • Example: "The school's negligence is putting children's health at risk."

Using these four distinct levels will make your annotations more consistent and your final model's output more interpretable than a simple 1-5 scale.

Drafting the Text Annotation Schema v1.0

Now, we can combine these two dimensions—violation type and severity—into a structured JSON schema. Since a single complaint can touch on multiple issues, our schema needs to support a list of violations, where each violation is an object containing a type and a severity.

This multi-label, multi-faceted approach is common in real-world complaint analysis.

Inspired by this, here is a proposed first version of your text annotation schema:

{
  "text_id": "complaint_456.txt",
  "source": "twitter", // Where the data came from (e.g., twitter, reddit, form)
  "is_usable": true, // Is the text on-topic and understandable?
  "complaint_text": "The dal was just watery today and my son said the teacher was rude when he asked for more. This is unacceptable.",
  
  "violations": [
    {
      "violation_type": "quality_poor_preparation",
      "severity": "disapproval"
    },
    {
      "violation_type": "staff_misconduct",
      "severity": "accusation"
    }
  ],

  "annotator_notes": "Complaint mentions two distinct issues. The 'unacceptable' seems to apply to both."
}

This schema is powerful because it is:

• Multi-Label: The violations array can hold multiple objects, correctly capturing complex complaints.
• Structured: Each violation is neatly paired with its own severity level.
• Grounded: The severity field uses the rigorous four-level scale we defined.
• Pragmatic: It includes fields for metadata (source), data cleaning (is_usable), and qualitative nuance (annotator_notes).

The Critical Need for Guidelines

As with the image schema, this text schema is only a blueprint. Language is inherently more ambiguous than a picture of a meal. What one person considers "disapproval," another might see as an "accusation." This is why detailed annotation guidelines are non-negotiable.

For every violation type and every severity level, you will need to write a clear definition with positive and negative examples. For instance:

• Severity: Accusation vs. Blame: "Use Accusation when the user states a fault (e.g., 'You served expired food'). Use Blame only when they also link that fault to a negative outcome (e.g., 'Because you served expired food, my child is sick')."
• Violation Type: quality_poor_preparation: "Use for complaints about the texture, consistency, or cooking process. Examples: 'undercooked rice', 'burnt curry', 'watery dal'. Do not use for complaints about taste or smell; use quality_bad_taste_smell instead."

Anticipating these edge cases and documenting them is the core work of creating a high-quality text dataset.

Conclusion

You have now successfully applied the same structured, data-centric principles from our last lesson to the domain of text. By breaking down complaints into violation types (aspects) and severity levels, you have created a schema that can capture the nuanced and often complex information contained in parent feedback.

Key Takeaways:

• Deconstruct Text Complaints: Frame the problem as identifying two key dimensions: the violation type (aspect) and the severity level.
• Ground Severity in Theory: Using a formal linguistic framework like Trosborg's four levels (No Explicit Reproach, Disapproval, Accusation, Blame) provides a more robust and consistent scale than subjective terms like "low/medium/high."
• Design a Multi-Label Schema: Use a nested structure (e.g., a list of violation objects in JSON) to allow a single complaint to be tagged with multiple violation-severity pairs.
• Prioritize Guidelines for Ambiguity: Language is subjective. Clear, detailed annotation guidelines are even more critical for text data to ensure all annotators interpret labels consistently.

With draft schemas for both your image and text data now complete, you have a solid foundation for your project's ground truth. In the next lesson, we will focus on the practical task of writing the comprehensive labeling guidelines for both schemas, preparing you to run a pilot annotation exercise.