Estimating Food Portions Using Reference Objects

From Bounding Boxes to Precise Area

In the last lesson, we extracted bounding boxes. While useful for locating objects, they are a crude tool for measuring area. A bounding box is always a rectangle, but food items are rarely rectangular. A box around a pile of rice or a piece of dal will include significant empty space, leading to inaccurate area calculations.

To get a more precise measurement, we need to know exactly which pixels belong to the food item itself. This is the job of instance segmentation. Unlike object detection, which draws a box around an object, instance segmentation creates a pixel-level mask that outlines the object's exact shape.

This image shows the power of instance segmentation. On the left is the original photo. The middle column shows the segmentation masks overlaid on the image, and the right column shows the food items extracted using these masks. This precise separation is what allows for accurate area calculation.

To perform instance segmentation, we'll use a variant of the YOLOv8 model. The models with a -seg suffix (e.g., yolov8n-seg.pt) are pretrained for this task. They conveniently output both bounding boxes and segmentation masks.

Accessing and Using Segmentation Masks

When you run a YOLOv8 segmentation model, the results object it returns contains a new attribute: masks. This object holds all the information needed to reconstruct the pixel mask for each detected item.

The following article provides a great introduction to using these segmentation models with the Ultralytics API.

As the article shows, the mask for each object is available through result.masks. We can extract this data, convert it to a NumPy array, and then calculate its area. Since the mask is a 2D array where non-zero pixels represent the object, calculating the area is as simple as counting these pixels.

This process is also demonstrated very clearly in the following video. It focuses on a scientific imaging problem, but the principles of extracting masks and calculating their properties are identical to our food-item task.

Implementing Portion Estimation

Our strategy will be to calculate the portion size as a ratio of the food's area to the plate's area. This relative measurement is robust to changes in camera distance and image resolution.

1. Detect all objects in the image using a YOLOv8 segmentation model.
2. Identify the reference object: Find the detection corresponding to the 'plate' (or 'bowl').
3. Calculate the reference area: Get the mask for the plate and count its non-zero pixels. This is our denominator. If no plate is detected, we cannot calculate a relative portion size.
4. Calculate food area: For every other detected food item, get its mask and count its non-zero pixels.
5. Compute the ratio: For each food item, the portion size is (food_mask_area / plate_mask_area).

This approach is validated by academic research in automated food analysis, which often relies on a food-to-plate area ratio.

Now, let's modify the script from our previous lesson to implement this logic. The new script will load a segmentation model and add a portion_ratio key to our output dictionaries.

import cv2
import numpy as np
from ultralytics import YOLO
import pprint

# Load a pretrained YOLOv8 instance segmentation model
model = YOLO('yolov8n-seg.pt')

def estimate_portions(image_path: str, confidence_threshold: float = 0.5) -> list:
    """
    Runs YOLOv8 segmentation, calculates mask areas, and estimates portion size
    relative to a detected plate or bowl.

    Args:
        image_path (str): Path to the input image.
        confidence_threshold (float): Minimum confidence score for detections.

    Returns:
        list: A list of dictionaries for each detected object, including a
              'portion_ratio' if a reference plate/bowl is found.
    """
    results = model.predict(image_path, verbose=False)
    result = results[0]
    
    # Ensure masks are present
    if result.masks is None:
        print("No masks found in the image.")
        return []

    class_names = result.names
    detections = []
    plate_area = None
    plate_detection = None

    # First, find the plate/bowl and calculate its area
    for i, box in enumerate(result.boxes):
        if box.conf.item() < confidence_threshold:
            continue
            
        class_id = int(box.cls.item())
        class_name = class_names[class_id]

        if class_name in ['plate', 'bowl']:
            # Use the mask area for the most accurate reference
            mask = result.masks.data[i].cpu().numpy()
            plate_area = np.count_nonzero(mask)
            
            # Store plate detection info but don't add to final list yet
            coords = box.xyxy.tolist()[0]
            plate_detection = {
                'label': class_name,
                'confidence': round(box.conf.item(), 4),
                'box': [int(c) for c in coords],
                'mask_area_pixels': int(plate_area)
            }
            # Assuming one main plate/bowl for reference
            break 
    
    if plate_area is None:
        print("Warning: No 'plate' or 'bowl' detected as a reference object.")
        # Optionally, you could fall back to using total image area as a reference
        # For now, we will proceed without calculating ratios.
    
    # Now, process all objects and calculate their portion ratio
    for i, box in enumerate(result.boxes):
        confidence = box.conf.item()
        if confidence < confidence_threshold:
            continue
            
        class_id = int(box.cls.item())
        class_name = class_names[class_id]
        
        # Skip adding the reference plate/bowl again if you only want food items
        if class_name in ['plate', 'bowl']:
            continue
        
        coords = box.xyxy.tolist()[0]
        mask = result.masks.data[i].cpu().numpy()
        food_area = np.count_nonzero(mask)
        
        detection_data = {
            'label': class_name,
            'confidence': round(confidence, 4),
            'box': [int(c) for c in coords],
            'mask_area_pixels': int(food_area)
        }
        
        if plate_area is not None and plate_area > 0:
            detection_data['portion_ratio'] = round(food_area / plate_area, 4)
        
        detections.append(detection_data)
        
    # Optionally, add the plate detection to the list if you want it in the output
    if plate_detection:
        detections.insert(0, plate_detection)

    return detections

# --- Main execution block ---
if __name__ == "__main__":
    # Replace with a path to one of your meal images
    image_to_process = 'path/to/your/meal_image.jpg'
    
    print(f"Processing image: {image_to_process}")
    
    # Get structured data including portion estimates
    detected_objects = estimate_portions(image_to_process, confidence_threshold=0.4)
    
    print("\n--- Detected Objects with Portion Estimates ---")
    if detected_objects:
        pprint.pprint(detected_objects)
    else:
        print("No objects detected with the given confidence threshold.")

Code Walkthrough:

1. Model Loading: We now load yolov8n-seg.pt, the segmentation model.
2. Two-Pass Logic: The estimate_portions function first iterates through the detections to find a 'plate' or 'bowl' to serve as our reference. It calculates its mask area and stores it in plate_area.
3. Area Calculation: The area of a mask is calculated with np.count_nonzero(mask). This is a direct and efficient way to count the pixels belonging to the object.
4. Ratio Calculation: In the second loop, for each food item, it calculates its mask area and then computes the portion_ratio by dividing the food area by the plate_area.
5. Enriched Output: The final dictionary for each food item now includes mask_area_pixels and, if a plate was found, portion_ratio.

Save this script (e.g., estimate_portions.py), point it to one of your meal images, and run it. You should see an output that not only identifies the food but also quantifies its size relative to the plate.

Conclusion

In this lesson, you've moved beyond simple object detection to the more powerful technique of instance segmentation. You've learned how to leverage pixel-level masks to calculate a much more accurate area for irregularly shaped food items. This allows you to generate a robust portion_ratio feature, a crucial piece of information for assessing meal quantity.

Your key takeaways are:

• Instance segmentation provides precise pixel masks, which are superior to bounding boxes for area calculation.
• YOLOv8's -seg models can be used to easily obtain these masks.
• The area of a mask can be found by counting its non-zero pixels using NumPy.
• Calculating a food-to-plate area ratio provides a reliable, scale-invariant estimate of portion size.

You now have a way to extract features describing both the what (classification) and the how much (portion estimation) from your images. In our next lesson, we will switch modalities and turn our attention to the text data. You will learn about the Transformer architecture and how to fine-tune a powerful language model, BERT, to classify the parent complaints from your dataset. This will form the second baseline model for your project.