Custom Object Detection with AI Builder

Hello! Welcome back to the "Applying AI Builder Models" module.

In our last lesson, you learned how to create and train a custom form processing model. We focused on the structured lifecycle: defining the data to extract, organizing documents into collections, tagging samples, and then training and testing the model.

Today, we'll apply that same fundamental lifecycle to a new and exciting domain: image recognition. Your learning outcome is to train a custom AI Builder object detection model using a custom image dataset. Instead of teaching an AI to read documents, you'll teach it to see and identify specific items within a picture. This skill is crucial for building solutions in areas like retail inventory management, manufacturing quality control, and asset tracking.

1. From Reading Documents to Seeing Objects

While the core process of training a custom AI model remains similar, object detection has its own unique concepts and requirements. In form processing, the goal was to extract text and tables. Here, the goal is to identify one or more objects within an image and draw a "bounding box" around each one, reporting what it found and where.

The overall model lifecycle, however, should feel familiar.

AI Builder Model Lifecycle Flowchart — This flowchart shows the general lifecycle of any AI Builder model: creating a draft, training it (which can be an iterative process), publishing it for use, and managing versions. This applies equally to both the form processing model you built last time and the object detection model we're building today.

2. The AI Builder Object Detection Process

Training an object detection model involves a clear, wizard-driven sequence of steps. Let's break them down.

Step 1: Define Your Object Names

Before you begin, you must decide what objects you want your model to detect. These are your "labels" or "tags". AI Builder offers two ways to provide these names:

Enter names manually: Simple and direct, suitable for a small, fixed list of objects.
Select from a Dataverse table: A more scalable and enterprise-ready approach. This allows you to manage your object list dynamically within a Dataverse table, which is a pattern you'll be very familiar with from your Dynamics 365 work.

The process often starts outside Power Apps, by preparing your list of objects.

Object Detection Application with AI Builder and Power Apps

The article 'Object Detection Application with AI Builder and Power Apps' provides a great real-world example of preparing your object names in an Excel file and importing them into Dataverse to create a new table. This is an excellent practice for managing your labels.

Please read the first two parts of the article, covering the prerequisites and the process of creating a new entity (table) from an Excel file. This demonstrates a practical way to manage your object labels before you even start building the AI model.

Step 2: Create the Model and Upload Your Image Dataset

Once your object names are ready (either in your head or in a Dataverse table), you can start creating the model.

Build your object detection custom model

The official Microsoft Learn documentation, 'Build your object detection custom model', walks you through the initial creation steps.

Please read the sections 'Select the model domain', 'Provide object names', and 'Upload images'. Pay close attention to: The concept of domains (Common objects, retail shelves, etc.), which optimizes the model for specific use cases. The two methods for providing object names. The ability to upload images from local storage, SharePoint, or Azure Blob Storage.

Key Considerations for Your Image Dataset:

The quality of your model is entirely dependent on the quality of your training images. This is where your developer mindset for creating robust test data comes into play.

Quantity: You need a minimum of 15 images per object. For a high-quality model, 50 or more is recommended.
Variety: This is critical. Do not use "perfect" studio shots. Your images should reflect the real-world conditions where the app will be used. Capture your objects:
- From different angles.
- In different lighting.
- Against various backgrounds.
- With other objects nearby (even partially obscuring them).

Step 3: Tag Your Images

This is the most hands-on part of the process. For each image you've uploaded, you must manually "teach" the model by drawing a bounding box around every object of interest and assigning the correct label.

AI Builder Object Detection Model Training Workflow — This image collage illustrates the key stages of building an object detection model. The central part shows the tagging interface, where you draw bounding boxes around objects (the tea cans) and assign labels. On the right, you can see the model performance score after training.

This tagging process creates the "ground truth" data that the machine learning algorithm uses to learn what each object looks like.

Build your object detection custom model

The Microsoft Learn guide provides a clear explanation of this crucial step.

Read the section 'Tag images'. Note the requirement of tagging at least 15 images per object name before you are allowed to proceed to the training step.

Step 4: Train, Test, and Publish

After tagging, the final steps are straightforward:

Train: You click the "Train" button and AI Builder does the heavy lifting, creating the model based on your tagged images. This can take some time.
Quick Test: Once trained, the model details page will show a performance score. More importantly, it provides a "Quick Test" button. Use this to upload a new image (one not used in training) to see how the model performs in practice.
Publish: If you are satisfied with the test results, you publish the model. This makes it available as a component in Power Apps and an action in Power Automate.

3. End-to-End Video Walkthrough

Now let's see this entire process in a live demonstration. The following video ties all these steps together into a coherent workflow.

Using the Power Apps AI Builder Object Detector 🤖 [Episode 41]

The video 'Using the Power Apps AI Builder Object Detector' from Pragmatic Works is a fantastic, practical guide that covers the entire process from data preparation to testing the final model.

Please watch the video from 01:10 to 08:43. As you watch, focus on how the presenter executes each of the steps we've just discussed: (01:10 - 03:25): Preparing the Dataverse table with object names and starting the new AI model. (03:41 - 06:43): Uploading images, discussing the importance of quantity and variety, and demonstrating the tagging process. (06:43 - 08:43): Reviewing the trained model's performance score, using the 'Quick Test' feature, and publishing the model.

Test your understanding!

Imagine you are tasked with building a Power App for a warehouse to quickly count the number of three specific types of boxes: "Type A", "Type B", and "Type C". The boxes are often stacked on pallets, sometimes with labels partially obscured, and under varying warehouse lighting.

Based on what you've learned, what are two of the biggest challenges you would anticipate when creating the image dataset for this AI model, and how would you mitigate them?

Show answer

Here are two major challenges and their mitigations:

Challenge: Image Variety and Realism. Simply taking pictures of individual boxes on a clean floor won't work. The model needs to learn to identify boxes in their real, messy environment.
- Mitigation: You would need to conduct a "photo shoot" in the actual warehouse. Take hundreds of pictures of the pallets from multiple angles (front, side, high, low). Capture images where boxes are partially hidden behind others, where shadows are present, and under the different lighting conditions found throughout the warehouse (e.g., near a bright loading dock vs. a dim corner).
Challenge: Similar Objects and Partial Occlusion. If Type A and Type B boxes look very similar, or if a crucial part of the label is often hidden, the model can get confused.
- Mitigation: During data collection, intentionally take many pictures that highlight the subtle differences between similar boxes. Take numerous photos of partially occluded boxes. When tagging, be meticulous. Even if only a small, unique portion of a "Type A" box is visible, tag it. This teaches the model to identify objects even from partial evidence, making it much more robust in a real-world scenario.

Conclusion

Congratulations! You have now walked through the complete process of training a custom object detection model. You've seen that while the domain is different from form processing, the core principles of providing high-quality, representative data and meticulously tagging it remain the same.

Key Takeaways:

Object detection models identify and locate objects within an image using bounding boxes.
The training lifecycle is: Define labels -> Collect varied images -> Tag objects -> Train, Test, and Publish.
Using a Dataverse table to manage object names is a scalable best practice.
The success of your model depends heavily on the quantity (15+ per object) and variety (different angles, lighting, backgrounds) of your training images.
The "Quick Test" feature is essential for validating your model's real-world performance before publishing.

What's Next?

In both this lesson and the last, we've looked at the performance score and used the "Quick Test" feature. But what does that score really mean? And how can you systematically improve it? In our next lesson, you will learn to evaluate and improve AI model performance using test data and performance metrics, diving deeper into the iterative process of refining your custom models.

Recommended resources