Integrating AI Builder Models into Power Apps

1. Methods of Integration in Canvas Apps

When you integrate an AI model into a Canvas App, you are essentially making an API call to the AI Builder service and then processing the JSON response it sends back. Power Apps simplifies this into two primary methods:

1. AI Builder Components: These are specialized, pre-built controls available in the "Insert" pane (e.g., Form processor, Object detector). They provide a user interface for capturing input (like using the device camera) and handle the call to the model. This is the most straightforward method for models that process images or documents.
2. Power Fx Predict Function: This is a more direct, code-based approach where you call the model using a formula: 'Your Model Name'.Predict(input). This method offers maximum flexibility and is ideal for models that process text or when you want to build a fully custom UI for triggering the model and handling its output.

First, let's see how to add your model to the app. Regardless of the integration method you choose, the first step is always to add the AI Builder model as a data source.

2. Integrating an Image-Based Model: Form Processing

Let's start with a common and powerful scenario: building an app to scan invoices and automatically extract key information. This uses the Form Processing model you've learned about and the dedicated AI Builder component.

This video from Reza Dorrani provides an excellent end-to-end demonstration. We'll focus on the part where he integrates the trained model into a Power App.

As you saw in the video, the process involves these key steps:

1. Insert the Component: Add the Form processor control from the AI Builder section of the Insert pane.
2. Link the Model: In the properties panel for the component, set the AI model property to the form processing model you added as a data source.
3. Process the Output: After a document is analyzed, the component's Results property is populated. This property contains all the extracted data.
   • Accessing Fields: To display a single extracted field, you access it through the Fields property. For example, if your component is named FormProcessor1 and you extracted a field named InvoiceID, the formula for a label's Text property would be:

     FormProcessor1.Results.Fields.InvoiceID.Value
     

   • Accessing Tables: To display line items from an extracted table, you use the Tables property as the Items source for a gallery. If you extracted a table named InvoiceItems, you would set the gallery's Items property to:

     FormProcessor1.Results.Tables.InvoiceItems
     

     You can then bind labels inside the gallery to the columns of that table, like ThisItem.Description or ThisItem.Amount.

3. Integrating an Image-Based Model: Object Detection

The pattern for integrating an object detection model is very similar. This is useful for scenarios like inventory counting, retail shelf analysis, or quality assurance.

Let's look at another practical video that builds an inventory counting app.

The key takeaway here is how the output of the Object detector component is structured.

• Accessing Detected Objects: The ObjectDetector1.VisionObjects property returns a table of all objects the model is capable of detecting.
• Displaying Results: You can set a gallery's Items property to ObjectDetector1.VisionObjects. Each record in this table contains:
  • DisplayName: The name of the detected object (e.g., "Coca-Cola Can").
  • Count: The number of times that object was detected in the image.
  • BoundingBox: The coordinates of the detected object(s).
• Filtering for Detected Items: To show only the items that were actually found in the picture, you can filter the gallery:

  Filter(ObjectDetector1.VisionObjects, Count > 0)
  

As shown in the video, you can then use this information within a ForAll or Patch function to update an inventory list in Dataverse, turning the AI detection into a powerful data entry tool.

LookUp(ObjectDetector_Main.VisionObjects, DisplayName = "Hard Hat").Count

4. Integrating a Text-Based Model with Power Fx

Now let's cover the "user input" part of our learning outcome by integrating a model that processes text. For this, we'll use the Predict function, which gives you more granular control. We'll use the pre-built Language detection model as a simple and effective example.

Here's a summary of the process:

1. Add the Model: Add the Language detection model as a data source.

2. Add Controls: Add a Text input control (TextInput1) for the user to type in and a Text label control (Label1) to show the result.

3. Use the Predict Function: Set the Text property of Label1 to the following formula:

   'Language detection'.Predict(TextInput1.Text).Language
   

   The Predict function returns an object, and we are accessing the Language property of that object.

Developer Best Practices

As the article mentions, blindly binding the Predict function to a control's OnChange property is inefficient. It will call the AI model on every keystroke, which consumes AI Builder credits rapidly and can slow down your app.

A much better pattern, especially given your development background, is to treat the AI model call like any other API call:

• Trigger On-Demand: Place a Button on the screen. Use its OnSelect property to call the model. This gives the user explicit control.
• Cache the Result: Store the entire output object from the Predict call in a variable. You can then use this variable to populate multiple labels (e.g., one for language, one for confidence score) without making redundant API calls.

Here's the improved implementation:

Button's OnSelect property:

// Call the model once and store the entire result object in a variable
Set(
    gblLanguageResult,
    'Language detection'.Predict(TextInput1.Text)
)

Label for Language (Text property):

gblLanguageResult.Language

Label for Confidence Score (Text property):

Text(gblLanguageResult.Confidence * 100, "0.0") & "%"

This approach is more performant, cost-effective, and aligns with professional software development practices. The Microsoft Learn article "Use AI Builder models in Power Apps" also includes a helpful table of inputs and outputs by model type, which is an invaluable reference for knowing the exact structure of the object returned by each model.

Conclusion

In this lesson, you have learned the two primary methods for integrating AI Builder models into Power Apps. You can now build sophisticated applications that leverage AI to process images, documents, and user-provided text.

Key Takeaways:

• AI models are added to Canvas Apps via the Data pane.
• Image and document models are best integrated using their dedicated AI Builder components (e.g., Form processor). The output is accessed via the component's Results property.
• Text-based models and custom scenarios are handled with the 'Model Name'.Predict(input) Power Fx function.
• The output of a model call is an object or table that you parse to display the results.
• For performance and cost-efficiency, trigger AI model predictions on-demand (e.g., a button's OnSelect) and store the results in a variable for reuse.

What's Next?

So far, you've used AI Builder interactively within a Power App. But what about automating backend processes? In the next lesson, you will learn how to integrate a trained AI Builder model into a Power Automate flow to automate document or text processing. This will enable you to build powerful, end-to-end automations, such as processing invoices attached to emails without any user interaction.