Skip to main content
Back to path

Welcome to a Grasp lesson

Create a free account to keep studying and create your own study path.

Lesson 71

Extract Text from Images with AI Builder

Recommended resources

Hello! Welcome back to our module on "Applying AI Builder Models."

In our last session, we successfully built a Power Automate flow that integrated a custom document processing model. You learned how to trigger a flow from an event, use the AI Builder action to extract structured data from a form, and then use that data to create records in Dataverse, even handling table data with Apply to each loops.

Today, we'll continue working with AI Builder in Power Automate, but we'll switch our focus to a powerful prebuilt model. Our goal is to learn how to use the prebuilt text recognition AI model in a flow to extract text from an image.

This is different from what we did last time. The document processing model is designed for structured or semi-structured documents where you know what fields to expect (e.g., "Invoice Number," "Total Amount"). Text recognition, on the other hand, is about Optical Character Recognition (OCR). It extracts all printed or handwritten text it can find in an image or PDF, making it perfect for unstructured sources like a photograph of a whiteboard, a scanned letter, or a screenshot. This is a fundamental capability for digitizing information and is a key skill for the PL-400 exam.

1. Introducing the Text Recognition Model

The text recognition model is one of AI Builder's prebuilt models, meaning you can use it immediately without any training. It takes an image or PDF file as input and returns the text it finds.

To get started, we'll build a simple cloud flow that uses this model. The official Microsoft Learn documentation provides a clear, step-by-step guide for this.

Use the text recognition prebuilt model in Power Automate

The article 'Use the text recognition prebuilt model in Power Automate' from Microsoft Learn is the best place to start. It will walk you through creating the flow and adding the correct AI Builder action.

Please read the section 'Initialize the Power Automate cloud flow'. Follow the steps to create an 'Instant cloud flow' that is manually triggered and accepts a file as input. Then, add the 'Recognize text in an image or a PDF document' action from the AI Builder connector. Pay attention to how the 'File Content' from the trigger is passed to the 'Image' input of the AI Builder action.

As you saw in the guide, the initial setup is straightforward:

  1. Trigger: We use a manual trigger with a file input for easy testing. In a real-world scenario, this could be When a new email arrives (V3) or When a file is created in a folder (SharePoint).
  2. Action: The core of the flow is the Recognize text in an image or a PDF document action.
  3. Input: You provide the file content from your trigger to the Image parameter of this action.

Now, let's look at what the model gives back. As a developer, understanding the output structure is critical.

Understanding the Model's Output

The Text Recognition model returns a structured JSON object. The most important parts are:

  • results: An array where each item represents a page in the document.
  • lines: Inside each result item, there is a lines array. Each item in this array is an object representing a single line of detected text.
  • text: The actual string of text for that line.
  • boundingbox: The coordinates of where the text was found in the image.

Power Automate also provides two convenient high-level outputs for simpler scenarios:

  • Full text of the document
  • Full text of the page

2. Processing Extracted Text in Your Flow

Once the AI model has run, you need to decide how to use the text it extracted. There are two main patterns for this.

Pattern 1: Using the Full Text

For simple use cases where you just need the entire block of text, you can use the Full text of the document dynamic content. This is the quickest way to get the result.

For example, you could extract all the text from an image and post it to a Teams channel for review.

Integrating AI Builder Text Recognition with Power Automate and Microsoft Teams
This flow demonstrates using the `Full text of the document` output from the AI Builder text recognition action directly in the message body of a 'Post message in a chat or channel' action in Microsoft Teams.

This approach is best when you don't need to parse or analyze the text but simply want to store or display it as a whole.

Pattern 2: Processing Text Line by Line

For more advanced scenarios, you need to iterate through the text line by line. This is necessary when you're searching for specific information, need to reformat the output, or want to perform conditional logic on each line.

Since the lines output is an array, you'll use the Apply to each control, just as you did in the previous lesson for table data. Power Automate is smart enough to know that the model can return text from multiple pages, so it will often automatically create two nested loops for you: one for pages (results) and one for lines (lines).

The following article provides a concise example of this pattern.

PowerApps AI Builder: How to Leverage Image and Text ...

The article 'PowerApps AI Builder: How to Leverage Image and Text Recognition' by Satyam Singh Chauhan offers another practical example. We will focus on the Power Automate part, which showcases the line-by-line processing technique.

Please read the section 'Here is an example of how to use AI Builder for image and text recognition in a Power Automate flow'. Notice the use of the 'Apply to each' control on the 'Lines' field and the expression items('Apply_to_each')['Text'] used to access the text of the current line.

The general steps for this pattern are:

  1. Initialize a Variable: Before the loop, create a variable (e.g., a string or an array) to store the results you care about.
  2. Apply to each: Add an Apply to each control and select the lines output from the AI Builder action.
  3. Process Each Line: Inside the loop, you have access to the Text of the current line. You can:
    • Append it to a string variable.
    • Use a Condition to check if it contains a keyword.
    • Use expressions to transform it.
  4. Take Action: After the loop, use the processed data stored in your variable.

The image below shows a complete flow that recognizes text, processes it in a loop, and then sends an email—a common business automation pattern.

Power Automate Flow for AI Builder Text Recognition
This flow illustrates a practical sequence: recognizing text from an image, initializing a variable, looping through the results, and finally using the extracted information to send an email.
Test your understanding!

Imagine you are using the text recognition model to process images of shipping container doors. The container ID you need to extract is always on a line that starts with the prefix "ID: ".

How would you design a flow to extract only the container ID number (without the prefix) from an image and store it in a string variable? Describe the key actions and logic.

Show answer
  1. Trigger and AI Action: Start with a manual trigger that accepts a file and use the Recognize text in an image or a PDF document action.
  2. Initialize Variable: Add an Initialize variable action. Create a string variable named ContainerID and leave its initial value empty.
  3. Apply to each: Add an Apply to each control and provide the lines output from the AI Builder action as its input.
  4. Condition: Inside the loop, add a Condition control. Set the condition to check if the Text dynamic content (from the current line in the loop) starts with the value "ID: ".
  5. Set Variable: In the "If yes" branch of the condition, add a Set variable action. Select the ContainerID variable. For the Value, use an expression to remove the prefix: replace(item()?['text'], 'ID: ', ''). This expression takes the text of the current item in the loop and replaces the prefix with an empty string.
  6. End of Flow: After the Apply to each loop, the ContainerID variable will hold the extracted ID number, ready to be used in subsequent actions like updating a Dataverse record.

Conclusion

In this lesson, you've added another powerful AI capability to your Power Automate toolkit. You learned how to use the prebuilt text recognition model to digitize text from images and PDFs, a common requirement in business process automation.

Key Takeaways:

  • Text Recognition vs. Document Processing: Use text recognition for unstructured text extraction (OCR) and document processing for extracting specific, named fields from structured forms.
  • Processing Outputs: You can either grab the Full text of the document for simple cases or use an Apply to each loop on the lines output for granular control and parsing.
  • Practical Application: This skill is crucial for automating tasks like digitizing scanned archives, processing photos of notes, or extracting information from non-standard documents where a custom model isn't practical.

What's Next?

We've now seen AI Builder integrated into both Power Apps and Power Automate. In our final lesson for this module, we will explore another prebuilt text model, but this time we'll integrate it directly into a Power App. You will learn how to use the prebuilt key phrase extraction AI model in an app to analyze user-provided text, which will enable you to build apps that can understand and react to user input in real-time.