In this eBook, you’ll learn about AI Builder, discover the different AI models available, and then drill down into some of the models in the form of demos. Finally, a list of learning resources will help you get started on your journey with AI Builder.
The Power Platform has four major pillars:
- Power BI – allows you to analyse your data
- Power Apps – lets you act on your data
- Power Automate – helps you to perform automation across all your data
- Power Virtual Agents – allows you to build intelligent chatbots
All four pillars of the Power Platform can connect to your data sources, no matter where they live (on-premises or in the cloud), via the data connectors. There is also a data layer that the Power Platform exposes, now called ‘Dataverse’. Finally, low-code artificial intelligence capabilities are available in the Power Platform, through AI Builder.
What is AI Builder?
AI Builder is a low-code platform capability that helps you add artificial intelligence (AI) to your apps and flows without any coding or data science skills. It is a solution that brings the power of AI through a point-and-click experience. AI Builder comprises ready-to-use AI models and custom models. AI Builder models can be used directly in apps and flows.
To get to AI Builder, go to ‘make.powerapps.com’. The AI Builder option is in the navigation on the left-hand side, where you can either build models or look at your existing models.
AI Builder is also an option in the left-hand navigation of Power Automate – go to flow.microsoft.com. Again, you can either build models or look at models you’ve already created.
Models that you build with AI Builder are specific to the environment in which they are created. Data related to AI Builder is stored in Microsoft Dataverse (formerly known as the Common Data Service or CDS).
Models built are environment-specific and can be leveraged by apps and flows in the same environment.
AI Builder Models
There are two main types of models available in AI Builder: prebuilt and custom.
Prebuilt models help you add intelligence to apps and flows without having to gather data and then build, train, and publish your own models. There are currently nine prebuilt models available in AI Builder:
|Prebuilt model type||Category||Description|
|Business Card Reader||Vision||Allows you to automatically process business card information.|
|Key Phrase Extraction||Language||Gives you the key talking points from the text that you provide to the model.|
|Language Detection||Language||Identifies the language being used in the text that you provide to the model.|
|Sentiment Analysis||Language||Helps you to analyse the sentiment of the text that you provide.|
|Entity Extraction||Language||Returns data about entities and their types.|
|Category Classification||Language||Allows you to categorise text based on its meaning.|
|Text Recognition||Vision||Optical character recognition automatically processes text from images.|
|Receipt Scanning||Vision||Allows you to receive and save information directly from receipts.|
|Text Translation||Language||Automatically translates text into a new language.|
Seven of the models are generally available. The only two that are not generally available and are not in Preview are the Category Classification and Receipt Scanning models.
Custom models are models that you build by choosing a model type. You build a model, train it to do specific AI tasks using your data, and then publish it so you can use it in apps and flows and share it.
There are five custom model types:
|Custom model type||Category||Description|
|Form Processing||Vision||Allows you to read and save information from standard document types.|
|Object Detection||Vision||Lets you recognise and count things in images.|
|Prediction||Prediction||Predicts whether or not something will happen. Heavily relies on data within Microsoft Dataverse.|
|Category Classification||Language||Allows you to categorise text by its meaning.|
|Entity Extraction||Language||Helps you recognise specific information about your business from data.|
Both Category Classification and Entity Extraction are available as prebuilt models. If you would like to add your own types to these models, you can use the custom model options.
Let’s see how we can leverage these models in our apps and flows to enhance our business processes.
To get started with AI Builder, go to make.powerapps.com (the maker portal for Power Apps). In the left-hand navigation, click on Build under AI Builder to see a list of all the AI Builder models available for you.
The AI models at the top are the five custom models. Below them are the productivity models – also known as prebuilt AI models.
Demo – Prebuilt Models
We’re going to look at how some of the AI Models can be used in apps within Power Apps. First, you need to turn on the preview features for AI Builder models– in Power Apps, select File, Settings, Advanced settings, search for AI Builder, and set ‘Use AI Builder models through the formula bar’ to On.
Demo – Sentiment Analysis
The sentiment analysis prebuilt model detects the sentiment of textual data that you provide to it. The model returns a sentiment of Positive, Negative, Neutral or Mixed.
In this example, we enter some text and click on ‘Detect Sentiment’. The model detects that the overall sentiment is positive. Individual statements are broken down, analysed, and displayed in a gallery. The only statement so far is ‘The food was delicious, and the staff was wonderful’, which results in positive analysis.
In the next example, we add a negative statement – ‘The customer representative though was very rude’. The overall sentiment is Mixed (neutral emoji), but the individual statements are broken down into positive (smiling emoji) and negative (sad emoji) sentiments.
How do you build something like this? In Power Apps, you need a text input control and a button. When you click on the Detect Sentiment button, it calls the AI Builder.AnalyzeSentiment function. For this function, all you have to provide is text, which is stored in a variable called ‘sentimentOutput’.
In the Items property of the gallery, we have used the sentimentOutput.sentences variable, which gives an output (into the gallery) of all the sentences the model has detected.
Within the gallery is a text field that provides the text from the statement.
In the Icon property, you can link sentiments to relevant emojis (smiling, sad, neutral).
Demo – Language Detection
The Language Detection prebuilt model detects the language of the text you provide. In the example below, we enter some text and click on ‘Detect Language’. The language is detected to be Spanish (es).
This screen was created in Power Apps using a text control button. When you click on the button it calls the AIBuilder.DetectLanguage prebuilt function. You just need to provide the text (txtInputLanguageDetect). The variable detects the language and gives a confidence score of between 0 and 1. 1 means it’s 100% confident that the language is accurate.
Demo – Text Recognition
Text Recognition performs optical character recognition (OCR). If you add an image to this prebuilt model, it will extract any text from the image.
The model also detects handwritten text.
This screen was created in Power Apps by selecting Insert, AI Builder and then the ‘Text recogniser’ control. This control appears in the top half of the example app. Once it has scanned the image the control concatenates the results and displays them in a label control.
Demo – Key Phrase Extraction
The Key Phrase Extraction model identifies the main points in a piece of text. In the
the example below, we see that the key talking points in the text are ‘staff’ and ‘food’.
Here’s another example:
This app is based on the AIBuilder.ExtractKeyPhrases control. You provide the text (txtInputKeyPhrase) and output is provided in the form of a detectedKeyPhrases record.
Demo – Business Card Reader
This model scans images of business cards and displays certain items that it detects.
The image above shows the Power App in a web browser. On a mobile device, the app would open the device’s camera so that the user could take a live photo of a business card. The same applies to the text recognition model.
In Power Apps, select Insert, AI Builder, Business card reader to add the control to an app. The control exposes many typical business card properties and you can display them within label controls.
Demo – Entity Extraction
The Entity Extraction model identifies key elements from text and classifies them into predefined categories. In the example below, the model extracts ‘May 5th’ as a DateTime property and gives a score of 0.8 accuracy (80%). ’86 degrees’ is highlighted as a temperature, and ‘blue’ is identified as a colour.
You can see a different range of entities that have been extracted in the next example.
This app uses AIBuilder.ExtractTextEntities and txtInputEntityExtract. The output is a record containing the properties ‘textScore’ and ‘entities’ (tabular data).
Demo – Receipt Scanning
This expenses-tracking app uses the Receipt Scanning prebuilt AI model.
The user fills in an expense report, entering data in the following fields:
- Start Date
- End Date
- Cost Center
For each report, you can add a line item.
For each line item, you need to upload a receipt. The Receipt Scanning model has been used in Power Apps to add a receipt scanning control to the data entry form.
When you click on ‘Scan Receipt’ you can upload the image of your receipt. The model then scans the receipt and fetches key data points from the image.
If you run the app on a mobile device, you can take a photo of the receipt to scan and process it.
Demo – Prebuilt Models in Power Automate
Each of the prebuilt AI models that we have seen in Power Apps can also be used in Power Automate. Let’s look at one scenario – a customer satisfaction survey that includes a comments or questions section.
We want to be able to detect the sentiment of the customer’s comments or questions. If the sentiment is negative, we want to send a notification message to ourselves.
You can do this in Power Automate. The trigger of your flow is when a new response is submitted in the customer satisfaction survey. The details are obtained and then analysed for positive or negative sentiment, using the prebuilt Sentiment Analysis model. You provide the text of the question from the Microsoft form.
Next, you add a condition to check the overall sentiment. If the sentiment is not equal to positive (which means it’s either mixed, neutral, or negative), the flow sends you an email.
When the customer survey is submitted, the response is recorded in Microsoft Forms. The flow in Power Automate listens for new entries in Forms. It takes the comment text and sends it to AI Builder, which detects that the overall sentiment is mixed.
Since the sentiment was not positive, an email is sent.
To call an AI Builder model in a flow, click ‘Add an action’, select ‘AI Builder’ and select
the relevant prebuilt model from the list.
Demo – Custom Models
One of the most popular custom models is Form Processing, which allows you to read and save information from standard documents.
When you select a model in Power Apps, you can give it a name – we’ll call this one ‘Invoicing Model’ and click ‘Create’.
The Form Processing model works with documents that have a consistent format. In this example, we’re trying to automate our invoicing process. We receive invoices from two different companies: Contoso and RD. The Contoso invoices have a consistent format, containing an invoice number, date, customer name and payment details. The elements in the RD invoices are arranged differently. How do we build and train a model to fetch data points from different invoices?
First of all, we’ll enter three data points that we want to capture: Invoice Number, Invoice Data and Customer Name.
We also want to fetch some tabular data – we create a table called ‘Invoice Table’ and
add columns Description, Quantity, Price and Total, and then click Next. You can create collections of similar documents.
When you click ‘Analyse’ the Form Processing model scans all the documents in your collections and fetches data points. After the analysis, you can tag the fields and tables you added. The model loads all the fields and tables it has detected as part of its AI training.
The fields and tables listed on the right are the ones you added manually. You now tag the detected elements with the ones you added. Select a highlighted element and then choose the relevant tag from the dropdown. For elements that have two parts to them, such as a first name and surname, drag over both parts to select them together.
In a similar way, you tag the individual elements of your tables.
The model learns as you tag so when you come to the next document in the collection you’ll find that the model has already tagged parts of it, based on what it has learned in the previous tagging process. When you have finished training all the documents in your collection, click on ‘Back’ and tag the documents in the other collection.
Click ‘Next’ to go to the Model summary, which shows the different fields and tables you are extracting. Click on ‘Train’ to start the process of training your model.
Once the training is complete, you can publish the model and use it in Power Apps and Power Automate.
When you click on ‘Use model’ and select ‘New app’ it creates a Power App and adds the control for the Form Processing model directly in Power Apps.
Click on ‘Analyse’ to upload a document or image to the app. On a mobile app, you can take a photo of an invoice document to upload it. The model now identifies the fields and tables in the document that you trained it to detect. Once it completes, you can add the key data points from the model – in this example, the invoice number has been added.
In Power Automate, you can create a flow that triggers when a new invoice document is uploaded. That flow can, in turn, call the forms processing model, leverage the training in the model and tag all the properties of the documents, including the invoice type (in this case, company name).
The Form Processing model is really powerful, especially when used in Power Automate. You can even add some SharePoint formatting to enhance your invoice processing solution further.
Let’s look at the flow in action for this use case.
The flow is triggered when a file is created in the ‘Invoices’ library.
The flow grabs the metadata properties associated with the file.
Next, we use the ‘Process and save information from forms’ action. We just needed to pick the model, select the type of form, and then provide the content of the file.
This produces dynamic properties, which are the same fields and tables for which we trained the model.
We add the tabular data in a secondary list.
With Power Automate, you can create automated scenarios wherein you can leverage the power of AI Builder to automate your business processes.
- AI Builder Get Started – https://docs.microsoft.com/en-us/ai-builder/overview
- AI Builder Licensing – https://docs.microsoft.com/en-us/ai-builder/administer-licensing https://powerapps.microsoft.com/en-us/ai-builder-calculator
- Invoice Processing Solution – AI Builder Forms Processing Model & Power Automate – https://www.youtube.com/watch?v=3ggCWEymNXI
- AI Builder Prebuilt Models Exploration Sample App – https://powerusers.microsoft.com/t5/Community-App-Samples/AI-Builder-Prebuilt- Models-with-Power-Apps-Ready-to-use/td-p/590780
AI Builder is a Microsoft Power Platform capability that helps you add artificial intelligence to your apps and flows without any coding or data science skills. Powerful and prebuilt and custom models can be used directly in apps and flow to enhance your business processes.