Automating desktop And web tasks With Power Automate – building UI Flows

Paul Stork

0 comments

This introduction to how Power Automate UI flows work will make it easier for you to automate the business processes that your company relies on. When you finish reading this eBook, you should understand how to create UI flows and include them as extensions to other Power Automate flows for a complex business process solution.

Using computers to Automate repetitive business processes is one of the best ways to gain efficiency in today’s business world. Power Automate has made this kind of business process automation available to intelligent business users and not just to professional developers. But until recently, Power Automate could only automate processes that ran on systems with pre-built connectors or that had a well-defined set of Application Programming Interfaces (APIs). But what about systems that had no connectors or APIs? With the addition of UI flows, these systems can also be automated using a recorded script that duplicates the actions of a user. Power Automate can now offer a single automation service that can handle almost any application or computer software system.

In this eBook, we will look at Power Automate’s entry into the world of Robotic Process Automation (RPA) called UI flows. We’ll cover how Power Automate flows can automate repetitive tasks even if a connector isn’t available. We’ll review how UI flows can automate repetitive tasks on your desktop in Windows or in a Web application. We’ll see how UI flows record and playback user interface actions, like keyboard entries, mouse clicks, etc. We’ll also examine how UI flows can be activated by other Power Automate flows to create a fully integrated automated business process.

When you finish reading this eBook, you should understand how to create UI flows and include them as extensions to other Power Automate flows for a complex business process solution. You’ll learn how to:

  • Install and configure all the prerequisite components required to create UI Flows
  • Record a UI flow that records your interaction with a desktop application
  • Record a Web UI flow that automates your interaction with a web application
  • Invoke and pass parameters to a UI flow from another Power Automate flow

What is Robotic Process Automation (RPA)?

Wikipedia describes Robotic process automation (RPA) as “a form of business process automation technology based on metaphorical software robots (bots) or on artificial intelligence (AI)/digital workers.” While that description is accurate, it does not really describe one of the unique features of RPA. One of the characteristics of RPA that set it apart from other forms of automation is the use of screen scraping and keyboard polling to create scripts to automate tasks. Where other forms of automation focus on interacting directly with software systems through connectors based on Application Programming Interfaces (APIs), RPA makes use of the existing user interface provided by an application. This lets companies with legacy systems automate their workflows even if the application does not have a published API. Companies are excited about RPA solutions because:

  • Many large enterprises still rely on older legacy systems
  • Many knowledge workers still complete automatable work using these legacy systems

What are Power Automate UI Flows

UI flows in Power Automate is Microsoft’s entry into the world of RPA. The new UI flows capability is designed specifically to automate older legacy desktop and web-based applications. Power Automate UI Flows can record two different kinds of interactions with legacy systems. Those two types include:

  • Desktop applications
  • Web-based applications using Edge Chromium or Google Chrome browser

The recorder for desktop applications is built into the Windows 10 or Windows Server 2016 or later operating systems. As a result, you can only record Desktop applications when running on recent versions of these operating systems. This may cause an issue for some legacy desktop applications that won’t run on these newer operating systems. Web applications use a third-party recorder called Selenium IDE. The Selenium IDE recorder runs as an extension on recent builds of Google Chrome or the new Edge Chromium browser.

UI Flow scripts can also be played back in two different ways. They can be run in an:

  • Attended mode – The UI flow runs Inside the current user’s session
  • Unattended mode – The UI flow runs in the background on a separate computer without no active user login

Why Use Power Automate UI Flows?

Using a UI flows recorder extension for desktop applications or the third-party Selenium Ide extension for web applications, you can record interaction steps with an application user interface (UI). The recorded script can include mouse clicks and keyboard entries. Once the UI flow recording has been saved, it can be invoked manually or added to any automated instant or scheduled flow.

Desktop application scripts can be edited in an editor like the one used by other Power Automate flows. Web scripts are edited directly in the Selenium Ide recorder. Using either recorder, you can add or delete specific step actions. This enables you to make changes to only the parts of the flow that need to be updated, without you having to rerecord the entire flow. You can also parameterize the steps to make use of dynamic content provided by Power Automate during invocation.

Using Power Automate UI Flows provides the following advantages. You can:

  • Automate data input for legacy systems without published APIs or connectors
  • Record, Edit, and Replay repetitious tasks
  • Accelerate low-code automation by recording existing processes

UI Flow Licensing

To record and run UI flows it’s important to understand how they are licensed. The screenshot below from the Power Automate UI flows pricing page shows the retail price for the two different licenses available for UI flows. Let’s compare how this licensing relates to the existing Power Automate licensing.

There are three different ways that a user can be licensed to run regular Power Automate flows. These licenses do not include licensing for the use of UI flow. A brief explanation of each Power Automate licensing scenario is listed below:

  • Seeded licensing – User’s with Power Apps licenses can run flows that support the Power Apps that they are licensed to create/run at no additional charge. The flow does not have to trigger from a Power App but cannot be a completely separate standalone scenario. For example, if a user creates a Power App to submit a document to a SharePoint document library, they can also write an Approval flow for the document library. This is covered by their Power Apps license even if the flow is triggered automatically by the creation of the document rather than triggered from a button in the Power App. If the user has standalone per user or per-app licensing, they can even use premium connectors in the flow.
  • Per-user licensing – Makers who do not have a Power Apps license or who want to create standalone flows that are not covered by a Power App scenario can also buy a standalone Power Automate license. The standalone license retails for $15 per user per month. Many standalone flows only require a license for the maker of the flow and not for every user in your organization. For example, a flow that would be triggered automatically when a Microsoft form is submitted does not require a flow license for the person submitting the form, only for the maker of the flow.
  • Per-flow licensing – If a given flow requires licensing for all the users in addition to the maker of the flow. And if those users do not have their own standalone flow licenses then the flow itself can be licensed. A standalone license for a single flow can be purchased for $100 per flow per month. A minimum of 5 per flow licenses must be included in the initial purchase. Additional per-flow licenses can be purchased individually.

There are two different licenses available for Power Automate UI flows, depending on whether you wish to run UI flows in an attended or unattended fashion.

  • Per-user licensing with RPA – A per-user license is available that includes the ability to run attended UI flows for a retail price of $40 per user per month. This is the same as the $15 per user per month standalone flow license mentioned above, but with the added ability to run attended UI flows.
  • Unattended RPA – Running flows unattended on another machine or VM requires at least one Per-user license with RPA. This is because you will need to have at least one per-user license with RPA to record the UI flow. Unattended UI flows can then be started in multiple parallel sessions on a single computer or on multiple computers. Each concurrent unattended session, otherwise known as a bot, requires an Unattended RPA license. The unattended RPA license retails for $150 per bot per month. How many unattended bots can run simultaneously on one workstation/server will be dependent on the performance characteristics of the machine.

Preparing to record UI Flows

One of the most frequent problems that new users run into when they first start using UI flows is that they do not verify all the prerequisites or required software components before they try to record. In this section, we will review each of the prerequisites required for different recording scenarios and walk through how to install them.

UI Flow prerequisites

Before you install any of the software components required for recording UI flows, make sure that you meet all the prerequisites. You must have the following to install UI flow:

  • A Paid or Trial Power Automate UI Flow per-user license
  • An Office 365 work or school account with access to Power Automate and local administrator privileges on the Laptop/workstation
  • The workstation must be a 64-bit device that running one of the following supported operating systems
  • Windows 10 Pro
  • Windows Server 2016 or later
  • Microsoft Edge Chromium (version 80 or later) or recent Google Chrome browser
  • A Power Automate environment with a Common Data Service (CDS) database
  • A keyboard set to use a supported layout attached to the laptop/workstation
  • US keyboard – English (United States)
  • US keyboard – English (Australia)
  • US keyboard – English (Canada)
  • Microsoft Pinyin – Chinese (Simplified Han, China)
  • German keyboard – German (Germany)
  • Microsoft IME – Japanese (Japan)
  • United Kingdom keyboard – English (United Kingdom)
  • French keyboard – French (France)
  • Russian keyboard – Russian (Russia)
  • Portuguese (Brazilian ABNT) keyboard – Portuguese (Brazil)
  • Portuguese (Brazilian ABNT2) keyboard – Portuguese (Brazil)
  • Microsoft IME – Korean (South Korea)
  • Spanish keyboard – Spanish (Spain)
  • Italian keyboard – Italian (Italy)
  • Latin American keyboard – Spanish (Mexico)
  • Polish (Programmers) keyboard – Polish (Poland)
  • United States-International keyboard – Dutch (Netherlands)
  • Turkish Q keyboard – Turkish (Turkey)
  • India keyboard – English (India)

Installing UI Flow Recorders

Once you have verified that you’ve met all the prerequisites, you can install the software that will let you record UI flows. Not all of the following software components are required for every scenario. The following list walks through what to install for each scenario.

  • Download and run UI Flow recorder installation program, Setup.Microsoft.PowerAutomate.Uiflow.exe. You can download it from the following link -https://go.microsoft.com/fwlink/?linkid=2102613
    1. Accept the terms of use and click Install.
  1. Wait for the installation to finish and click Close.
  • Verify that the UI flows browser extension has been activated in Edge or Chrome.
  • Click Add the Extension to accept the terms and add the extension.
  • Enter the email address of the work or school account you will be using to record/run UI flows and click Sign in. Complete the Office 365 login dialogue if it pops up.
  • Click Next to register the new on-premises gateway with Power Automate.
  • Give the Gateway a name and enter a recovery key that can be used to restore the gateway if it needs to be reinstalled later. Click Configure to complete the configuration.
  • Wait for the configuration to finish and click Close.
  • Once your on-premises gateway is installed, you can use the Power Automate website to create a UI Flow connection to your desktop. You can record UI flows without the connection, but the connection is required to invoke a UI flow from another Power Automate flow.
  • Login to the Power Automate website at https://flow.microsoft.com/ using your work or school account.
  • Expand the Data tab in the left sidebar and Select Connections.
  • Click on New connection
  • Search for UI flows and select the UI flows connector.
  • Provide your work or school account login information and choose the gateway you created previously. Click Create.
  • You should now have a UI flows connection showing a status of Connected in your list.

Now that all the prerequisites have been verified and UI flow components have been installed, you are ready to record your first UI flow.

UI Flow Limitations

Before you try to record your first UI flow, we should discuss the limitations that currently exist for UI flows. The list below covers all the things that you cannot currently do when recording a UI flow.

  • The Windows 10 Home operating system is not supported
  • Desktop UI Flow limitations
  • You can only record actions on a Single Monitor, although the computer can be set up with multiple monitors.
  • Certain mouse actions cannot be recorded. These include Double click and mouse hovering.
  • Touch/pen interactions are not recorded
  • You can only record actions inside Windows Application. Interactions with the Operating System or built-in apps like File Explorer are not supported.
  • Microsoft Office applications earlier than Office 2016 are not supported
  • There are issues with some Legacy applications, particularly Windows Forms applications. Workarounds exist for many of these limitations.
  • Web browser interactions cannot be recorded as part of a Desktop UI flow. Use a Web UI flow instead.
  • Web UI flow limitations
  • Right clicking to access a context menu is not recorded.
  • User session information not recorded with the script. For example, User login information must be parameterized or re-added to the script after recording.

You may also run into limitations with specific desktop applications. This is especially true if the legacy application isn’t compatible with Windows 10 desktop or one of the modern browsers.

Desktop UI Flows

Now that you have all the prerequisites and components installed, let’s look at how you create a UI flow script. For this walkthrough, we will use a Windows Form application written in C# and compiled in Visual Studio that simulates an Expense reporting system. This is exactly the kind of legacy app that your organization may still be using. Since it was custom developed, it doesn’t provide a set of Web APIs that can be used to bypass the user interface. This is a common trait in legacy applications, whether they were custom or commercially developed. Recording the user interface interactions with UI flow will make it possible to automate interaction with this type of application.

Recording a Desktop UI Flow

This walkthrough will use a custom Windows Form application, but you could automate other Windows desktop apps by using similar steps.

  • Login to Power Automate at https://flow.microsoft.com using your work or school account.
    Note: Make sure you are using either Edge Chromium or Google Chrome.
  • Select  My flows > UI flows > + New.
  • Choose the Desktop app and click Next.
  • Enter a name for your UI flow and click Next.
  • You can use this screen to create Input variables that can be used as generic parameters while recording your script. You will also be able to create these input parameters while doing the recording. Add an Input variable with the following values:
  • Input Label: Expense Name
  • Sample Data: Sample Expense
  • Input Description Sample Expense Item Description

Click Next to proceed to the next screen.

  • This is the screen where you will be able to edit the actions in your UI flow after you record it. Click Launch recorder to open the recorder control to begin recording your interactions with the Desktop Application.

Note: Values for sensitive text inputs will be obscured when entered and will not be persisted after saving for flow.

  • After a few seconds the recorder control will display at the top of your screen. Click Record to start recording your interactions with the Desktop Application.
  • Start the Expense Reporting legacy application and fill in a Sample Expense. Click Submit.

Note: If you use your mouse to navigate in the application make sure you wait for the blue outline around the input field before selecting each control and continuing.

  • While recording you can use the controls on the recorder to add additional Input variables or define output variables. Output variables can be used to retrieve status codes or text from the application to be returned to the script.
  • After you have completed your Expense entry, click Done in the recorder control at the top of the screen. This will return you to the UI flow editor.
  • The editor will contain a list of the actions that you took while interacting with the application. We will look at how to edit and test these in the next section. Click Save to save your work so far.
  • Close the application that you recorded.

Editing a Desktop UI Flow

Now that you’ve created your recording you can begin editing the script. You can add new actions, delete existing actions, or just edit the properties of an action. Let’s look at each of these in turn.

Add New Actions

Adding an action in the UI flow editor operates the same way it does in the regular flow editor. Hover your mouse between two actions, and a dialog will appear that lets you add an action.

Using the dialog, add an Insert text input action.

Note: Desktop UI flows have a limited set of only eight (8) actions available.

Edit Action Properties

Once an action has been added, you can fill in the properties of the action with dynamic content the same way you do in regular Power Automate flows. Input and Output variables that you created in the script will display in the dynamic content.

  • Add the Expense Name input variable we created to the Value property

Note: You will find two important properties under the advanced options in most actions.

  • Wait after action – This will contain an ISO time format string for how long to wait between this action and the next in the UI flow. It will be set based on the actual timing of the recording. You can change it to speed up or slow down the script. For example, an entry of PT1S will delay one second before proceeding to the next action.
  • Selector – The Selector property contains JSON that identifies the control that is the focus of the input.
    NOTE: At this time, there is an issue with the recorder not recording the selector properly when using Input controls. The workaround is to input the values in the recording using the keyboard, then replace that action with an Insert text input action and copy the Selector from the Replay keystrokes action before deleting it.

Delete Existing Actions

Actions can be deleted using the context menu on each action in the editor. Now that we’ve added an Insert text input action to replace the first Replay keystrokes action let’s delete that action.

  • Click the ‘’ elipse on Replay keystrokes 1 action to access the context menu. Select Delete and verify the deletion by clicking OK in the confirmation dialog.

Testing a Desktop UI Flow

Once you are done editing the actions in your flow you can test it to make sure it works as expected. The final step in the recorder wizard provides a screen where you can add sample values for Input variables and then text the script.

  • To test a Desktop UI flow click Next until you read the Test panel of the wizard.
  • Click the Test now button to run the UI flow. A reminder dialog will remind you not to interact with your desktop while the script is running.

Note: Taking any action on your desktop while the test is running may affect the running of your UI flow.

  • When the flow finishes, you will see a run report that designates which actions ran successfully and whether any actions failed similar to the one in the screenshot below. This screen is very similar to the one you receive when a regular Power Automate flow runs.
  • Once the application has run successfully, you can save and exit the wizard and begin using your Desktop UI flow.

Web UI Flows

In addition to recording Desktop application interactions, UI flow can also record interactions through a web browser with a web application. For this walkthrough, we will use an ASP.Net Web application that simulates the same Expense reporting system that we used for the Desktop recording. The primary difference is that Web UI flows us a popular third party recorder called the Selenium IDE.

Recording a Web UI Flow

This walkthrough will use a custom ASP.Net Web application, but you could automate other Web site apps using similar steps.

  • Login to Power Automate at https://flow.microsoft.com using your work or school account.
    Note: Make sure you are using either Edge Chromium or Google Chrome.
  • Select  My flows > UI flows > + New.
  • Choose Desktop app and click Next.
  • Enter a name for your UI flow and the Base URL of the Website you wish to record.
  • Click Launch Recorder to load the Selenium IDE recorder.
  • Click the Rec button to start recording your interactions with the web site. This will automatically open a browser to the website at the Base URL you specified.
  • As you enter values in the web site, an indicator will flash in the lower right corner of the screen to show you that the Selenium IDE is recording your steps.
  • After you finish entering all the information on the website, go back to the Selenium IDE recorder and click the Stop button in the upper right corner.
  • Click the Save icon to save your work so far.

Editing a Web UI Flow

Just as you can edit the Desktop UI flow actions using the recorder, you can edit the Web UI flow actions using the Selenium IDE editor. Unlike the Desktop interaction, you can only define Input and Output parameters after you finish recording the UI flow. You can also add new actions using the panel at the bottom of the editor. Let’s look at each in turn.

Edit Action Properties

To edit an existing action in the script, you select the line and modify the Command, Target, or Value. One common reason for editing existing actions is to change their value from a hardcoded value to an input parameter that will be passed in when the script is invoked.

  • Select the Type Command line that inputs the value of the Expense Name.
  • Replace the Value property with a parameter called ExpenseName by changing the Value property to ${ExpenseName}
  • Leave the Command and Target unchanged.

This will create a parameter called ExpenseName that can be set using Dynamic Content when the UI Flow is invoked from another power Automate flow.

Add New Actions

To add a new action in the Selenium IDE, you need to specify the Command, the Target control, the Value to be entered (if any), and an optional description. One of the reasons for adding an action after you have finished the recording is to capture a value from the web site as an Output variable.

  • Select the line at the end of the script or just below where you want to add a command. Right Click on the row and select Insert new command.
  • Select the blank command line that was added and enter the following for the Command the Target and the Value.
  • Command: store text
  • Target: Click the arrow icon highlighted in the screenshot below. This will re-open web application. Click on a label or text box on the screen to select the field that will be output. This will store the label text in the output variable.
  • Value: result

Note: store commands are used to retrieve content from the web page and store it in the variable specified in the value.

Delete Existing Actions

Actions can also be deleted using the context menu on each action line in the editor.

  • Right click the line that contains the action you wish to delete. Select Delete from the context menu.

Note: Deletions in the Selenium IDE are immediate. There is no confirmation dialog like there is in the Desktop recorder.

Testing a Web UI Flow

Once you are done editing the script, you can test it using the Selenium IDE recorder. By default Web UI flows will run as fast as the web site will accept input. You can slow the replay speed down for testing purposes.

  • To run a script, click the play button in the menu.
  • To change the play speed, adjust the slide between Fast and Slow on the stopwatch icon dropdown.

When the script finishes running, each line will be color-coded to show whether it ran successfully or not. You can also examine the Log file for the run at the bottom of the screen.

Automating UI Flows with Power Automate

Although you can run UI flows directly from the Power Automate Web site, doing so is only meant to be used for testing purposes. Efficiency is also not high when running the flows one at a time. The real benefits of automation are achieved by scripting the runs as batches with input and output parameters. We can do this by using a regular Power Automate Flow to run the UI flow. Dynamic content in flow can then be used to fill in the Input parameters for the UI flow. The output parameters from the UI flow will also be available after each flow run as Dynamic content so we can respond to the success or failure of a particular UI flow.

Let’s look at a simple example of how to call a UI flow, provide an Input parameter, and evaluate an output parameter.

  • Login to Power Automate at https://flow.microsoft.com using your work or school account.
  • Select  My flows > My flows > + New > + Instant—from blank.
  • Enter a Name for the flow and select to Manually trigger a flow, then click Create.
  • Click the New Step button to add a new action. Select Compose to create an array of dynamic content that we can use as Import parameters in our UI flow.
  • Add the following array of Expense Names to the Compose Inputs field.
    Inputs: [“Sample Expense 1”, “Sample Expense 2” , “Sample Expense 3”]
  • Click the New Step button to add a new action. Select Apply to each to create a loop to process each entry in the Compose array.
  • Use the Dynamic content panel to add the Compose Outputs to the Apply to each loop.
  • Click the Add an action inside the Apply to each loop to add a new action. There are two UI flow actions available, Run a UI flow for web and Run a UI flow for desktop. Select Run a UI flow for desktop.
  • Select a UI flow from the dropdown.
  • Select a Run Mode of Attended or Unattended from the Dropdown.
  • Use the Dynamic content panel to add the Current item from the loop to the Expense Name parameter field.
  • You can now save and run the Power Automate flow. It will run the UI flow three times, once for each entry in the array.

Conclusion

That brings our discussion of how to automate desktop and web tasks with Power Automate UI Flows to a close.

If a connector or API exists for an application, you will find it more efficient to use a regular Power Automate flow. But in cases where these do not exist, Power Automate UI flows can be a very powerful tool for automating legacy applications in your environment.

I hope this introduction to how Power Automate UI flows work will make it easier for you to automate the business processes that your company relies on. Power Automate UI flows can bring even legacy processes into the modern world of office automation and streamline the way your knowledge workers conduct business.

He has been awarded the Microsoft Most Valuable Professional (MVP) award for the last 12 consecutive years and earned his Microsoft Certified SharePoint Masters (MCSM) certification in 2013. He has also been awarded dual Super User status in the Power Apps and Power Automate forums.

His core skills include architecting and customizing SharePoint environments both on-premises and in the cloud. He has significant experience leading SharePoint migration teams, troubleshooting infrastructure issues, and evaluating SharePoint environments for adherence to “Best Practices” and industry standards. His current focus is on Office 365 security, DevOps using DSC, and Microsoft PowerApps and Flow. He’s authored several books and is a frequent contributor to the Microsoft Power Platform forums.

0000-00-00 00:00:00


Leave a Reply

Your email address will not be published. Required fields are marked

{"email":"Email address invalid","url":"Website address invalid","required":"Required field missing"}