Microsoft Unveils Autonomous Computer Use for AI Agents in Copilot Studio

Microsoft lets Copilot Studio use a computer on its own

Copilot Studio will be able to create AI agents that can automate data entry, perform market research, or even process invoices. This new tool will even detect when buttons and screens change in apps or websites and continue working without failing or breaking, according to Microsoft. Microsoft added a similar feature, dubbed Actions, to its consumer Copilot earlier this month. Actions can be performed in the background while you work on other tasks. Copilot can now do things like book restaurant reservations, event tickets, and purchase items from online stores. The Actions experience is limited to a number of partners though, and it sounds like Copilot Studio will be free to work with more websites and apps.

With Copilot Studio's new skill, your AI agent can use websites and apps just like you do

According to Microsoft, 'if a person can use the app, the agent can too.' Here's how to try it out. Imagine an AI agent that can not only conduct research on the web but also interact with websites to accomplish specific tasks -- all on its own. That's the idea behind a new innovation from Microsoft. On Wednesday, the software giant announced an interactive new skill designed to empower the AI agents that people create using its Copilot Studio product. That skill is computer use. Any agent you build can work with a desktop application or website to carry out specific actions just as you might. Also: What are AI agents? How to access a team of personalized assistants Arriving through an early access research preview, the new skill will allow agents to interact with apps and sites by clicking buttons, selecting menus, and filling out fields and forms on a screen. The agents can accomplish interactive tasks even if no API is available to use the app or site. As Microsoft phrased it: "If a person can use the app, the agent can too." To carry out tasks on their own, the agents you design will be able to use mainstream browsers, such as Microsoft Edge, Chrome, and Firefox. The feature itself runs on a backend hosted by Microsoft, so you don't need your own servers or systems. The data generated remains on the Microsoft cloud and won't be used to train the company's AI model. The agents can automatically adapt to changes in desktop apps and websites, so you shouldn't have to modify them if a button or screen is revised. Since they're able to "see" the screen, they can decide what to do and how to do it in real time without your intervention. They also possess built-in reasoning to grapple with any problems or obstacles along the way, so you shouldn't have to step in if an issue arises. No programming or coding is required to build an AI agent. At the prompt in Copilot Studio, you simply describe what you want the agent to do using natural language. You can test and fine-tune your prompt as you see the steps play out in a simulated mode before sending the agent out on its mission. Plus, you can view a history of your agent's computer use, complete with screenshots and its own reasoning steps. Also: ChatGPT just made it easy to find and edit all the AI images you've ever generated In its blog post, Microsoft cited three examples in which the AI agents could play a role. Automated data entry: You need to input large amounts of data from different sources into a central system. The AI agent can take on this task, reducing the time, effort, and manual labor on your end. Market research: Someone in marketing needs to collect data from different online sources for analysis. The AI agent could browse to the various sites and collect the needed information on its own. Invoice processing: Someone in finance has to extract data from invoices and add the information into an accounting system. The AI agent could automatically grab and transfer the data into the right system. This all sounds great in theory. But today's AI is far from perfect. Even a plain old AI bot can make mistakes. Add in the complexity of automated and independent computer use, and the AI agent could conceivably take a misstep, click the wrong button, or simply give up on its mission. Also: Copilot Studio is like a LEGO set for building AI agents The only way to tell is by creating an agent to work with a website or app and see how it fares. With that in mind, Microsoft is inviting Copilot Studio users to fill out and submit a sign-up form to get early access to the computer use feature. If you don't already use Copilot Studio, you can try it for free. To set it up, you'll need to use a work or school account. Once signed in, you're placed at the main Copilot Studio page, where you can describe the type of agent you want to create.

Microsoft: Why not let our Copilot fly your computer?

Redmond talks up preview of AI agents navigating apps through the UI Microsoft will soon let Copilot agents drive computers through the GUI just like humans - by clicking buttons, selecting menus, and even completing forms on screen. On Wednesday, the Windows empire said it plans to enable computer use from within Copilot Studio - Microsoft's platform for building and deploying AI agents. This will spare employees from having to click buttons and fill forms themselves, while still keeping enterprise data corralled inside Microsoft's cloud - Redmond insists none of it is used to train its models. "Computer use enables agents to interact with websites and desktop apps by clicking buttons, selecting menus, and typing into fields on the screen," explained Charles Lamanna, corporate VP for business and industry, Copilot, in the corp's marketing bumf. "This allows agents to handle tasks even when there is no API available to connect to the system directly. If a person can use the app, the agent can too." AI agents are, as far as we can tell, pieces of software that talk to other pieces of software as well as users, using generative AI to make decisions and form outputs. Today, Microsoft Copilot Studio enables customers to create AI-driven agents to automate certain tasks, but these agents only work with specific services, like SharePoint. The new type of agents should be much more flexible. For instance, you could create an agent and prompt it to carry out a series of steps that involve browsing a previously unseen website, extracting some data, and passing that data to a desktop app. Lamanna suggests several scenarios where the new Copilot agents could come in handy, such as automating the input of large amounts of data from multiple sources to a central repository, automatically collecting market data for research, or using AI text and image recognition capabilities to process invoices. AI automation differs from programmed instructions in that the agent can adapt on the fly when it encounters obstacles or unexpected changes in the interface. Instead of crashing with an error, it uses built-in reasoning to muddle through, at least according to Microsoft. "Computer use adapts to changes in apps and websites automatically," Lamanna claimed. "It adjusts in real time using built-in reasoning to fix issues on its own, so work continues without interruption." With any luck, said reasoning does not involve unexpected deletions or policy violations, as one concerned user fretted about in a a social media thread solicited by a Copilot Studio product manager. However, turning over computational tasks to Copilot may involve unanticipated costs. As with cloud services, the bill for AI's boil-the-ocean approach to computation use isn't necessarily easy to anticipate and there's potential for bill shock if certain tasks turn out to be computationally demanding. Concerns about costs have been raised by users of OpenAI's computer use API, and by users of Anthropic's computer use API. Microsoft is bringing computer use to Copilot Studio users through an early access research preview that requires a signup. Expect to hear more about this at Microsoft Build 2025 next month. ®

Microsoft Launches Computer Use Agent for Copilot Studio | AIM Media House

Microsoft has announced new autonomous computer use capabilities in the Copilot Studio. This feature is accessible through an early access research preview. The feature can autonomously navigate through a user interface and perform actions like clicking buttons, selecting menus and typing text into fields. "With computer use in Copilot Studio, makers can build agents that automate tasks on user interfaces across both desktop and browser applications, including Edge, Chrome, and Firefox," Microsoft said. "If a person can use the app, the [AI] agent can too," it added. The company will reveal more details about this feature at the Microsoft Build event in May. Microsoft has now joined the ranks of companies such as Anthropic, OpenAI, and Google by offering autonomous computing capabilities on its platform. Last December, Microsoft launched the Copilot Vision feature, which allows users to browse the web with AI assistance. With user permission, Copilot Vision can read and analyse web pages, providing insights, simplifying information, and more. Recently, Microsoft announced that it is rolling out the Copilot Vision feature to Windows Insiders, a community of users who have opted in to access and test these features pre-release. Last month, the company announced a new 'Researcher' feature in Microsoft 365 Copilot, similar to the deep research tools offered by today's AI models and platforms. The feature combines OpenAI's deep research model with Microsoft 365 Copilot's capabilities. It can also integrate third-party data from platforms like Salesforce, ServiceNow, and Confluence. Moreover, Microsoft announced an 'Analyst' feature designed to function like a "skilled data scientist". This feature is built on OpenAI's o3-mini reasoning model and can run Python to execute data queries, allowing users to view the code the tool runs in real time. "You can use Analyst to turn raw data scattered across multiple spreadsheets into a demand forecast for a new product, a visualisation of customer purchasing patterns, or a revenue projection," Microsoft said.

Satya Nadella Says Microsoft's New Tool Allows Anyone To Create AI Agents That Click, Type, And Navigate On Your Computer For You - Microsoft (NASDAQ:MSFT), Salesforce (NYSE:CRM)

On Tuesday, Microsoft Corporation MSFT announced a research preview of "Computer Use," a new tool within Copilot Studio that enables AI agents to perform tasks on desktop and web applications by interacting directly. What Happened: The tool allows agents to click buttons, type into fields, and navigate menus -- even in environments where no APIs exist. Microsoft CEO Satya Nadella also took to X, formerly Twitter, and said, "We're bringing computer use agent capabilities to Copilot Studio - enabling anyone to build agents that take action on the UI across both desktop and web apps." Powered by a large language model, the "Computer Use" tool can quickly adjust when app or website layouts change. Moreover, users don't need to write any code -- just type what they want in plain English and give the required instructions to the tool. See Also: Netflix Debuts OpenAI-Backed Search Engine That Lets You Discover Movies And TV Shows Based On Emotions, Not Just Titles Users can also see a side-by-side view of what the AI is doing and how it's making decisions, making it easy to test and fine-tune the instructions. Microsoft also highlighted enterprise readiness, stating that the tool runs on Microsoft-hosted infrastructure, with all customer data remaining within Microsoft Cloud boundaries. The company said that data would not be used to train any large language models. Subscribe to the Benzinga Tech Trends newsletter to get all the latest tech developments delivered to your inbox. Why It's Important: During Microsoft's second-quarter financial results, Nadella said, "We are innovating across our tech stack and helping customers unlock the full ROI of AI to capture the massive opportunity ahead," adding, "Already, our AI business has surpassed an annual revenue run rate of $13 billion, up 175% year-over-year." Last month, Microsoft launched Dragon Copilot, an AI-powered assistant developed to simplify clinical workflows by integrating voice dictation, ambient listening, and generative AI. In January, Salesforce Inc. CRM co-founder, chairman, and CEO Marc Benioff criticized Microsoft's Copilot, stating that the chatbot has already flopped. He argued that it lacks user adoption and is essentially a rebranded version of OpenAI's ChatGPT. In February, it was reported that Microsoft was scaling back on its AI data center footprint in the U.S. by canceling a large number of leases. This development came as a surprise, given that back in January, Microsoft had announced plans to invest approximately $80 billion in building AI-driven data centers and expanding the global rollout of AI and cloud-based applications. Price Action: Microsoft shares have fallen 7.85% year-to-date and are down 6.96% over the past 12 months, according to Benzinga Pro data. The average price target for Microsoft stands at $502.57, based on assessments from 32 analysts. The most recent targets from BofA Securities, UBS, and Wells Fargo average approximately $486.67, suggesting a potential upside of 26.47%. Benzinga's proprietary Edge Rankings give Microsoft a momentum rating of 46.80% and a growth rating of 64.72%. For deeper stock analysis and potential growth picks, consider subscribing to Benzinga Edge. Check out more of Benzinga's Consumer Tech coverage by following this link. Photo Courtesy: katuSka On Shutterstock.com Read More: 'Most People Don't Have The Balls To Do It,' Says Mark Cuban, Praising Musk For Going 'All In' With His Own Money For His Startups Disclaimer: This content was partially produced with the help of AI tools and was reviewed and published by Benzinga editors. CRMSalesforce Inc$252.11-0.96%Stock Score Locked: Want to See it? Benzinga Rankings give you vital metrics on any stock - anytime. Reveal Full ScoreEdge RankingsMomentum40.52Growth95.80Quality68.80Value7.67Price TrendShortMediumLongOverviewMSFTMicrosoft Corp$384.82-0.77%Got Questions? AskWhich companies may adopt Microsoft's AI tool?How will salesforce adapt to AI advancements?Could cloud computing stocks see a rise with Microsoft's innovations?Which tech firms will compete with Copilot Studio?How might enterprise software companies leverage AI agents?What impact on productivity tools from AI integration?Are investors bullish on Microsoft's AI initiatives?Which industries will benefit from automation via Microsoft's tools?Could AI-driven applications disrupt traditional software markets?How will customer service transform with AI agents?Powered ByMarket News and Data brought to you by Benzinga APIs

Microsoft's New 'Computer Use' Feature Empowers AI Agents To Seamlessly Navigate And Interact With Websites And Apps

The tech community is increasingly focused on using AI to automate processes and streamline tasks, and it keeps on working to introduce novel, innovative features that would make users' lives easier. OpenAI, with its Operator, intensified the competition in the industry by bringing ahead AI agents that can handle tasks on users' behalf. Now, Microsoft seems to be heading in the same direction as it is taking a big step with its new computer use feature in Copilot Studio this week, which is meant to allow AI agents to use websites and apps automatically. Tech giants are focusing more on using AI to do mundane tasks for users in order to reduce the time consumed in repetitive tasks. OpenAI, by introducing Operator, brought forward the experience of human-like interactions through its super AI agent. It looks like Microsoft is also working on making Copilot more than just an assistant and has now brought ahead a new feature in Copilot Studio that is called "computer use:" which would allow AI agents to use websites and apps just like a human would. The reason why this is a big leap is that the tech giant is paving the way for AI agents to automate tasks even if there are no built-in integrations or APIs available. Businesses can use this feature to build agents in the Copilot Studio that are able to fill reports, log into websites, or even handle customer service tasks. By taking action on behalf of the user, the platform automatically becomes more capable. A senior leader at Microsoft, Charles Lamanna, says that the new feature is meant to make AI agents appear like real users as they would able to click, choose, and type information just like a person would. While explaining, Lamanna stated: This allows agents to handle tasks even when there is no API available to connect to the system directly. If a person can use the app, the agent can too. Microsoft's new update is meant to make AI agents smarter and more reliable, especially when automating real-world tasks. Whether invoices need to be processed, market research needs to be performed, or data needs to be entered across websites, Copilot Studio can now be used to create AI agents that can handle these tasks. The feature's most interesting part is the AI agents' ability to adapt to changing interfaces, leading to more dependable automation. Microsoft had a similar feature called Actions introduced a while back, which allowed users to let the Copilot handle tasks in the background so that they could focus on other tasks. While Action is meant for personal use, Copilot Studio's computer use feature seems to be a more advanced tool help automation on a business-level.