Microsoft Expands Copilot Vision AI to Analyze Entire Desktop in Windows 11

Microsoft Is Testing Letting Copilot AI Interact With Your Whole Desktop

Thomas is a native of upstate New York and a graduate of the University at Albany. As a member of CNET's How To team, he writes about the intersection of policy, information and technology, and how you can best be served in that area. Outside of work, he can most often be found watching too many movies, reading too much, drinking too much coffee, or spending time with his cats. Microsoft has confirmed an upcoming expansion to its Copilot AI chatbot's screen-sharing abilities: Soon it will be able to process your entire desktop, not just certain parts of it. The company said Tuesday that this expansion of Copilot's Vision capabilities has begun public testing. This update, now rolling out to the company's Windows Insider program testers, will allow the AI chatbot to view, process and react to all aspects of a user's desktop, where beforehand it was limited to specific apps or windows. With this update in place, Copilot should be to process and react to multiple windows and apps at the same time. Microsoft touted the ability for it to assist with all aspects of a project you might be working on. The Vision feature can be activated by selecting the "glasses" icon and selecting which desktop you'd like to share, and can be closed out by hitting "Stop" or "X" in the composer. "When you share your desktop (or any specific browser or app window), Copilot can see what you see and talk to you about it in real time," Microsoft's blog post said. "It can help analyze content, provide insights, and answer your questions, coaching you through it aloud. Get tips on making improvements to your creative project, help with improving your resume, or guidance while navigating a new game."

Now Microsoft's Copilot Vision AI can scan everything on your screen

Jay Peters is a news editor covering technology, gaming, and more. He joined The Verge in 2019 after nearly two years at Techmeme. Microsoft is rolling out an update to Copilot Vision for Windows Insiders that lets the AI tool see everything that's on your screen. Previously, the tool was able to look at two apps at a time and connect the dots between the two, but with this change, it can look at your whole desktop or "any specific browser or app window," Microsoft says. Unlike Recall, which can automatically take regular snapshots of what's on your computer screen, using Copilot Vision is more like screen sharing during a video call: you can activate the feature by clicking the glasses icon in the Copilot app and selecting the desktop you want it to see. Copilot Vision can "help analyze content, provide insights, and answer your questions, coaching you through it aloud," according to Microsoft. "Get tips on making improvements to your creative project, help with improving your resume, or guidance while navigating a new game." Microsoft initially started testing Copilot Vision last year as a feature that could see what you're looking at while browsing the web in Edge. Copilot Vision can also answer questions about what it sees through your mobile phone's camera.

Windows 11 Will Soon Let You Show Copilot Your Whole Desktop

An upcoming change to Microsoft's Copilot Vision tool will let you share your whole desktop with the AI and get real-time feedback from the chatbot. Previously, Microsoft has only allowed the tool to look at a maximum of two apps at a time, but this change lets it look at all of your open tools at once. You can share your whole desktop, or a specific app but this update is currently limited to a beta. Think of it like hosting a video call where you can share your whole desktop to get live feedback, but instead of a virtual meeting with people you'll be hearing from Copilot AI. To turn it on, you can press on the glasses icon after hovering over the Copilot tool within the taskbar. It then lets you select the desktop you want to share, or whether you want it to have access to just oneapp. You can then turn off the session at any point by pressing on Stop in the same menu. Microsoft's announcement says, "It can help analyze content, provide insights, and answer your questions, coaching you through it aloud. Get tips on making improvements to your creative project, help with improving your resume, or guidance while navigating a new game." This update is currently limited to those on the Windows Insider program, but it's available across all channels. You also need to be in a market where Windows Vision is enabled, which is true for those in the US. The update is coming as part of version 1.25071.125. It's likely Microsoft will be undergoing a short testing period before rolling this feature out to a wider audience. However, we don't yet know how long it will be limited to Windows Insiders. Previous Copilot features include Recall that takes regular screenshots of what it sees on your screen to offer feedback through the AI. Copilot Vision has previously been focused on Microsoft Edge before expanding to other tools.

Copilot on Windows Can Now See Your Screen

Microsoft is rolling out a big update to its Copilot Vision for Windows Insiders that lets the AI tool see everything on your screen. This is a pretty significant change, as it could only look at two apps at a time before. Copilot will see your desktop, or a specific browser or app window. This new "Desktop Share" feature gives you the option of sharing your desktop with Copilot whenever you need. Once you do, it can see what you see and even talk to you about it in real time. It's a similar idea to how the ChatGPT app works on Windows and Mac, and how Gemini can see your screen if you let it. I've used this feature in AI, and it is pretty accurate for ChatGPT and Gemini. I imagine that Copilot will have the same accuracy that these chatbots have, which could be helpful in the right circumstances. Copilot can analyze content, provide insights, and answer your questions, all while coaching you through it aloud. The only thing you need to do is click the glasses icon in the composer. Then, select the desktop you want to share, and ask Copilot for help with whatever you're working on. When you're done, simply press 'Stop' or 'X' in the composer to stop sharing your screen with the AI. It's important to note that this isn't like Recall, which automatically takes regular snapshots of your screen in the background. Using Copilot Vision with desktop share is more like screen sharing during a video call. You actively choose to turn it on and off, and screenshots aren't taken and stored somewhere else. Related Copilot Is Finally Catching Up With Other Chatbots Big things happening. Posts 1 This update also makes it easier to access Vision from your voice conversations. If you're already talking with Copilot and realize you need to show it something on your screen for more context, you can now simply click the glasses icon and share your screen right then and there. This is a feature that you may not use all the time, but is incredibly useful when you do find a use for it. Just keep in mind that it still has the limitations that AI tends to have in a regular conversation. I have used this to check homework, and was given plenty of incorrect suggestions by Gemini despite it seeing exactly what I saw. Even still, there were many times it was better to show the AI the screen to get the information I needed from documents instead of fiddling with the Find search bar. This new Copilot app update is gradually rolling out to Windows Insiders across all channels through the Microsoft Store. So, you might need to be patient as it makes its way to your computer. Anyone who isn't part of this program will have to wait until the full release to use this feature, but the time in between will be spent fixing bugs and making sure it works correctly. Source: Microsoft, The Verge

Microsoft's Copilot Vision AI can now see your whole desktop -- here's what it does

Microsoft is rolling out an update to its Copilot Vision AI on Windows 11 PCs, allowing it to see your whole desktop and talk to you in real time. Coming to Windows Insiders, the update will let users share their "whole desktop with Copilot," allowing it to see what's on screen and offer contextual help. This was previously only available to use in a couple of apps at a time, but now Copilot Vision can analyze content on everything shown on a PC. Through "Desktop Share," Copilot Vision can deliver insights, answer questions on everything you do and give you tips on tasks, similar to Google's Gemini Live and Apple Intelligence. According to Microsoft, it will be able to "get tips on making improvements to your creative project, help with improving your resume, or guidance while navigating a new game," the blog post states. This Copilot Vision "Desktop Share" feature works as it says on the tin, by sharing your desktop with Copilot in real time (unlike Microsoft Recall, which captures screenshots of your display). This means you can ask questions on anything that's seen on screen, like "what does this error message mean?" or "how do I create a table in Excel?" To turn it on, simply click the glasses icon in the Copilot app and select the desktop screen you want to share. Of course, you can turn it off at any time by selecting "Stop" in the app. You'll also be able to activate Vision AI using your voice, as Microsoft is rolling out access to Vision from "Voice conversation." It also states you can add more context in conversations by selecting the glasses icon and typing in what's needed. Along with the recent Highlights feature, which automatically offers useful content based on the apps, documents or websites in browsers you're looking at, Microsoft is looking to give its Windows 11 PCs even more of an AI advantage. You can try out these 7 useful Copilot prompts to make the most of it. The Copilot Vision update is now gradually rolling out to Windows Insider channels where Vision AI is available, so if you haven't received the update yet, it will be coming eventually. You can join the Windows Insiders program by heading to Settings > Windows Update > Windows Insider Program, and then clicking Get Started.

Microsoft's Copilot Vision can now see your entire desktop - but it probably won't judge you for the clutter

To give users control, Copilot Vision must be enabled, so it will only look at what you want the AI to see Microsoft has opened the eyes of its Copilot AI on desktops to see the entire screen. The feature, which enables Copilot to look at and analyze things on your screen, can see any window or browser you want, as a feature Microsoft calls "Desktop Share." When Copilot Vision first launched, it was limited to just spotting what was happening in Edge. You'd ask a question about a webpage, it'd scan that page, and the AI would help explain what was there. With this update, it can now zoom out and see your whole desktop or any specific app window you choose to share You can pick what the AI assistant looks at by clicking on the eyeglasses icon in the Copilot app. Then you select the area you want the AI to look at, and you can discuss what's on the screen with the AI without having to describe what you're looking at endlessly. Let's say you're revising a resume and want it to sound more like a confident human and less like a panicked grad student. You can open the Word document, share the window with Copilot, and ask it to highlight specific types of experiences. Copilot sees the document itself and suggests edits. The same goes for PowerPoint, Photoshop, and even some games. You can ask Copilot for help with the settings menu in a game you've never played before, and it'll walk you through options with tips and suggestions for when you get stuck. In case you're worried about Copilot snooping on your secret romance novel or something else you'd rather it not see, you can always turn it off, too. For now, the update is only available to Windows Insiders who already have Copilot Vision. It will likely come to the general Windows 11 population soon. It might not seem too big a deal compared to other generative AI features, but it's a subtle way for Microsoft to embed its AI into quite literally anything on your computer. Instead of writing an email in one tab and having to describe it to the AI in another tab for help, it's all right there. Copilot Vision makes the process more intuitive. You digitally point to something you want help with, ask a basic question, and get an answer that makes sense in context. It's hard to overstate how much friction this could eliminate. It's not perfect, of course. Some specificity is needed since the AI can't read your mind. But an extra set of eyes to help you over the hump of designing a PowerPoint or shortening a meeting summary is not nothing. The dream of AI as a personal assistant has always been about context, the idea of an AI that could know what we mean, not just what we say. Copilot Vision is Microsoft's clearest step in that direction yet. Copilot Vision could quietly become one of the most useful AI tools Microsoft has ever made. And unlike Clippy, it will only pass judgment if you ask.

Microsoft's Copilot AI assistant can now optionally see your entire desktop -- maybe it will be able to finally explain what Windows error code 0x8007002c means

Rejoice, for Microsoft has upgraded its Copilot Vision AI assistant with access to your entire desktop in Windows 11. Mercifully, this is an opt-in feature limited to preview builds of Windows 11, not something being rolled out compulsorily via Windows Update, for now. So our AI bot overlords haven't quite taken over the PC. Microsoft announced the move on the official Windows blog. "Now you can choose to share your whole desktop with Copilot. When you share your desktop (or any specific browser or app window), Copilot can see what you see and talk to you about it in real time. It can help analyze content, provide insights, and answer your questions, coaching you through it aloud," Microsoft explains. Previously, Copilot Vision could only be given access to individual apps and windows. Microsoft says you can use the new capability to, "get tips on making improvements to your creative project, help with improving your resume, or guidance while navigating a new game." To get the whole process going, you "click the glasses icon in your composer, select the desktop that you'd like to share, and ask Copilot to help with whatever you're working on. To stop sharing, press 'Stop' or 'X' in the composer." In theory, you should be able to ask Copilot Vision almost anything related to what's being shown on the screen using natural language voice prompts. Maybe it could be helpful for, I dunno, working out how to do a macro in Excel. I'll tell you what could be really handy, having those inscrutable alphanumeric Windows errors codes explained. If it can do that, colour me impressed. How this wil fit in with Microsoft's plan to enable customisable persona for Copilot isn't clear. Apparently, you'll even be able to have Copilot appear like ye olde Clippy, the animated paperclip assistant from Office 97. So, if you want Clippy riffling through everything you do on the desktop, this update may be for you. As that implies, there are obvious privacy concerns. Microsoft doesn't exactly have a impeccable track record in this area, what with it transpiring that the Recall feature in Windows 11 was originally storing desktop snapshots as unencrypted images. But, for now, this is very much an opt-in for hardcore Windows fans and devs on the WIndows Insider program. Speaking of which, this update for Copilot is part of Windows 11 build 1.25071.125 and currently limited to the Windows Insider program. You'll also need to be in a market where Windows Vision is enabled, which right now means only the US.

Copilot Vision now sees your whole screen

Microsoft is rolling out an update for Copilot Vision to Windows Insiders, enabling the AI tool to scan an entire screen. This expands its previous capability of analyzing two applications concurrently, allowing it to view the full desktop or specific browser/app windows. Unlike the Recall feature, which automatically captures periodic snapshots of a computer screen, Copilot Vision operates similarly to screen sharing in a video call. Users activate the feature by selecting a glasses icon within the Copilot application and choosing the desired desktop for the AI to view. This interaction requires direct user initiation. Microsoft states that Copilot Vision is designed to "help analyze content, provide insights, and answer your questions, coaching you through it aloud." The company has provided examples of its potential applications, including offering guidance on improving creative projects, assisting with resume enhancements, and providing support for navigating new video games. This functionality aims to offer real-time, context-aware assistance. Microsoft initiated testing for Copilot Vision last year. Initially, the feature focused on analyzing content viewed while browsing the web within the Edge browser. Furthermore, Copilot Vision possesses the capability to respond to user inquiries based on visual information captured through a mobile phone's camera, extending its utility beyond desktop environments.

Paranoid about AI? You won't like new Windows 11 feature that lets Copilot peer at your desktop

As an Amazon Associate, we earn from qualifying purchases. TweakTown may also earn commissions from other affiliate partners at no extra cost to you. Windows 11 is now giving people the ability to let Copilot peer into their desktop and help them with whatever task might be in progress, if only in testing for now. Sharing the desktop with Copilot (Image Credit: Microsoft) In a blog post, Microsoft announced that the Copilot app is getting this new ability via an update from the Microsoft Store for Windows Insiders. The new feature is called 'Desktop Share' for Copilot Vision, and it allows you to share your desktop with the AI, or a single browser or app window if you prefer. Copilot can then check out the content you're sharing and provide analysis, help, or answers to any questions you might have. As Microsoft puts it: "Get tips on making improvements to your creative project, help with improving your resume, or guidance while navigating a new game." You can stop sharing at the click of a button, though doubtless there are Windows 11 users out there who aren't going to want to start sharing in the first place, and may have trust issues with letting an AI loose on their desktop. Still, if you don't want to share your desktop via Copilot Vision, you don't have to, and you can steer well clear. The new version of the Copilot app carrying this ability, v1.25071.125, has started rolling out to testers right now, so if you're a Windows Insider, you shouldn't have long to wait if you're keen to give this a whirl. There's another new piece of functionality introduced, too, which is the chance to turn on Copilot Vision from a voice conversation with the AI (by clicking the glasses icon).

Copilot Vision Can Now See Your Entire Desktop and Assist With Tasks

Copilot Vision was rolled out to Windows users in the US in June Microsoft announced two new upgrades for Copilot Vision on Tuesday, which are aimed at increasing its usability. The Redmond-based tech giant is introducing a new Desktop Share feature that will allow the artificial intelligence (AI) chatbot to analyse the user's entire desktop and all the active apps, and assist users with various tasks. Additionally, Copilot Vision can now also be activated when the user is interacting with the AI via voice conversations. Both of these new features are now being rolled out to Windows Insiders. In a blog post, the Windows maker announced two new features for Copilot Vision with the latest update to the app. The Copilot app's version 1.25071.125 and newer are now being rolled out to all Insider channels via the Microsoft Store. The company said the update will first be available to Insiders in regions where Windows Vision is enabled. This means non-European countries might have to wait a little before receiving the new features. Additionally, Microsoft stated that the update is being shipped in a phased manner, so it might take a few days before all Insiders see it. Desktop Share is a major upgrade to Copilot Vision, which was only able to access up to two apps at a time so far. To activate this mode, users will have to click the glasses icon in Composer, then select the desktop they would like to share with the AI, and then ask Copilot any query they have. To stop sharing, they can click the "Stop" or "X" button in the Composer. With Desktop Share, Copilot will be able to see and analyse the entire content on the screen in real-time. Users can ask queries, and by taking in the context from multiple apps, it will be able to answer and help users. Copilot Vision can also speak the responses, making it easier for users to follow the instructions while they work on the desktop. According to the tech giant, the chatbot can help users with complex tasks such as improving a resume, navigating through a new app or game, and more. Apart from this, Copilot Vision now also supports the voice mode. This means even when a user is interacting with the AI chatbot verbally, they can activate vision mode without having to switch to text mode. To activate Copilot Vision while on voice mode, users can click on the glasses icon and select the desktop they want to share.

Windows 11's Copilot Vision AI Now Sees Everything on Your Screen

Earlier, it could only see tabs from Microsoft Edge, but now, it can see all apps and interact using voice. Microsoft is finally rolling out AI-powered Copilot Vision for all apps on Windows 11, but it's currently limited to Insiders only. Unlike the earlier version of Copilot Vision, which allowed users to share their browsing tabs from the Edge browser, now, users can share their whole screen, including the Windows 11 desktop and all apps. You can use the real-time Vision capability to interact with Copilot using voice. It can be helpful while shopping for new items, asking questions along the way, or while studying. Copilot can see the screen, analyze the visual content using AI, and respond in real-time, explaining things for you and acting like an AI assistant that can see and hear you. Desktop sharing was first announced in April during Microsoft's 50th anniversary event, where the company unveiled several new Copilot features. Finally, the upgraded Copilot Vision is rolling out on Windows 11. Note that the feature is free to use and doesn't require a Copilot Pro subscription. You can already use the Copilot Vision feature on Android and iPhone and share the camera as well. To use the Copilot Vision desktop share feature on Windows 11, you must join the Windows Insider program and move to one of the channels. Now, update the Copilot app via the Microsoft Store to version 1.25071.125 or higher. You should see the glasses icon beside the microphone. Click on it to share the whole screen and start your conversation. I am on the Windows 11 Release Preview channel and have already received the new update, which means it's rolling out broadly. That said, Microsoft says in its blog post that the feature is rolling out gradually, so not all Insiders will get the update right away.

Microsoft Upgrades Copilot with Real-Time Desktop Sharing Feature

Microsoft's latest upgrade to Copilot Vision is a significant advancement for real-time AI-powered assistance. This new capability enables the AI to view on-screen content, applications, and workflows simultaneously in real-time. The live view will allow Copilot to make suggestions based on context (similar to if a live assistant were beside you), provide troubleshooting help, or find information relevant to the user's display. It transforms an interaction from AI being a conversational agent to a perceptive digital assistant. Also Read:

Microsoft's Copilot AI can now see your entire desktop screen and here's why you should be worried

Privacy experts warn of potential risks if sensitive on-screen data is accessed unintentionally. Microsoft is now rolling out a big update to its Copilot app for Windows, bringing an expanded feature through Copilot Vision. The new update is available via Microsoft Store for version 1.25071.125 and above and allows the users to share their entire desktop or specific windows with Copilot, enabling the real time visual assistance directly on the screen. With this new Desktop Share feature, the Copilot will now see and offer contextual help, insights, and voice-guided support. It can be anything from editing a resume, working on a creative project, or navigating a new video game. It can analyse the content and offer improvement tips, and answer questions based on what's visible on the screen. If you want to activate the feature, you will have to click the glasses icon in the composer and select the desktop or app window they wish to share. Sharing can be stopped at any time by clicking Stop or X. Along with this, Microsoft has also announced that it is introducing the ability to enable Vision directly from a voice conversation. This means, that if users want to give Copilot more visual context while speaking, they can now simply activate screen sharing by selecting the glasses icon mid-conversation. Also read: Using Meta AI? A bug may have exposed your conversations to other users It is worth noting that the update is currently rolling out for the Windows Insider Channels and is currently limited to regions where Copilot Vision is enabled. The company also stated that not all Insiders will see the changes immediately, as the release is staged to ensure stability and performance. Additionally, it is using the preview phase to refine the experience via feedback from early testers. Interestingly, Microsoft has clarified that, unlike the controversial Recall feature, which automatically records screen activity, Copilot Vision functions more like screen sharing during a video call, requires active user consent, and can be turned off at any time. However, the feature raises privacy concerns among both users and security researchers. While intended to increase productivity, it also means that an AI assistant may have access to sensitive information displayed on your desktop, which at times can be scary. It can be your images, icons, folders, your wallpaper or anything.