Microsoft's Copilot Vision: A New Era of AI Assistance in Windows 11

Windows 11's Copilot Vision wants to help you learn to use complicated apps

Some elements of Microsoft's Copilot assistant in Windows 11 have felt like a solution in search of a problem -- and it hasn't helped that Microsoft has frequently changed Copilot's capabilities, turning it from a native Windows app into a web app and back again. But I find myself intrigued by a new addition to Copilot Vision that Microsoft began rolling out this week to testers in its Windows Insider program. Copilot Vision launched late last year as a feature that could look at pages in the Microsoft Edge browser and answer questions based on those pages' contents. The new Vision update extends that capability to any app window, allowing you to ask Copilot not just about the contents of a document but also about the user interface of the app itself. Provided the app works as intended -- not a given for any software, but especially for AI features -- Copilot Vision could replace "frantic Googling" as a way to learn how to use a new app or how to do something new or obscure in complex PC apps like Word, Excel, or Photoshop. I recently switched from Photoshop to Affinity Photo, for example, and I'm still finding myself tripped up by small differences in workflows and UI between the two apps. Copilot Vision could, in theory, ease that sort of transition. If you want to use Copilot Vision, you need to share your app window with Copilot -- not just the interface but all the content within the app window. Because Copilot Vision uses cloud processing to work (it's not a Copilot+ feature that can run locally on a recent PC), that means sharing this information with Microsoft. In an earlier post about Copilot Vision, Microsoft attempted to address privacy concerns. The company says that "all data about what you say and the context you share with Copilot is deleted" once a Vision session is ended, although Copilot's output is recorded "to improve [Microsoft's] safety systems." Data collected and stored is subject to Microsoft's Privacy Statement. The new Copilot update also adds new file searching capabilities. including the ability to read contents in some files from within the Copilot window without actually having to open the files. Signing up for the Windows Insider program requires a Microsoft account and requires you to share more diagnostic information from your PC with Microsoft.

Microsoft starts testing Copilot Vision update that can "see" your screen and apps

Tom Warren is a senior editor and author of Notepad, who has been covering all things Microsoft, PC, and tech for over 20 years. Microsoft has started testing a new update to its Copilot app on Windows that will let you share your screen or apps with the AI assistant. Copilot Vision was originally limited to Microsoft's Edge browser, but it's now extending to any app on your PC. Copilot Vision will be able to do things like coach you through using Adobe Photoshop features, or analyze the photos and webpages you're looking at. I got to see an early version of Copilot Vision on Windows at Microsoft's 50th anniversary party last week, where the AI assistant guided me through a game of Minecraft and helped to optimize settings in Microsoft's Clipchamp video editor. I haven't been able to fully test Copilot Vision on Windows through the Insider beta because Microsoft is limiting the experience to US testers only. Copilot will be able to highlight parts of your screen to guide you through apps, although this initial beta version won't have the feature enabled just yet. Copilot Vision might sound similar to Microsoft's Recall feature that automatically takes snapshots if you allow it, but it's actually more like screen sharing an app or your entire desktop in a Microsoft Teams call. Microsoft has also started testing file search in Copilot on Windows, allowing you to ask the AI assistant about the contents of a file on your PC. File search supports .docx, .xlsx, .pptx, .txt, .pdf, .json files, and you can use Copilot to easily find the documents you were working on recently. Both of these Copilot features just require the Copilot app on Windows, and not a full Copilot Plus PC. You can also use Copilot Vision on iOS and Android. Microsoft has started testing these new Copilot features with Windows Insiders, ahead of a broader rollout to all Windows 11 users in the coming weeks or months.

Windows 11 tests sharing apps screen and files with Copilot AI

Copilot on Windows 11 is testing OS-level integration that would allow you to share your favourite apps' screen with Copilot. This feature is rolling out to Windows Insiders in the United States, and it might sound like a creepy idea because it basically means Copilot can see what you are doing on your screen -- if you choose to share it with the AI app. However, Microsoft says the integration has some nice use cases. For example, you might want to share your screen with Copilot Vision because it can help you navigate the app if you're struggling with it. When you share your screen with AI, Copilot can then help analyze, offer insights, answer your questions, and speak to you in real time. If you're in the Settings app and unable to find a specific option, Copilot Vision can show you where to locate that particular toggle in real time. It's unclear how Copilot Vision processes your screen data, but remember it's cloud-based, and Microsoft is not relying on local AI models for this integration. If you're a Windows Insider, you can try this feature by clicking the glasses icon in the Copilot app, selecting the browser or app you want to share the screen with, and then asking Copilot to help you out. You can stop sharing your screen by clicking the Stop button or 'X' in the Copilot app. The second new feature is called "File Search", which allows you to find a particular file on your device and ask questions about its contents. You can use everyday language to ask Copilot questions like "Can you find my resume?" and Copilot will find the resume.docx for you by scanning your PC's storage. In a blog post, Microsoft noted that Copilot on Windows 11 can already read .docx, .xlsx, .pptx, .txt, .pdf, and .json files. At the moment, Copilot Vision (screen share) and File Search features are gradually rolling out in the Windows Insider Program. As I mentioned at the outset, Microsoft hasn't clarified how the data is being processed or encrypted, but we'll likely learn more about the privacy policies when the feature exits the testing program.

Windows Copilot aims to be your best friend forever, but can you trust it?

If you're someone who was upset about Windows Recall, look away now. Microsoft Copilot is more than just a tool-it's designed to learn your preferences and tastes, and even anticipate your needs by buying stuff. Whew! That's a big concept to try and get your head around. But that's what Microsoft is trying to do, as it transitions to the next stages of its Copilot assistant. We've used the term "assistant" rather loosely over the past few years. In part, that's due to the aspirational marketing of companies like Google and its "Google Assistant," which has provided answers to spoken questions and not much more. And we will tolerate those machines anticipating some of our preferences: autofilling commands, suggesting movies, and even automatically ordering refills as we run out of ink or milk. Knowing that we typically order orange chicken over chow mein? OK, fine. Microsoft's, um, thinking different. It wants to proactively offer you reminders and suggestions. How? By assigning a Copilot a "memory": your nephew's birthday and what gifts he might like, the foods you enjoy, and what movies you like. It all sounds very much like an advertising profile, and that's probably what it is. And if Microsoft's Copilot starts to recognize you as a person, Microsoft hopes you might begin to think of Copilot as a unique individual, too. "As we explore the full range of what true personalization means, we are experimenting with new ways for you to interact with your Copilot," Microsoft said. "For example, what if your Copilot had an appearance and you could shape and form it to be whatever you want? We're early in this thinking, but soon you'll have the ability to personalize Copilot and interact with your AI companion in a fun way while it offers advice and support when you want it. You can strike up a conversation with Copilot and now have an entertaining appearance that's unique to you." Creepy? Well, let's put it this way: someone will use that term. But there's more. Microsoft would like Copilot to finally realize its agentic aspirations. Remember how ChatGPT was supposed to search out an open table at your favorite restaurant? Through partnerships with 1-800-Flowers.com, Booking.com, Expedia, Kayak, OpenTable, Priceline, Tripadvisor, Skyscanner, Viator, and Vrbo. Microsoft wants to make that a reality. Not only will it know what flowers your wife likes, but it could remember her birthday and order some, too. "Use simple chat prompts to ask Copilot to book event tickets, grab dinner reservations, or send a thoughtful gift to a friend, and it will check that task off your list," Microsoft said in a blog post. It's unclear, however, how much Microsoft will allow Copilot to actually make transactions on your behalf. In the demonstrations Microsoft showed, the user always made the final purchase. Whether that will end up being the case is anyone's guess. Microsoft does already have some shopping skills, however, since it has the ability to track prices and alert you during a key sale. Those sorts of agentic actions would be handy during the closing seconds of an eBay bidding war. For some users, none of this will matter. Microsoft took pains to reassure users that they could control which types of information Copilot remembers about them and even opt out entirely. But anyone who's simply clicked and clicked and clicked again through the dozens of screens that make up the setup process in a Windows PC probably knows that it's rather easy to simply hurriedly agree to Microsoft slurping up your personal data. Those users will probably remember Recall, the Microsoft indexing tool that constantly snapped photos of your desktop before archiving them and allowing you to search the results. (Recall was revamped and then re-released for Copilot+ PCs.) Even though Recall and the updated Copilot are two different things, some users are sure to see the updated Copilot as additional evidence of Microsoft trying to intrude in your privacy, once again. "Every day we hear the most incredible stories about the many and varied ways people enjoy Copilot," Mustafa Suleyman said. "So, as we continue on our journey to create a Copilot for everyone, we're focusing on one critical goal: to make it truly yours. Each will have its own unique style and blend of attributes that distinctly suit each and every one of us. All the while, we'll stick to our core promise. You remain in control; you are the pilot, and you make the calls and set the boundaries." That sounds promising, but your memory probably goes back a year or two when Microsoft debuted Bing Search, then Copilot, and weathered a firestorm of highly personable Microsoft chatbots doing strange things like inquiring about the status of a person's marriage. Now Copilot is about as bland as it can be. So can the new Copilot walk the line between a companionable assistant and one that tries just a little too hard to be your friend? History says no. But Microsoft sounds like it wants to take a shot at it. Hold on tight.

I didn't care about Copilot, but this massive upgrade could make Microsoft's AI the personal assistant I've always wanted

Let me preface this by saying, that when I say I don't care about Copilot, I mean it. I've not owned a Windows PC since 2006, and I've never even installed the Copilot app on my Mac, iPhone, or even Android device. I write about AI for a living, but there are very few AI tools I actually use in my daily life. Usually, I test them for work, come up with article ideas, and write about experiences with AI chatbots like ChatGPT and Gemini - but I rarely use AI in my spare time. Microsoft's Copilot is an AI tool that I've neglected, not only in my home life but also at work. I rarely cover Copilot, and when I do it's usually just a brief news story here or there. Anyways, you get the idea Copilot just isn't really an AI chatbot that was on my radar. But after Microsoft's 50th Anniversary Copilot Event, that's all changed. Copilot is often seen as a business tool rather than a consumer product. So much so, in fact, that I fully expected Microsoft's Copilot event to center around business use and barely touch on what that means for a consumer. I couldn't have been more wrong. At Microsoft's event today, the company showcased a new vision for Copilot, which it now calls 'Your AI Companion', and I think it's time I take notice. Today, in celebration of its 50th anniversary, Microsoft highlighted that the company's future is very much counting on software like Copilot, and that's a very exciting thing. The company announced major upgrades to the AI chatbot including a memory, similar to that found in Gemini and ChatGPT, meaning Copilot will learn important information about you. Microsoft says Copilot can now "learn who you are deeply." In a blog post announcing the new features, CEO of Microsoft AI Mustafa Suleyman said, "With your permission, Copilot will now remember what you talk about, so it learns your likes and dislikes and details about your life: the name of your dog, that tricky project at work, what keeps you motivated to stick to your new workout routine." Elsewhere Copilot is getting vision, giving the AI on-screen awareness in Windows and access to your camera in mobile apps similar to Apple's Visual Intelligence or Google's Lens. Microsoft announced much more at today's event, and you can read about it all in detail here. It's the agentic capabilities called "Actions", however, that really stand out and make me interested in giving Copilot a shot at becoming my daily AI driver. My dream AI scenario is a personal assistant that does the mundane in my life so I can focus on the things that matter. Whether it's filling out calendar entries, telling me when to leave for an appointment, or booking my next beard trim, I want AI to alleviate a little bit of the stress of life so I have one less thing to think about. Apple Intelligence-powered Siri looked like my ideal scenario when it was showcased at WWDC 2024, but with WWDC 2025 fast approaching we've still not seen a tech demo to prove it's actually capable of what's advertised. Today, Microsoft's Copilot announcement with an emphasis on an AI companion piques my interest and makes me wonder if the company to fulfill my AI want will ultimately be the Seattle tech giant. It's hard to know for sure, considering I've not tested any of the newly announced Copilot features yet. But you better believe the first thing I'm going to do as soon as Copilot's update is available, is use it in my daily life. I want to leave you with another quote from the CEO of Microsoft AI. He said, "The important point is that throughout Copilot is more than an AI, it's yours. It remembers not just what you said, but who you are. Copilot helps you stay organized, think clearly, and learn more intuitively." I've been burnt before by putting trust in AI marketing campaigns but Microsoft's Copilot announcements today make me excited for the future, and I can't wait to see if it can live up to the billing.

Microsoft Copilot is putting eyes on your screen, and I don't mind it - as long as it stays private

Microsoft followed at least one part of my wishlist for its Copilot event when it announced Copilot Vision will be embedded in Windows 11 and the mobile app. Copilot Vision will let the AI actually see what's on your screen and the world around you. Microsoft first toyed with the idea in Edge, where Copilot could peek at the browser and offer help. But now it's going system-wide, across apps, websites, and even what your smartphone camera shows on your screen. The idea is to offer a proactive AI assistant that can scan an Excel sheet, summarize a PDF, suggest how to adjust your settings, or identify the wild berry you're not sure is edible or poisonous. The appeal is obvious, even when compared to tools with similar features like ChatGPT and Gemini. Those tools can be great on a desktop or laptop, but they don't have the luxury of being built directly into your operating system. Copilot has a home-field advantage on Windows 11. Vision leverages that advantage. It doesn't have to wait for a prompt; it can see what you're doing and offer context-relevant help. Microsoft clearly anticipates some trepidation over Copilot Vision's expansion based on how eager the company is to highlight the feature's privacy aspects. Microsoft says Copilot Vision only works when you grant access. Still, there's no denying the privacy weirdness. The digital equivalent of someone looking over your shoulder sounds creepy out of context. Of course, it's not like we haven't all been handing over swaths of data to the cloud to get slightly better movie recommendations or similar benefits. But I can understand why letting AI literally read your screen as you do the same might feel intimate in a disconcerting way. Control and trust might make the difference in Copilot Vision's success. If Microsoft starts making Copilot Vision default-on or hides the permission settings in some deeply buried system menu, more people might turn away from Copilot entirely and turn to another option with the same abilities but with the user controlling it more. I still think it will be a genuinely useful addition to Copilot, especially on the surprisingly well-designed Copilot mobile app. You'll want to keep track of what permissions you give and when Copilot Vision is active. I'd encourage trying it out, but keep your eyes open when the AI is doing the same.

Microsoft could soon let Copilot see your entire screen, but that's not a bad thing

Microsoft is testing a new feature on Copilot, and this time, it's coming to PC. Copilot Vision, recently made available on mobile, is now available for testing on PC, and the capabilities are both awe-inspiring and a little scary. Microsoft is also giving file search an AI boost. Copilot Vision will allow the app to see everything on your screen -- provided you'll want it to. You can share browser windows, your entire desktop, or even a game to get real-time feedback from Copilot. Upon launch, you'll choose which browser window or app you want to share with Copilot, and from then on, you'll be able to speak to Copilot the way you would to a tech support agent. You can stop sharing at any given time. Microsoft shared a screenshot of the feature, showing a Word file being shared with the app. You can talk to Copilot Vision by voice or by text, and the app is said to guide you through the process of whatever you're trying to do. Recommended Videos The Verge showed off a demo of Copilot Vision that was made available for testing during Microsoft's 50th anniversary party. You could see Copilot respond to questions during a game of Minecraft, giving the user advice on what to do with certain items. It also guided the user through video editing in Clipchamp, complete with highlighting something on their screen. That last part, the highlighting, isn't coming in this beta build yet, but it will be available eventually. Microsoft is also making file search more natural in Windows. The Copilot app will soon be able to help you find files on your PC. You can ask questions such as "Can you find my resume", and Copilot will know what to look for. Fortunately, you can adjust the permissions for what Copilot is and isn't allowed to see. According to the announcement on Microsoft's Windows blog, the features are rolling out to Windows Insiders in the United States. This means that we won't all be able to try them out, and it might take a while for Copilot Vision to be rolled out to a broader audience.

Holy smokes, something that might actually be useful is coming to Copilot on Windows

Forgive me for being surprised by this after the last two years of REM sleep-inducing AI updates and betas, but Microsoft's latest Copilot update seems like it might (*drumroll*) actually be useful. That's because it should finally allow you to search system-wide for contents within files and also help you in any browser or app. The latest Windows Insider (ie, beta) build is now rolling out these Vision and file search features via a Copilot app update in the Microsoft Store (for "version 1.25034.133.0 and higher"). These two features feel like the kinds of things we were promised from Copilot upon its launch but which we've seen little of until now. Better late than never, though, eh? File search, Microsoft says, allows you to "find, open and ask questions about the contents of a file on your device from the Copilot on Windows app" and supports most file types. It also looks like it should be good at contextual commands and requests -- one example Microsoft uses is, "Look at my budget file and tell me how much I spent on dining last month." Vision is a feature that finally opens up the playing field for Copilot, giving it access to any app or browser you choose: "To get started click the glasses icon in your composer, select which browser window or app you want to share, and ask Copilot to help with whatever you're working on." Apparently, "Copilot can then help analyze, offer insights, or answer your questions, coaching you through it aloud." We will, of course, have to see just how good it is at such tasks in practice. Both of these features excite me more than previous Copilot ones. As someone with a penchant for creating the most disorganised digital file systems known to man, I can see genuine use in file search. And I suppose Vision could be useful for learning new apps and tools, provided it actually does what it says on the tin. They're certainly more exciting Copilot features than the usual text summaries and email drafts *yawn*. And as exciting as 'gaming sidekick' Copilot for Gaming might be, game assistants are hardly a Microsoft exclusive. Vision and file search, on the other hand, could be genuine benefits to the Copilot bundle that other companies can't easily match, given these will presumably be baked into -- and make use of -- the Windows operating system itself. I just hope having a full-blown AI 'companion' (per head of Microsoft's AI section, Mustafa Suleyman), isn't required for it. Let's not ruin a good thing, for once, okay?

How to Try Copilot's New 'Vision' and File Search Updates

Copilot can now see more of what you're doing on your PC, if you sign up for the Windows Insider program. Microsoft announced a plethora of Copilot upgrades at the start of the week, and we're now starting to see some of them roll out -- to Windows Insiders, at least. The Copilot app for Windows has now been updated to support Copilot Vision and file search, so the AI bot is better able to interact with whatever's on your PC. Copilot Vision first appeared on the web last year, but it's new to the Windows app: It essentially lets the AI "see" what's happening on your computer, so you can ask it about websites you're viewing, or documents you're editing, or songs you're listening to, without having to take a screenshot or do any copying or pasting. It's somewhat similar to the Windows Recall feature that remains in limited testing, and which also gives the operating system more access to what's on your screen. This does of course raise a few worries around privacy, but Microsoft promises users stay in full control of what's accessible, and that all data is deleted after every Copilot Vision session. Microsoft imagines users taking advantage of Copilot Vision to chat through plans for a trip, or for help when shopping online. As the AI can see what you're seeing, it's able to pick out ideas and give feedback on what you're talking about, with interactions primarily handled via voice. Then there's file search, which means Copilot can find, open, and chat about files stored locally on your system. You might want to look for your résumé, for example, or bring up a slideshow you were working on last month -- all these requests can be handled using natural language queries. You can even ask about the contents of your files -- so you could, for example, get Copilot to pull out spending figures from a spreadsheet. Again, Microsoft says you'll be in full control over what Copilot can and can't access in terms of your files: Everything is going to be handled through the permissions configured in Copilot's settings. Eventually, these features will appear in the stable version of Windows for all users, though that may take a while (Microsoft hasn't announced an exact timetable). In the meantime, you can test these new Copilot features early, if you're prepared to sign up for the Windows Insider Program. You can do that for free here. The Windows Insider Program is like any other beta testing program: You get access to new features earlier than everyone else, but there are likely to be a few more bugs that you come across. It's generally not recommended to run Insider versions of Windows on a laptop or desktop that's critical to your day-to-day computing. Once you're up and running as a Windows Insider, you'll find you can sign up for the beta channel of Windows 11 through the Windows Update part of Settings in Windows. Shortly after, you'll get a new download, which will install the beta version on your computer (you can switch back to the stable version any time you like). With all that taken care of, you need the latest version of Copilot on your computer: Look for version 1.25034.133.0 or later on the Microsoft Store. Microsoft says these features will come to Insiders in the U.S. first, then be available to everyone else after that -- with the features "gradually rolling out" over time. Copilot Vision can be launched by clicking the glasses icon to the right of the main input box inside Copilot for Windows. You're then prompted to indicate which app or browser window you want to share with Copilot, and you can then chat with the bot using your voice. Click the Stop or X buttons to end a session. File search is built right into the Copilot app for Windows, so you can simply type out or speak out a query as you would in any other chat -- something like "what were the places I shortlisted in my vacation plan document last week?" for example. As always with AI, I'd recommend double-checking the answers you get for anything important.

You'll Actually Want to Use Copilot on Windows Thanks to These Search Upgrades

Microsoft is adding two new features to Copilot in the latest Windows Insider Update. This makes it more useful for searching through your files and explaining what you see. Copilot Can Now Read Your Files to Help You Search Through Them The first big improvement is in File search, where the latest update allows it to have access to everything in your PC's storage. According to the Windows Insider blog, you will still have ultimate control over your files, as it lets you adjust permissions on what the AI can access, retrieve, or read. Copilot should be able to see almost all file types when searching through your documents, and it can also read the contents of .docx, .xlsx, .pptx, .txt, .pdf, and .json files. Because of this, you can ask file-related questions and give it commands -- like "Open the article I was working on yesterday," "Find the folder where I saved my latest bank statement," or "Summarize the PDF file I downloaded earlier today." This will make it much easier for most users to find files without having to deal with the clunky Windows Search. This should go hand-in-hand with the native language search feature Windows recently received, meaning you don't have to lose files if you misplace them. Let the AI See Your Screen and Guide You It's also introducing a second feature called Copilot Vision on Windows. This allows the AI app to see your browser or app window. From there, it can do the following: Analyze your screen Offer insights, like describing what it sees Answer questions about what it sees, and then guide you through it You can turn on Copilot Vision by hovering over the Copilot on the taskbar and clicking on the glasses icon when the hover menu appears. From there, you can select which app you want it to see. Once you want to end sharing your screen with Copilot, you can just click on 'Stop' or 'X', and it will end the session. Tech enthusiasts will likely be the first to use this feature, but this also has the potential to help those who aren't adept with apps and programs to figure things out. For example, if a user does not know how to insert a table into a Word file, they can just ask Copilot Vision on Windows to guide them through with step-by-step on-screen instructions. This has the potential to become a controversial feature, though, similar to how Windows Recall was initially considered a privacy nightmare. But because it's not on all the time, you have control over when it will run and what it will see. It's Coming to Windows Insiders First At the moment, these two features are available to Windows Insiders running version 1.25034.133.0. Microsoft also added that while File search is available to all Insiders worldwide, Copilot Vision on Windows is limited to US users only. It will release the features gradually, so you might not get it immediately if you're an Insider and update Copilot. The company likely did this so it can test how these features will run when used by the public. That way, if it runs into issues, it can fix them before they go to general release. Once they're available to everyone, then these will surely be part of the underrated Microsoft Copilot features you shouldn't miss. The only question we have left is whether this would require an NPU to run or not. The company hasn't said anything about it, but if it needs newer hardware, then millions of users who rely on older CPUs will miss out on these capabilities.