Microsoft's Copilot Vision: AI Assistant That Can See Your Browser Activity

Microsoft's new "Copilot Vision" AI experiment can see what you browse

On Monday, Microsoft unveiled updates to its consumer AI assistant Copilot, introducing two new experimental features for a limited group of $20/month Copilot Pro subscribers: Copilot Labs and Copilot Vision. Labs integrates OpenAI's latest o1 "reasoning" model, and Vision allows Copilot to see what you're browsing in Edge. Microsoft says Copilot Labs will serve as a testing ground for Microsoft's latest AI tools before they see wider release. The company describes it as offering "a glimpse into 'work-in-progress' projects." The first feature available in Labs is called "Think Deeper," and it uses step-by-step processing to solve more complex problems than the regular Copilot. Think Deeper is Microsoft's version of OpenAI's new o1-preview and o1-mini AI models, and it has so far rolled out to some Copilot Pro users in Australia, Canada, New Zealand, the UK, and the US. Copilot Vision is an entirely different beast. The new feature aims to give the AI assistant a visual window into what you're doing within the Microsoft Edge browser. When enabled, Copilot can "understand the page you're viewing and answer questions about its content," according to Microsoft. The company positions Copilot Vision as a way to provide more natural interactions and task assistance beyond text-based prompts, but it will likely raise privacy concerns. As a result, Microsoft says that Copilot Vision is entirely opt-in and that no audio, images, text, or conversations from Vision will be stored or used for training. The company is also initially limiting Vision's use to a pre-approved list of websites, blocking it on paywalled and sensitive content. The rollout of these features appears gradual, with Microsoft noting that it wants to balance "pioneering features and a deep sense of responsibility." The company said it will be "listening carefully" to user feedback as it expands access to the new capabilities. Microsoft has not provided a timeline for wider availability of either feature. Mustafa Suleyman, chief executive of Microsoft AI, told Reuters that he sees Copilot as an "ever-present confidant" that could potentially learn from users' various Microsoft-connected devices and documents, with permission. He also mentioned that Microsoft co-founder Bill Gates has shown particular interest in Copilot's potential to read and parse emails. But judging by the visceral reaction to Microsoft's Recall feature, which keeps a record of everything you do on your PC so an AI model can recall it later, privacy-sensitive users may not appreciate having an AI assistant monitor their activities -- especially if those features send user data to the cloud for processing.

Microsoft Copilot can now see what you see on the internet | Digital Trends

You might want to start treating your web browser like you're always at work, at least if you want to use Microsoft's new Copilot Vision feature. The feature, which is natively built into Microsoft Edge, is able to "see what you see, and hear what you hear" as you navigate your browser, according to Microsoft's Executive Vice President Yusuf Mehdi. All of this AI snooping isn't for nothing. Copilot Vision looks at what you're doing online to answer questions, provide recommendations, and summarize content. It can work with the new Copilot Voice feature, for example. Microsoft demoed the capabilities on Rotten Tomatoes, showing a user chatting with Copilot while browsing the website and looking for movie recommendations. Ultimately, Copilot settled on an Australian comedy for the Australian speaker, saying it made the choice because, "well, you're Australian." I guess that's taking personal context into account. Recommended Videos Understandably, the idea of an AI looking at everything you do online might rub some people the wrong way. Microsoft says the feature is opt-in only, meaning you have to manually turn it on, and that there will be an indicator onscreen showing that Copilot is watching what you're doing. Microsoft also says that data from your session is deleted after you close the session, and that the data won't be used for training. In addition, Microsoft says that Copilot Vision will only initially work with a limited number of websites. We don't have the full list yet, but at the very least, the feature won't work on "paywalled or sensitive content." Microsoft also points out that the feature doesn't process anything on a webpage -- it just sees the text and images on a page and analyzes them. Copilot Voice works in conjunction with Copilot Vision, but it also works on its own. Across desktop and mobile devices, Microsoft says you'll now be able to hold a conversation with Copilot using just your voice. Building on the new voice capabilities, Microsoft is releasing Copilot Daily. It's almost like a personalized morning news show, rounding up stories and giving you daily details like traffic and weather. This, too, is built with a generative AI voice that walks you through your morning news every day. It's generated based around the content you interact with. Copilot Voice and Copilot Daily are rolling out to Windows users now in the U.S. and UK. Voice is also available in Australia, Canada, and New Zealand, but only in English. Microsoft says it will bring the feature to more countries and languages soon. Copilot Vision is starting to roll out today in the U.S., but only for Copilot Pro subscribers. The feature is part of Microsoft's new Copilot Labs program, which gives Copilot Pro subscribers early access to Microsoft's latest AI features that are still being tested.

Copilot Vision Peeks at Your Edge Browser Sessions to Answer Questions

Copilot Vision is a user-activated feature that will show Copilot your Edge browser's screen to help answer questions or have a conversation. When viewing your Edge tabs, Copilot Vision won't engage with the web for you; it's not designed to post Reddit comments or add posts to your Instagram grid (though it could generate some ideas). Instead, Vision is in read-only mode but can still generate audio responses about what it's seeing on your screen. It will only be allowed to engage with a specific list of pre-approved websites to start, though Microsoft plans to add more in the future. Vision is not supposed to help you get around website paywalls or engage with sensitive content (like porn), however. If you ask Vision to respond to a paywalled or restricted site, it's designed to ignore or reject the request. Still, Vision is a big step toward having an AI assistant that can respond to what you're seeing and effectively follow you around the web, while providing context and hopefully not too many AI hallucinations along the way. Vision has been designed to answer questions for you using the internet. Sure, you could look up answers yourself pretty quickly. But Copilot Vision can view photos or read text and pull data from the web to help you quickly fill in the gaps about a cooking recipe or tell you about a location in an image, according to a Microsoft demo video viewed by PCMag. It can also tell you more about movies and TV shows listed on sites like Rotten Tomatoes to help you find something to watch this month without having to do the research yourself. While some users might show Copilot Vision some personal data, Microsoft promises "absolutely none" of the content shown to Copilot Vision will be stored by Microsoft or used for AI training. Within Edge, Copilot will also generate "daily briefings," or audio rundowns of news you might have missed that morning. Microsoft's also testing a "Think Deeper" feature, which is intended to force Copilot to take its time and provide a more substantial, analytical response to questions. Copilot Vision and Think Deeper are first being rolled out to those in Copilot Labs, a new Windows Insider-like program for Copilot Pro subscribers also announced this week. Microsoft says the Copilot Labs group will be a "small subset" of users who can give Microsoft feedback on its upcoming AI tools. To access Copilot within Edge, type @copilot into your address bar. Then, if you have access to Vision, navigate to the lower pop-up Copilot menu and choose the square icon on the right of the Copilot chat box. From there, select "Continue." You can then speak into your microphone to have conversations with Copilot Vision.