Microsoft's Copilot Vision: AI Assistant That Can See Your Browser Activity

Microsoft Unveils Copilot Vision and Labs

Microsoft has introduced two new experimental features for a limited group of Copilot Pro subscribers: Copilot Vision and Copilot Labs. These additions aim to enhance the AI assistant's capabilities and provide users with more interactive and context-aware experiences 1

1

.

Copilot Vision: A Window into Your Browsing

Copilot Vision is designed to give the AI assistant visual access to what users are viewing within the Microsoft Edge browser. When enabled, it can understand the content of web pages and answer questions about them 1

1

. This feature aims to provide more natural interactions and task assistance beyond text-based prompts 2

2

.

How Copilot Vision Works

The feature allows Copilot to "see what you see, and hear what you hear" as you navigate your browser, according to Microsoft's Executive Vice President Yusuf Mehdi 2

2

. It can analyze text and images on web pages to answer questions, provide recommendations, and summarize content 3

3

.

Privacy Concerns and Limitations

Recognizing potential privacy concerns, Microsoft has implemented several safeguards:

1. Opt-in only: Users must manually activate the feature 2

   2

   .
2. Visual indicator: An on-screen indicator shows when Copilot is active 2

   2

   .
3. Data deletion: Session data is deleted after closing and not used for training 2

   2

   .
4. Limited website access: Initially, Copilot Vision will only work with a pre-approved list of websites 1

   1

   .

Copilot Labs and Think Deeper

Copilot Labs serves as a testing ground for Microsoft's latest AI tools. The first feature available in Labs is "Think Deeper," which uses step-by-step processing to solve more complex problems 1

1

. It's based on OpenAI's new o1-preview and o1-mini AI models 3

3

.

Additional Features

1. Copilot Voice: Allows voice-based conversations with Copilot across desktop and mobile devices 2

   2

   .
2. Copilot Daily: Provides personalized morning news briefings using generative AI 2

   2

   .

Availability and Rollout

Copilot Vision and Labs are currently available to a small subset of Copilot Pro subscribers in select countries. Microsoft plans to expand access based on user feedback and further testing 1

1

3

3

.