Microsoft Confirms Windows 11 AI Agents Hallucinate and Introduce Novel Security Risks

I get why some people are suddenly freaking out about AI agents in Windows 11 - I'm worried, too, but let's not panic just yet

Windows 11 is in the firing line once again, and this time some recent updates to documentation around AI agents have provoked fresh concerns about how these entities will work in the OS - and what threats they might pose. This latest controversy actually stems from an old support document about 'Experimental Agentic Features' (well, not that old - it was published in October 2025), which Microsoft updated a couple of weeks ago. As Windows Latest highlighted (via PC Gamer), the documentation was refreshed when Microsoft rolled out the switch to turn on experimental agentic features in Windows 11 test builds (which happened in preview version 26220.7262 in the Dev and Beta channels). This was when the deployment of the first agent, Copilot Actions, was kicked off in testing (this can work with files on your PC to, say, organize a collection of photos and delete duplicates, for example). The reality of Copilot Actions officially going live seems to have got people looking more closely at the fine print, some parts of which are vaguely ominous, while others are more alarming. Notably highlighted in the media coverage are warnings that AI models "occasionally may hallucinate and produce unexpected outputs", meaning they can get things wrong, or indeed talk absolute garbage. We knew that already, though; it's just the nature of AI - specifically an LLM (large language model) - as a clearly fallible entity. Also flagged is the warning about AI agents potentially introducing "novel security risks, such as cross-prompt injection", and this is where it gets a lot more worrying. Of course, Microsoft has been banging on about these possible new attack vectors that might be leveraged via such AI systems since last year. Which is to say that the systems it has been creating for Windows 11 have very much been built with these threats in mind - so hopefully its defenses are going to be tight enough to deflect any such attempted intrusions. As Windows Latest points out, the way AI agents are boxed away in Windows 11 seems pretty watertight. They live in an 'agent workspace' effectively as a separate local user, with a distinct account completely walled off from the user's account, and limited file access based on permissions granted (aside from a handful of default folders). This should keep these agents contained, and even if compromised, they should theoretically only have limited means of exploiting the system. Of course, the proof will be in the pudding of this system being used in the real world, and the trouble is if we look at the collapsed cake that was Recall - or at least this AI feature's initial design - that doesn't give us much confidence. Of course, Microsoft has learnt from that episode, right? Well, I certainly hope so, and the security planning behind agent workspaces does seem to be suitably thorough, expansive and much more convincing overall. However, as PC Gamer notes, the biggest issue is that when talking about those novel security risks (cross-prompt injections) and potential nastiness that could be leveraged therein, like data exfiltration - stealing your files - Microsoft has added a new caution in its recent revision of this document. Namely that: "We recommend you read through this information and understand the security implications of enabling an agent on your computer." That's the most sinister sentence in this document when it comes to the content relating to security. What is this saying? That this is some sort of get-out clause for Microsoft, and you've got to weigh up the risks on your own by poring through documents? Now, you may think that's reading too much into this, and that's fair enough, but it has certainly sent alarm bells ringing in the articles - and online comments - that are now popping up around this. It certainly doesn't feel very comforting to read that, but then again, this is early testing for AI agents. Copilot Actions is in a purely experimental phase right now, in fact, so another way of looking at this would be: what do you expect? Sign up now and there probably are some very real risks involved. Just imagine you were using an 'experimental' operating system, and it went down in flames, taking your files with it in the ensuing fireball - you'd only have yourself to blame, wouldn't you? So, the message is to proceed at your own risk, which at this experimental stage is fair enough really. However, my actual worry here is when these AI agents come to a full implementation in the finished version of Windows 11, can we trust that Microsoft will have realized that in a watertight way? What if there's a hole in this system somewhere? Given that Microsoft is seemingly breaking even basic things in Windows 11 with some regularity, I can see why folks might be concerned here. I'm nervous, after all, and if something does go wrong, it could be disastrous for the involved users who are running AI agents - and for Microsoft's reputation, too. The software giant can't afford an episode where AI goes rogue in some wild way, as it will be difficult to recover the trust in Windows 11's agents if an unfortunate episode occurs along these lines.

Microsoft confirms that its new AI agent in Windows 11 hallucinates like every other chatbot and poses security risks to users

Hallucinating, hack-prone operating systems are the new normal. Like the rest of the tech world and its LLM-powered pooch, Microsoft has been on a big AI push of late. Its latest achievement in that regard is the rollout of agentic AI capabilities for Windows 11 courtesy of the 26220.7262 update (via Windows Latest). Oh, and with that comes the warning that the new AI features are prone to "hallucinate" and "introduce novel security risks." As to the details, Microsoft says security flaws include "cross-prompt injection (XPIA), where malicious content embedded in UI elements or documents can override agent instructions, leading to unintended actions like data exfiltration or malware installation." In other words, you could download, say, a PDF which contains hidden text instructing your Windows agent to execute nefarious tasks. And it might just carry out those instructions. So, surely Microsoft has some mitigations in place? Up to a point. Firstly and mercifully, these new agentic features are not enabled by default. However, once switched on, they're enabled for all users, all the time. They are at least labelled as "experimental agentic features", and a warning is delivered during the setup process. Microsoft also says the new agentic AI features operate under three core principles. First, "all actions of an agent are observable and distinguishable from those taken by a user." Second, "agents that collect, aggregate or otherwise utilize protected data of users meet or exceed the security and privacy standards of the data which they consume." And third, "Users approve all queries for user data as well as actions taken." However, those principles do not appear to be guarantees, but rather aspirations, hence the security warnings. Microsoft also says, "We recommend you read through this information and understand the security implications of enabling an agent on your computer." But it's hard to see how typical users are meant to understand the security implications. How is one to judge the risk? How likely is a successful security attack that relies on the agentic AI vulnerability to prompt injection? That's surely impossible for most users to "understand." All of which means that Microsoft is, in effect, shunting the responsibility onto users, for now. It's up to them to decide whether to turn these features on and up to them to judge the risks. Of course, AI models hallucinating and being vulnerable to prompt injection attacks is hardly news. Pretty much every major AI suffers from these problems. Heck, even poetry can be used to trick AI. But it is remarkable to observe Microsoft nonchalantly adding a feature with such self-confessed problems to its mainstream and utterly dominant PC operating system. Apparently, it's now completely fine to release a feature with major known flaws and security vulnerabilities. The assumption here is that Microsoft feels the competitive impetus is absolutely overwhelming. If it does not add these features to Windows, it risks being totally overwhelmed by competitors who will. And maybe that's true. But it's still remarkable to see norms around reliability and safety to become comprehensively defenestrated. When it comes to AI, it seems buggy and insecure is the new normal. And that's really weird, isn't it?

Microsoft confirms its Windows 11 AI Agents hallucinate and pose a serious security risk

TL;DR: Microsoft's Windows 11 is evolving into an AI-powered "Agentic OS," featuring background AI Agents with separate accounts that perform tasks via natural language. While enhancing productivity, these experimental features pose significant security risks like data leaks and malware, prompting cautious user supervision and strict privacy controls. Microsoft has proclaimed on multiple occasions that Windows 11 and Windows in general are transforming into an 'Agentic OS,' and the latest 'Experimental Agentic Features' included in a recent Windows 11 preview build offer a first honest look at a Windows PC becoming an AI PC. The quick summary is that AI Agents will have their own accounts and privileges and run in the background while you're using your PC, leading to a situation where multiple users are logged in to your PC, with you being the only human. Basically, you'll be able to interact with your PC using natural language. At the same time, these AI Agents will handle everything from launching office apps and creating charts to browsing, finding a deal, buying a new appliance, and searching through images to find something specific. These agents will run in the background, with Copilot as the primary interface. Microsoft notes that you'll be able to monitor AI Agents like you can apps, while also confirming that these agents are prone to hallucinating and can even be tricked into installing malware or sending sensitive data and files to bad actors, which makes you wonder why anyone would enable these 'Experimental Agentic Features' when Microsoft is adamant that they pose a real security risk. "AI models still face functional limitations in terms of how they behave and occasionally may hallucinate and produce unexpected outputs," Microsoft writes. "Additionally, agentic AI applications introduce novel security risks, such as cross-prompt injection (XPIA), where malicious content embedded in UI elements or documents can override agent instructions, leading to unintended actions like data exfiltration or malware installation." Microsoft's 'Agentic security and privacy principles' outline the company's goals with these agents, stating that even though they're autonomous entities, they'll keep detailed logs of their activities, with the ability for users to supervise them. In addition, access to sensitive information will be "granular, specific and time-bound" and will only occur in "user-authorized contexts." Also, agents aren't allowed to engage with other agents; only the owner is allowed to do so. Basically, it reads like the beginning of a sci-fi story where a rogue AI kickstarts the end of civilization by leaking someone's credit card and passport details to a group of hackers. Even the best-case scenario sounds counterproductive. Instead of jumping onto a PC and clicking on a button and typing, your hands are tied as you try to wrangle a team of stoners and guide them through multiple steps just to send an email.

Microsoft's AI obsession is scaring me

There was a time when I opened Windows and felt in control. Now? I feel like I'm living in Microsoft's experiment lab. Even after four years of launch, Windows 11 still feels like a work in progress. It started innocuously enough. A feature here, an AI integration there. But somewhere along the way, something shifted. Microsoft didn't just add AI to Windows 11 -- it made AI the entire point. And frankly, that scares me for the future of Windows. The AI takeover nobody asked for AI everywhere, whether you want it or not If you've paid attention to the host of updates Microsoft has launched for Windows 11, you'd have noticed that Microsoft has been stuffing Copilot into every corner of Windows. From AI-generated images in Paint to AI autocomplete capabilities in Notepad, it seems like every built-in Windows tool is becoming increasingly AI-centric. Recall watching everything you do on your computer, and the latest push for an Agentic AI that runs in the background all the time is just few examples of how deep AI integration is running in Windows. I don't have a gripe with AI. If done right, it can be a great tool that makes life easy. But your PC is watching everything you do and taking actions autonomously even when you're not looking? That raises some concerns. What bothers me most isn't Microsoft's AI push; it's the way Microsoft is forcing it into the OS without giving users a genuine choice. You can disable Copilot, sure, but you need to know about the Group Policy Editor (which requires a Windows 11 Pro version), registry hacks, or be tech-savvy enough to navigate multiple workarounds and complex tutorials. That's fine for people who're into tech, but a vast majority of Windows users just want to check their email, browse the web, get their work done, and call it a day. Not giving proper settings and options to fully disable these increasingly invasive AI features makes them essentially non-negotiable for the average person. It's just bloatware with machine learning ambitions. Windows is becoming a privacy nightmare Your PC is sending home more data than you think Windows 11 had already upset a lot of privacy-conscious users with its requirements of a Microsoft account and the huge amount of data and telemetry the OS collects. But the AI features bolted on top make the situation much worse. Recall is perhaps the most prominent example of the privacy disaster that these AI features are. Security experts and privacy advocates roasted the feature to a fine crisp when Microsoft first announced it. And for good reason. Despite Microsoft's reassurances that the feature is secure and stores everything locally, it was still relatively easy to exploit. For a feature that takes a screenshot of your screen every five seconds, indexes everything, and stores it, security is an absolute non-negotiable. It can potentially capture your passwords, banking info, sensitive documents, private messages, and anything else you don't want preserved in a searchable database. Microsoft shelved it after the backlash, but guess what? It's back. Microsoft may call it opt-in now, but the implications haven't changed. That searchable data of everything you do on your computer is a privacy disaster waiting to happen. The database is allegedly encrypted with BitLocker, but the moment you log into your Windows account, the encryption is pointless. And if malware gets even temporary access, an attacker can scrape everything you've ever done in seconds. To make it worse, even if you opt out of Recall, but your colleague or family member doesn't, anything they receive from you gets captured on their machine anyway, without your knowledge or consent. You can't control what happens to your information on someone else's device. Reverse Take You should seriously give Microsoft Copilot a chance Microsoft forced this app on me, and I liked it. Posts By Keval Shukla Nov 7, 2025 The broader integration of agentic AI into Windows is opening up entirely new attack vectors as well. Microsoft itself has warned about Cross-Prompt Injection (XPIA) attacks, where malicious content embedded in documents or UI elements can trick Copilot agents into ignoring their instructions and doing what an attacker wants instead. Then there are the cascading hallucinations where the AI generates false or misleading information that stays in its memory and can trigger real-world consequences. An agent could make a wrong API call, pull incorrect regulatory criteria, or pass fabricated information to other systems. The problem is that Copilot has access to everything from your Microsoft 365 data, emails, documents, and communications, and it needs this access to be useful. When it starts taking actions on its own, any mistake can cause real havoc. The OS itself is falling apart Broken core features, performance issues, and a laundry list of user complaints. As Microsoft shoves more AI features into Windows, the rest of the OS seems to be crumbling around itself. Major core features are broken, the Start menu, Taskbar, File Explorer, Windows settings -- all of them have problems related to XAML rendering. These aren't minor glitches; they're fundamental parts of the OS not working as they should years after launch. Even something as basic as Windows Search is still slow. I use three search apps on Windows 11 to fix problems Windows Search still has. Performance is another casualty. Windows 11 comes loaded with bloatware and background processes that consume massive amounts of system resources. It's fine if you've got a powerful PC with enough headroom, but older or budget machines can really struggle to run the OS itself, let alone memory-hungry browsers and apps now that native Windows apps are being replaced with Electron equivalents. Removing bloatware or disabling additional features shouldn't be required, let alone be difficult. If you want to disable Copilot, you can't just uninstall it. If you're on Windows 11 Home, the Group Policy Editor isn't even available. Your best bet is complicated registry edits, PowerShell commands, or just accepting that an AI assistant is baked into your OS and constantly watching over your shoulder. Related I Love Windows, but Can't Ignore These 9 Huge Annoyances in Windows 11 Seriously, Microsoft? Posts 39 By Pankil Shah Feb 19, 2025 And even when you think you've disabled Copilot or found a workaround to a problem, Microsoft is constantly deprecating methods that work. Even the legacy "Turn off Windows Copilot" group policy is being phased out in favor of newer methods that are less accessible to the average user. Windows could be great -- if Microsoft actually listened There's still hope for Windows 11, if only Microsoft can focus on the right things I'm not anti-AI. I use AI tools almost daily, use them to write code, plan budgets, and handle a lot of menial tasks throughout my day. I think there's genuine value in what these systems can do. But there's a difference between offering a tool and forcing it onto users who may or may not want to use it. What worries me most is that Microsoft is pivoting toward a Windows experience where AI isn't a feature, it's the backbone. Where your OS is designed primarily to be an agentic platform that runs AI agents autonomously in the background. Where privacy is baked in as an assumption of risk rather than a right. And between all of this, the OS that you're paying for doesn't get the attention that it does. Instead of fixing broken features and prioritizing user experience, Microsoft is treating Windows like a testing ground for an AI vision that I don't think anyone asked for. And the rest of Windows is suffering for it.