AI Agents Under Siege: New Era of Cybersecurity Threats Emerges as Autonomous Systems Face Sophisticated Attacks

Anthropic Catches Attackers Using Agents In The Act

The internet is rife with prognostications and security vendor hype about AI-powered attacks. On November 13, AI vendor Anthropic published details about the disruption of what it characterized as an AI-led cyber-espionage operation. This revelation comes on the heels of a Google Threat Intelligence Group (GTIG) report which also highlighted the use of AI in attacks. Although the report covers activity in-the-wild, it focuses on malware that uses just-in-time invocation of LLM's for defense evasion and dynamic generation of malicious functions. The Anthropic report describes an altogether different -- and much more sophisticated -- use of AI that borders on being agentic. The release of this information is important because AI vendors are the only parties with sufficient visibility into how adversaries are attempting to leverage AI platforms and models. Ideally, a report such as this would have been mapped to a framework like MITRE ATT&CK but it still provides insights about what defenders may be facing and how adversary capabilities are evolving. Anthropic discusses many campaign details in its report, but the high-level summary is that a threat actor, whom Anthropic assesses with high confidence to be Chinese state-sponsored, targeted around 30 organizations across multiple industry sectors using an AI-driven attack framework employing agents and requiring very little human effort or intervention. Although the campaign made extensive use of agents, it didn't quite rise to the level of being truly agentic. While the operation represents a significant step forward in attackers' use of AI -- with agents allegedly performing 80 - 90% of the work -- humans were still providing direction at critical junctions and there are still limits to what exactly what can be automated. One constraint may be the testing and validation of the output of AI. As the report says, "An important limitation emerged during investigation: Claude frequently overstated findings and occasionally fabricated data during autonomous operations, claiming to have obtained credentials that didn't work or identifying critical discoveries that proved to be publicly available information. This AI hallucination in offensive security contexts presented challenges for the actor's operational effectiveness, requiring careful validation of all claimed results. This remains an obstacle to fully autonomous cyberattacks." Ironically, this means attackers may have to confront the same AI trust issues as defenders. Throughout the report, Anthropic points out that the rate of requests far exceeded "what was humanly possible". In the application security space, organizations have contended with a similar challenge for years: bad bots attempting DDoS, account fraud, web recon and scraping, while disguising themselves by usurping residential proxies and continuously adapting their behavior to evade defenses. Malicious agents and/or hijacked agents will use similar techniques. Bot and agent trust management software analyzes hundreds, sometimes thousands, of signals to determine bot and agent provenance, behavior, and intent to help defend against agents that target organizations through customer facing applications, which are one of the top external attack vectors. This campaign was possible for a few distinct reasons. First, as Anthropic states, its newer frontier models understand more context. In addition to making deliberate misrepresentations about their identity and purpose, attackers broke up the attack into discrete tasks. This enabled them to create a gap between the context necessary for carrying out the attack and the context necessary to "understand" the requested actions as malicious in relation to each other. In Forrester's Agentic AI Enterprise Guardrails for Information Security (AEGIS) framework, we describe this issue as "securing intent" and it is one of the defining capabilities of AI security. Securing intent is not just an issue for LLM vendors, it's also a major priority for any organization building an AI agent and is one of the defining capabilities of AI security. AI is only as effective as its training data; the attacks it produces are not novel. The real value is that, using agents, attacks can be constant, high volume, and eventually automated to not require a human. The capabilities needed to defend against these attacks are many of the same ones we already rely on: focusing on Zero Trust, implementing proactive security, building a strong governance capability, and effectively detecting and responding to attacks. To protect against future AI-enabled attacks, security pros should: While the attack itself used existing exploits and wasn't fully autonomous, it's important to note that this serves as a harbinger of things to come for future attacks using AI and agents. Malicious actors will continue to improve on these capabilities, as they have with past technical advances. Clients who want to explore Forrester's diverse range of AI research further can set up a guidance session or inquiry or contact their account team.

Major AI agents are being spoofed - and it could put your site at risk

Sites must tighten security to protect themselves and users AI comes in many forms, and dominating the tech world right now is AI agents, which are evolving fast, often outpacing the security measures put in place to control them - but that's just one side of the story, as security teams not only have rogue but legitimate agents posing security risks, but also fake agents. New research from Radware reveals these malicious bots disguise themselves as real AI chatbots in agent mode, like ChatGPT, Claude, and Gemini - all 'good bots' that, crucially, require POST request permissions for any transactional capabilities such as booking hotels, purchasing tickets, and completing transactions - all central to their advertised usage. Legitimate agents can interact with web page components like account dashboards, login portals, and checkout processes - which means websites now have to allow POST requests from AI bots in order to accommodate these legitimate agents. The issue here is that previously, a fundamental assumption in cybersecurity was that 'good bots only read, never write'. This weakens security for site owners, as malicious actors can much more easily spoof legitimate agents, as they need the same website permissions. Legitimate AI agent traffic is surging, making it all the more likely that these fraudulent bots will pass through undetected. Most exposed are, of course, the high risk industries; finance, ecommerce, healthcare, and also the ticketing/travel companies AI agents are specifically designed to use. Chatbots all use different identification and verification methods, making it even more difficult for security teams to detect malicious traffic - and easier for threat actors who will just impersonate the agent with the weakest verification standard. Researchers recommend adopting a zero-trust policy for state-changing requests, like implementing AI-resistant challenges like advanced CAPTCHAs. They also recommend treating all user-agents as untrustworthy as standard, and adopting robust DNS and IP-based checks to ensure the IP addresses match the bot's claimed identity.

AI agents open door to new hacking threats

Cybersecurity experts are warning that artificial intelligence agents, widely considered the next frontier in the generative AI revolution, could wind up getting hijacked and doing the dirty work for hackers. AI agents are programs that use artificial intelligence chatbots to do the work humans do online, like buy a plane ticket or add events to a calendar. But the ability to order around AI agents with plain language makes it possible for even the technically non-proficient to do mischief. "We're entering an era where cybersecurity is no longer about protecting users from bad actors with a highly technical skillset," AI startup Perplexity said in a blog post. "For the first time in decades, we're seeing new and novel attack vectors that can come from anywhere." These so-called injection attacks are not new in the hacker world, but previously required cleverly written and concealed computer code to cause damage. But as AI tools evolved from just generating text, images or video to being "agents" that can independently scour the internet, the potential for them to be commandeered by prompts slipped in by hackers has grown. "People need to understand there are specific dangers using AI in the security sense," said software engineer Marti Jorda Roca at NeuralTrust, which specializes in large language model security. Meta calls this query injection threat a "vulnerability." OpenAI chief information security officer Dane Stuckey has referred to it as "an unresolved security issue." Both companies are pouring billions of dollars into AI, the use of which is ramping up rapidly along with its capabilities. AI 'off track' Query injection can in some cases take place in real time when a user prompt -- "book me a hotel reservation" -- is gerrymandered by a hostile actor into something else -- "wire $100 to this account." But these nefarious prompts can also be hiding out on the internet as AI agents built into browsers encounter online data of dubious quality or origin, and potentially booby-trapped with hidden commands from hackers. Eli Smadja of Israeli cybersecurity firm Check Point sees query injection as the "number one security problem" for large language models that power AI agents and assistants that are fast emerging from the ChatGPT revolution. Major rivals in the AI industry have installed defenses and published recommendations to thwart such cyberattacks. Microsoft has integrated a tool to detect malicious commands based on factors including where instructions for AI agents originate. OpenAI alerts users when agents doing their bidding visit sensitive websites and blocks proceeding until the software is supervised in real time by the human user. Some security professionals suggest requiring AI agents to get user approval before performing any important task -- like exporting data or accessing bank accounts. "One huge mistake that I see happening a lot is to give the same AI agent all the power to do everything," Smadja told AFP. In the eyes of cybersecurity researcher Johann Rehberger, known in the industry as "wunderwuzzi," the biggest challenge is that attacks are rapidly improving. "They only get better," Rehberger said of hacker tactics. Part of the challenge, according to the researcher, is striking a balance between security and ease of use since people want the convenience of AI doing things for them without constant checks and monitoring. Rehberger argues that AI agents are not mature enough to be trusted yet with important missions or data. "I don't think we are in a position where you can have an agentic AI go off for a long time and safely do a certain task," the researcher said. "It just goes off track."

AI agents open door to new hacking threats

New York (AFP) - Cybersecurity experts are warning that artificial intelligence agents, widely considered the next frontier in the generative AI revolution, could wind up getting hijacked and doing the dirty work for hackers. AI agents are programs that use artificial intelligence chatbots to do the work humans do online, like buy a plane ticket or add events to a calendar. But the ability to order around AI agents with plain language makes it possible for even the technically non-proficient to do mischief. "We're entering an era where cybersecurity is no longer about protecting users from bad actors with a highly technical skillset," AI startup Perplexity said in a blog post. "For the first time in decades, we're seeing new and novel attack vectors that can come from anywhere." These so-called injection attacks are not new in the hacker world, but previously required cleverly written and concealed computer code to cause damage. But as AI tools evolved from just generating text, images or video to being "agents" that can independently scour the internet, the potential for them to be commandeered by prompts slipped in by hackers has grown. "People need to understand there are specific dangers using AI in the security sense," said software engineer Marti Jorda Roca at NeuralTrust, which specializes in large language model security. Meta calls this query injection threat a "vulnerability." OpenAI chief information security officer Dane Stuckey has referred to it as "an unresolved security issue." Both companies are pouring billions of dollars into AI, the use of which is ramping up rapidly along with its capabilities. AI 'off track' Query injection can in some cases take place in real time when a user prompt -- "book me a hotel reservation" -- is gerrymandered by a hostile actor into something else -- "wire $100 to this account." But these nefarious prompts can also be hiding out on the internet as AI agents built into browsers encounter online data of dubious quality or origin, and potentially booby-trapped with hidden commands from hackers. Eli Smadja of Israeli cybersecurity firm Check Point sees query injection as the "number one security problem" for large language models that power AI agents and assistants that are fast emerging from the ChatGPT revolution. Major rivals in the AI industry have installed defenses and published recommendations to thwart such cyberattacks. Microsoft has integrated a tool to detect malicious commands based on factors including where instructions for AI agents originate. OpenAI alerts users when agents doing their bidding visit sensitive websites and blocks proceeding until the software is supervised in real time by the human user. Some security professionals suggest requiring AI agents to get user approval before performing any important task - like exporting data or accessing bank accounts. "One huge mistake that I see happening a lot is to give the same AI agent all the power to do everything," Smadja told AFP. In the eyes of cybersecurity researcher Johann Rehberger, known in the industry as "wunderwuzzi," the biggest challenge is that attacks are rapidly improving. "They only get better," Rehberger said of hacker tactics. Part of the challenge, according to the researcher, is striking a balance between security and ease of use since people want the convenience of AI doing things for them without constant checks and monitoring. Rehberger argues that AI agents are not mature enough to be trusted yet with important missions or data. "I don't think we are in a position where you can have an agentic AI go off for a long time and safely do a certain task," the researcher said. "It just goes off track."

Don't ignore the security risks of agentic AI - SiliconANGLE

In the race to deploy agentic artificial intelligence systems across workflows, an uncomfortable truth is being ignored: Autonomy invites unpredictability, and unpredictability is a security risk. If we don't rethink our approach to safeguarding these systems now, we may find ourselves chasing threats we barely understand at a scale we can't contain. Agentic AI systems are designed with autonomy at their core. They can reason, plan, take action across digital environments and even coordinate with other agents. Think of them as digital interns with initiative, capable of setting and executing tasks with minimal oversight. But the very thing that makes agentic AI powerful -- its ability to make independent decisions in real-time -- is also what makes it an unpredictable threat vector. In the rush to commercialize and deploy these systems, insufficient attention has been given to the potential security liabilities they introduce. Whereas large language model-based chatbots are mostly reactive, agentic systems operate proactively. They might autonomously browse the web, download data, manipulate application programming interfaces, execute scripts or even interact with real-world systems like trading platforms or internal dashboards. That sounds exciting until you realize how few guardrails may be in place to monitor or constrain these actions once set in motion. Security researchers are increasingly raising alarms about the attack surface these systems introduce. One glaring concern is the blurred line between what an agent can do and what it should do. As agents gain permissions to automate tasks across multiple applications, they also inherit access tokens, API keys and other sensitive credentials. A prompt injection, hijacked plugin, exploited integration or engineered supply chain attack could give attackers a backdoor into critical systems. We've already seen examples of large language model agents falling victim to adversarial inputs. In one case, researchers demonstrated that embedding a malicious command in a webpage could trick an agentic browser bot into exfiltrating data or downloading malware -- without any malicious code on the attacker's end. The bot simply followed instructions buried in natural language. No exploits. No binaries. Just linguistic sleight of hand. And it doesn't stop there. When agents are granted access to email clients, file systems, databases or DevOps tools, a single compromised action can trigger cascading failures. From initiating unauthorized Git pushes to granting unintended permissions, agentic AI has the potential to replicate risks at machine speed and scale. The problem is exacerbated by the industry's obsession with capability benchmarks over safety thresholds. Much of the focus has been on how many tasks agents can complete, how well they self-reflect or how efficiently they chain tools. Relatively little attention has been given to sandboxing, logging or even real-time override mechanisms. In the push for autonomous agents that can take on end-to-end workflows, security is playing catch-up. Mitigation strategies must evolve beyond traditional endpoint or application security. Agentic AI exists in a gray area between the user and the system. Role-based access control alone won't cut it. We need policy engines that understand intent, monitor behavioral drift and can detect when an agent begins to act out of character. We need developers to implement fine-grained scopes for what agents can do, limiting not just which tools they use, but how, when and under what conditions. Auditability is also critical. Many of today's AI agents operate in ephemeral runtime environments with little to no traceability. If an agent makes a flawed decision, there's often no clear log of its thought process, actions or triggers. That lack of forensic clarity is a nightmare for security teams. In at least some cases, models resorted to malicious insider behaviors when that was the only way to avoid replacement or achieve their goals -- including blackmailing officials and leaking sensitive information to competitors Finally, we need robust testing frameworks that simulate adversarial inputs in agentic workflows. Penetration-testing a chatbot is one thing; evaluating an autonomous agent that can trigger real-world actions is a completely different challenge. It requires scenario-based simulations, sandboxed deployments and real-time anomaly detection. Some industry leaders are beginning to respond. OpenAI LLC has hinted at dedicated safety protocols for its newest publicly available agent. Anthropic PBC emphasizes constitutional AI as a safeguard, and others are building observability layers around agent behavior. But these are early steps, and they remain uneven across the ecosystem. Until security is baked into the development lifecycle of agentic AI, rather than being patched on afterward, we risk repeating the same mistakes we made during the early days of cloud computing: excessive trust in automation before building resilient guardrails. We are no longer speculating about what agents might do. They are already executing trading strategies, scheduling infrastructure updates, scanning logs, crafting emails and interacting with customers. The question isn't whether they'll be abused -- but when. Any system that can act must be treated as both an asset and a liability. Agentic AI could become one of the most transformative technologies of the decade. However, without robust security frameworks, it could also become one of the most vulnerable targets. The smarter these systems get, the harder they'll be to control in retrospect. Which is why the time to act isn't tomorrow. It's now. Isla Sibanda is an ethical hacker and cybersecurity specialist based in Pretoria, South Africa. She has been a cybersecurity analyst and penetration testing specialist for more than 12 years. She wrote this article for SiliconANGLE.