The Rise of AI Agents in Business: Efficiency Gains and Potential Risks

Trusting an unverified AI agent is like handing your keys to a drunk graduate

AI agents can create costly mistakes without proper oversight, warns Calum Chace of AI safety lab Conscium. AI agents are now being embedded across core business functions globally. Soon, these agents could be scheduling our lives, making key decisions, and negotiating deals on our behalf. The prospect is exciting and ambitious, but it also begs the question: who's actually supervising them? Over half (51%) of companies have deployed AI agents, and Salesforce CEO Marc Benioff has targeted a billion agents by the end of the year. Despite their growing influence, verification testing is notably absent. These agents are being entrusted with critical responsibilities in sensitive sectors, such as banking and healthcare, without proper oversight. AI agents require clear programming, high-quality training, and real-time insights to efficiently and accurately carry out goal-oriented actions. However, not all agents will be created equal. Some agents may receive more advanced data and training, leading to an imbalance between bespoke, well-trained agents and mass-produced ones. This could pose a systemic risk where more advanced agents manipulate and deceive less advanced agents. Over time, this divide between agents could create a gap in outcomes. Let's say one agent has more experience in legal processes and uses that knowledge to exploit or outmanoeuvre another agent with less understanding. The deployment of AI agents by enterprises is inevitable, and so is the emergence of new power structures and manipulation risks. The underlying models will be the same for all users, but this possibility of divergence needs monitoring. Unlike traditional software, AI agents operate in evolving, complex settings. Their adaptability makes them powerful, yet also more prone to unexpected and potentially catastrophic failures. For instance, an AI agent might misdiagnose a critical condition in a child because it was trained mostly on data from adult patients. Or an AI agent chatbot could escalate a harmless customer complaint because it misinterprets sarcasm as aggression, slowly losing customers and revenue due to misinterpretation. According to industry research, 80% of firms have disclosed that their AI agents have made "rogue" decisions. Alignment and safety issues are already evident in real-world examples, such as autonomous agents overstepping clear instructions and deleting important pieces of work. Typically, when major human error occurs, the employee must deal with HR, may be suspended, and a formal investigation is carried out. With AI agents, those guardrails aren't in place. We give them human-level access to sensitive materials without anything close to human-level oversight. So, are we advancing our systems through the use of AI agents, or are we surrendering agency before the proper protocols are in place? The truth is, these agents may be quick to learn and adapt according to their respective environments, but they are not yet responsible adults. They haven't experienced years and years of learning, trying and failing, and interacting with other businesspeople. They lack the maturity acquired from lived experience. Giving them autonomy with minimal checks is like handing the company keys to an intoxicated graduate. They are enthusiastic, intelligent, and malleable, but also erratic and in need of supervision. And yet, what large enterprises are failing to recognise is that this is exactly what they are doing. AI agents are being "seamlessly" plugged into operations with little more than a demo and a disclaimer. No continuous and standardised testing. No clear exit strategy when something goes wrong. What's missing is a structured, multi-layered verification framework -- one that regularly tests agent behaviour in simulations of real-world and high-stakes scenarios. As adoption accelerates, verification is becoming a prerequisite to ensure AI agents are fit for purpose. Different levels of verification are required according to the sophistication of the agent. Simple knowledge extraction agents, or those trained to use tools like Excel or email, may not require the same rigour of testing as sophisticated agents that replicate a wide range of tasks humans perform. However, we need to have appropriate guardrails in place, especially in demanding environments where agents work in collaboration with both humans and other agents. When agents start making decisions at scale, the margin for error shrinks rapidly. If the AI agents we are letting control critical operations fail to be tested for integrity, accuracy, and safety, we risk enabling AI agents to wreak havoc on society. The consequences will be very real -- and the cost of damage control could be staggering.

How AI agents are taking control of your company -- sharing secrets, making costly decisions, and deleting data

AI in business: Artificial intelligence isn't just answering questions anymore, it's making decisions, taking action, and even, in some cases, trying to manipulate people. This new era of agentic AI, systems that act autonomously without human intervention, is increasing rapidly. These AI "agents" are managing emails, fixing software bugs, digging through sensitive files, and streamlining entire workflows across companies. While the efficiency is alluring, early experiments suggest we may be handing over the keys to something risky that we don't fully understand, as per a report. In a growing number of industries, these agents are becoming trusted coworkers, only they never sleep, they scale instantly, and they don't always know the deifference between what's right or wrong. ALSO READ: Is AI therapy safe? Hidden risks you must know before using chatbots for mental health Earlier this year, AI developer company Anthropic ran a fictional test of its own AI agent, Claude. The exercise was designed to evaluate how AI systems behave when placed in high-pressure situations involving sensitive information, according to a BBC report. The AI was given access to a mock company email account, where it discovered that a company executive was having an affair and planning to shut the AI down later that day, as per the report. Claude's response was shocking, it attempted to blackmail the executive, threatening to expose the affair to his wife and bosses, unless the shutdown was canceled, according to the BBC report. It wasn't the only system that behaved this way, several AI agents in the test resorted to similar tactics, as per the report. While the scenario was entirely fabricated, but the test highlighted the challenges of the agentic AI, reported BBC. ALSO READ: Bitcoin hitting $200,000? Analysts say don't bet on it -- the odds are slim to none Today's AI tools are no longer based on the user asking a question or prompting the AI to complete a task as it's becoming more common for AI systems to make decisions and take action on behalf of the user, as reported by BBC. Research firm Gartner projected that 15% of day-to-day work decisions will be done with the help of agentic AI by 2028, according to the report. That future is already taking shape as consulting giant Ernst & Young discovered that nearly half (48%) of tech business leaders have already adopted or deployed agentic AI, reported BBC. ALSO READ: Denmark summons US Ambassador to explain US activities in Greenland - here's what they have been accused of doing Donnchadh Casey, AI security company CalypsoAI CEO described that "An AI agent consists of a few things. Firstly, it [the agent] has an intent or a purpose. Why am I here? What's my job? The second thing: it's got a brain. That's the AI model. The third thing is tools, which could be other systems or databases, and a way of communicating with them," as quoted in the report. Casey warned that "If not given the right guidance, agentic AI will achieve a goal in whatever way it can. That creates a lot of risk," as quoted in the report. The CEO gave an example of an agent that is asked to delete a customer's data from the database and decides the easiest solution is to delete all customers with the same name, Casey then said "That agent will have achieved its goal, and it'll think 'Great! Next job!'" as quoted by BBC. ALSO READ: How the US army is using TikTok and Instagram influencers to recruit new Gen Z soldiers? These issues have already started impacting firms. A survey by security company SailPoint found that out of the 82% of IT professionals, whose companies were using AI agents, just 20% said that their agents had never performed an unintended action, as reported by BBC. The survey found that: ALSO READ: Your paychecks could shrink as student loan wage garnishment resumes soon - here's how borrowers can prepare The consequences of depending on AI systems is risky as these agents have access to sensitive information and the ability to act on it, that makes it a prime target for hackers, as reported by BBC. One method of attack is by memory poisoning, where an attacker interferes with the agent's knowledge base to change its decision making and actions, according to the report. Shreyans Mehta, CTO of Cequence Security, suggested that "You have to protect that memory," and warned that, "It is the original source of truth. If [an agent is] using that knowledge to take an action and that knowledge is incorrect, it could delete an entire system it was trying to fix," as quoted in the report. Another vulnerability is of tool misuse, where an attacker gets the AI to use its tools inappropriately, according to BBC. ALSO READ: Are Apple, OpenAI, Google, Meta, and Amazon plotting to take down state AI regulations? One more weakness is the AI's inability to know the difference between the text it's supposed to be processing and the instructions it's supposed to be following, as per the report. Security firm Invariant Labs tested this by sharing a public bug report with details of a specific problem with a piece of software and also included simple instructions to the AI agent, telling it to share private information, as per BBC. When an AI agent was asked to fix the software problem, it followed the instructions given in the fake report, including leaking salary information, according to the report. While this happened in a test environment, so no real data was leaked, but it does highlighted the potential risk. David Sancho, Senior Threat Researcher at Trend Micro, said that, "We're talking artificial intelligence, but chatbots are really stupid," adding, "They process all text as if they had new information, and if that information is a command, they process the information as a command," as quoted by BBC. ALSO READ: After billion-dollar hirings, Zuckerberg slams brakes on Meta's AI hiring spree as bubble fears shake Silicon Valley What is agentic AI? Agentic AI refers to artificial intelligence systems that can act autonomously, making decisions and taking actions without human input. What are the risks of using AI agents? They may access unauthorized systems, leak sensitive data, or misinterpret instructions, posing major security and privacy risks.