Major firms halt AI rollouts as hidden data surfaces, exposing critical governance gaps

When revealed data brings AI rollouts to a screeching halt - and how to manage it

Follow ZDNET: Add us as a preferred source on Google. ZDNET's key takeaways * AI can boost productivity and improve data access. * Tech leaders have had to halt rollouts due to data concerns. * Long-fogotten insights emerge with AI prompts. Agentic and generative AI have opened up information and insights to professionals in enterprises. However, evidence suggests that trend could be too much of a good thing. At a recent conference, veterans of enterprise AI rollouts issued cautionary words to professionals considering diving headfirst into AI. The issues these professionals encountered even led to temporary halts in AI rollouts meant to boost employee productivity, as executives reassessed information that could be exposed internally. At the same time, the executives, who spoke on a panel at the recent Veeam conference in New York City, emphasized that AI wasn't the source of the challenge. Both panelists' organizations had accumulated vast stores of data, and one required a new governance structure. Also: 96% of IT pros use AI now: Their top 7 agentic applications and biggest implementation roadblocks Steve MacIntyre, senior vice president at Fidelity Investments, described how his 400,000-employee company saw data long tucked away in the recesses of its organization -- on SharePoint sites or in network-attached storage, for example -- suddenly surface via AI prompts. "It wasn't an AI problem," he said. "It was the productivity and the ability of AI to find things quickly." Wim Geurden, chief architect for enterprise tech at EY, described his company's challenge as pinning down data ownership across its global network of independent affiliates -- data that was also surfacing through its AI engine. "When big enterprise search was launched, all kinds of stuff started to surface in places that people went," he said. "EY Global doesn't own any of the data. Every member firm owns its data. That is where the first questions were raised. What's all this? How many SharePoint sites? We had multiple petabytes of data, and it was the Wild West. There was no lifecycle management on these SharePoint sites, and half of them had no owners. We didn't know when they were last accessed." Also: 51% of professionals say AI workslop lowers their productivity - stop it in 2 steps At Fidelity, information was emerging from a vast library of PowerPoint and PDF reports. "We have an entire history, decades of research notes at Fidelity, such as PDFs," said MacIntyre. "We gave out a few licenses for Copilot, and immediately, two days in, legal came to me and said we have an AI problem. One of my team did a search to find something and AI came back with all the PowerPoints that were on SharePoint from years ago." AI is a "tremendous search engine that runs at speed," MacIntyre continued. "Suddenly, it's searching everything that it has access to, and surfacing that to us in a meaningful way. Everybody thought we had an AI problem, but what it showed was a problem of securing data. This issue hit home when we immediately realized that we had all of this data that we didn't think we cared about -- unstructured data -- and along came LLMs, and suddenly all of that data becomes gold." Establishing guardrails At EY, as the gates of its vast data stores opened to AI, the priority was to "find who owns the data," said Geurden. "The second thing we did was we shut everything off." Users could only access the Copilot tool if they were licensed. Also: Building an agentic AI strategy that pays off - without risking business failure The data ownership verification process included identifying and labeling the data found across the EY enterprise, Geurden continued. For example, labels included "confidential" or "financial services." AI itself offered a means to help label the company's knowledge repositories of unstructured data, Geurden explained, noting the challenge of human labeling with a 25% annual turnover rate. However, labeling needs to go deeper than simple high-level tags. "The first thing is we have to know what was there when the AI ran," said Geurden. "We need to have the historical picture, the versions." Then, "we have to go way beyond the labeling of confidential information. We need to have geo-restrictions, geo-labeling, line-of-business labeling, linked to our contracts, because we get an enormous amount of client data specifying what we can do and what we cannot do." Also: Over 80% of US government agencies already use AI agents - and it's only the beginning All of this metadata has to be codified into contracts, he added: "That's the easy part. Then we have to codify it in some technological structure. That is, for now, still very, very cumbersome." Governance is the key to success across all aspects of these AI implementations, the executives emphasized. "We have to know what's being used," said MacIntyre. "That brings into play the idea of shadow AI, shadow IT, all those kinds of things -- and it goes back to the endpoint data. We have to know that the asset inventory is accurate. Are they aligned with the use cases that are registered and approved? That way, at least we know that if someone's working on something, they should be using Claude, because it's tied to a particular project that was approved for that." Also: These 4 critical AI vulnerabilities are being exploited faster than defenders can respond Next, "we have to think about what's the safe environment where we want these agents to run?" MacIntyre continued: "How do we want them to interact with the foundational models? What architecture do we put in place to funnel all that activity into a place that gives us the right visibility and telemetry so we can see that agents and applications using AI are behaving in the way that was intended? Or misbehaving?" An additional challenge -- perhaps the most vexing for all digital leaders at this time -- is establishing agent identity, said MacIntyre: "How do you give an agent identity? They then become an employee. But what if my agent only lives for seconds? It's a really interesting problem, and I don't know if anybody's solved it really well yet."

Agentic AI - an infinite regress of cost/benefit challenges?

A computer can never be held accountable. Therefore, a computer must never make a management decision. Those were the words famously used by IBM in a 1979 training manual and slide presentation. But according to a new Harvard Business Review report, Solving Agentic AI's Data Infrastructure and Telemetry Needs, published this month, that axiom has been thrown out completely by business leaders. It says: Agentic AI is moving from experimental to operational inside most businesses. Companies aren't just autocompleting emails or summarizing documents anymore, they're asking software to decide what to do next. [...] Such uses of technology represent an epochal shift in how work gets done. In other words, management decisions are being ceded to AI agents. But is that wise? The report adds: There's the need for good governance to dictate how agents are allowed to act. Agents' speed and autonomy mean that companies must have strict guardrails around what they can access and do - and the ability to stop them when necessary. Because agents can potentially make changes to business records and data sources, these concerns go hand in hand with considering liability and compliance. Consider that statement for a moment - "Agents can potentially make changes to business records and data sources". In essence, that is an admission that an agentic AI can re-write history, in effect, and leave your organization de-coupled from verifiable fact, evidence, and a reliable audit trail. Separate research from Freshworks, The Global Cost of Complexity Report: The Midmarket AI Complexity Trap, published this week, surveyed over 12,000 IT professionals. Among its many findings, the report says: Managing AI is now adding to the workload it was meant to reduce, with teams fixing flawed outputs and governing tool sprawl across a growing stack of AI products. More than eight in ten (86%) of mid-market IT leaders say managing AI complexity has actually increased their team's workload, and 80% report that AI outputs are introducing noise, errors, or rework, a phenomenon the report terms 'AI slop'. This re-defines AI slop from meaning lazy, ersatz, AI-generated art to what others have called "vibe citing": hallucinations presented as business insight. And according to the Harvard report, that is happening in an overwhelming majority of organizations. The Freshworks report adds: AI is generating work faster than it is eliminating it, and IT teams are absorbing the difference. So much for the infinite productivity uptick predicted by some AI acolytes... Inevitably, this frustrating new burden also impacts on organizations' ability to demonstrate ROI from AI, especially given the many reports in recent years that have identified cost and time savings as the dominant reasons for enterprise adoption of the technology. More, it has worrying implications: what of the errors that have never been spotted, and which have yet to work through the system? How many management decisions - human or agentic - have been based on hallucinated data? And, to echo that fifty-year-old IBM observation, who is responsible for those failings? (AI vendors are working to ensure it isn't them, of that you can be sure.) Factor in the 1,494 known examples worldwide of fake AI caselaw being presented in completed court cases, (over two-thirds of them in the US), plus EY Canada retracting a report on loyalty scheme fraud this month due to its countless AI-generated citations, and Deloitte part-refunding the Australian government last year for a report that was also full of hallucinations, and there is ample evidence that even expert service providers are not just trusting AI too quickly, but also abandoning their professional responsibilities in the process. Indeed, it is remarkable just how quickly due diligence appears to have been set aside by experienced professionals. So, how can we expect less experienced junior employees to do better? The Harvard Business Review report is far from evangelical in its own summary of agentic AI's impact on business.Sponsored by AI telemetry platform Cribl, it notes: We see the same pattern over and over: forward-looking executives have big ambitions for agentic AI, but they're trying to run those agents on top of fragmented, expensive, and opaque telemetry. Most businesses don't have an artificial intelligence (AI) problem in the abstract; they have a telemetry and platform problem in practice." Ouch. And yet other research clearly does find that organizations have an AI problem too, in the sense of seeking deterministic answers from systems that are probabilistic in nature. Bear in mind, Anthropic CEO Dario Amodei recently wrote on his own website: Models are trained on vast amounts of literature that include many science-fiction stories involving AIs rebelling against humanity. This could inadvertently shape their priors or expectations about their own behavior in a way that causes them to rebel against humanity. That is surely a simple admission that AIs can't tell truth from fiction, so perhaps we should not be surprised when they generate fiction in place of evidenced, verified, peer-reviewed fact? (And who trained the system?) The Harvard Business Review report goes on to say that, while 96% of senior leaders say agentic AI will be "critical to business strategy within two years", only 23% believe they currently have the strategy and infrastructure in place to support it. Other findings include the 47% of organizations deploying AI agents who say that infrastructure costs have far exceeded their expectations, while almost as many (46%) cite unclear ROI and performance metrics resulting from that. In short, the ambition is enormous but the infrastructure supporting it isn't, notes the report: The problem isn't a lack of vision. It's that most enterprises are trying to run a new generation of AI on legacy observability and security stacks that were never designed for it. [...] For organizations at the leading edge of agent deployment, those legacy systems are not just struggling, they are already failing under the load. Cribl co-founder and CEO Clint Sharp adds: Data is growing at a 30% CAGR, but budgets are not, and now AI agents are multiplying that problem by an order of magnitude. The infrastructure most enterprises built for the last decade simply wasn't designed for the agentic workloads of the next one. But this leaves many enterprises with an infinite regress of challenges: they are being exhorted to invest more in a supporting infrastructure that is fit for the agentic age, and yet they are already struggling with mounting costs. Meanwhile, the promised ROI is absent to marginal in many cases - see diginomica, passim - and significant amounts of saved time are taken up with tracing and fixing errors and hallucinations. The point is this: it is a huge ask to expect enterprises to spend even more money in the hope that ROI will finally appear as promised - infinite speculation with no guarantee of accumulation, perhaps. With early sight of the above research, I put some of these findings to Jakob Freund, co-founder and CEO of enterprise orchestration platform Camunda last week at the company's agentic-focused CamundaCon event in Amsterdam. After all, the core theme of that event was the need to rein in and manage agentic AIs, and not let them manage you and risk damaging your business. What did he make of the Harvard Business Review finding that 96% of organizations see agentic AI as critical, but only 23% have the infrastructure in place to capitalize on the technology? And how could those businesses be persuaded to invest yet more money, given the soaring costs already reported by many? Freund said: I'm not surprised at all to hear these two numbers. It's exactly the problem that we're tackling. So, in that sense, that is another confirmation of why there's a need for [orchestration]. But in a way, that is also a self-selected filter. I'm not so concerned that leaders wouldn't see the need for that investment, because they do need to put together an AI-first operating infrastructure. The challenge now is how to get there. That said, I am sure there are still a lot of leaders and organizations out there that have not fully realised that yet. OK, but many of those leaders report that costs are already soaring, and yet they are not seeing the anticipated payback. Freund replied: Yeah, but it depends on how we define an infrastructure cost. I think many are negatively surprised [sic] by the cost of tokens in LLMs, right, and token consumption efficiency is an imperative. Deterministic orchestration based on if-this-then-that [ITTT] statements is very cheap, with regards to resource consumption, so you probably want to do deterministic orchestration as much as possible and dynamic, LLM-driven orchestration only as much as necessary, because it is by definition way more expensive. And far less reliable, perhaps? He concurred: And less reliable, exactly. It is less efficient and more risky. So, what's the takeaway from all this? Essentially it is this: things have not changed that much from the days of IBM's famous axiom. You are still in charge, and you should not let your technology manage you or take the decisions for you. And beyond that, you should not assume that it has presented you with the verifiable facts. Take the time and the responsibility to check, because you won't be able to blame the AI.