AI governance in regulated industries struggles as agentic AI adoption outpaces human oversight

AI Governance Lessons for Leaders

Enterprise AI adoption is accelerating rapidly. Organizations across industries are moving beyond pilot projects to deploy AI systems that influence critical business decisions, customer interactions, and operational workflows. As the deployment scales, so do the risks. Early adopters have learned valuable lessons about governance frameworks, risk mitigation, and responsible AI implementation, the lessons that can guide organizations navigating their own AI journeys. Why AI Governance Matters Now AI governance matters more now than ever before. Here's why: traditional software follows clear rules, if X happens, do Y. AI systems work differently. They learn patterns from data and make educated guesses, which creates some real challenges. Unpredictable Outputs: AI can surprise you when it encounters situations it hasn't seen before. Take a customer service chatbot trained on formal business emails. Put it in front of casual text messages or regional slang, and it might completely miss the mark. Amplified Bias: AI learns from historical data, which means it can pick up and magnify existing biases. Several financial institutions learned this the hard way when their credit scoring models turned out to discriminate against certain groups, leading to regulatory investigations and serious reputation damage. Opacity and Explainability: Neural networks are notoriously opaque. When an AI decides, it's often hard to explain exactly why. This becomes a real problem in regulated industries where you need to show your work. Data Privacy and Security: AI systems are data-hungry, often processing sensitive personal information. Keeping that data secure while staying compliant with regulations like GDPR and CCPA isn't optional -- it requires solid governance from day one. These challenges echo what happened with early cloud adoption a decade ago. Companies that took governance seriously from the start pulled ahead. Those that rushed in without proper controls paid for it later. Key Lessons from Early Adopters Organizations across industries have developed governance frameworks tailored to their specific risks, yet common patterns emerge that apply broadly. Model Risk Management and Validation Successful organizations treat AI governance as an extension of existing risk management practices, adapting frameworks from financial services and healthcare to their specific contexts. They inventory all AI systems and classify them by risk level based on business impact and regulatory exposure, with high-risk applications receiving enhanced oversight. Beyond accuracy metrics, organizations test for fairness across demographic groups, robustness under edge cases, and performance degradation over time. Human Oversight and Accountability Despite AI's capabilities, early adopters maintain human oversight for critical decisions through tiered authority structures. Low-risk, high-volume decisions operate autonomously, while medium-risk decisions trigger human review when confidence scores fall below thresholds, and high-risk decisions always require human validation. AI systems must explain their recommendations, when a loan application is denied or a medical diagnosis is suggested, the system identifies key factors influencing the decision, enabling human operators to validate reasoning and ensure compliance. Human operators can override AI recommendations when contextual factors suggest inappropriate outputs, with these overrides logged and analyzed to identify systematic model weaknesses and inform improvements. Centralized Governance with Distributed Execution Technology companies scaling AI across multiple products have found success with centralized governance teams that establish standardized review processes proportional to risk level, ensuring consistent standards without creating bottlenecks for low-risk applications. These centralized teams develop reusable tools for model testing, bias detection, and performance monitoring, preventing redundant efforts and ensuring consistent practices across the organization. Building Effective Governance Frameworks Successful AI governance frameworks share common elements that organizations can adapt to their specific contexts: Cross-Functional Collaboration: Effective governance requires coordination between technical teams (ensuring models perform as intended), legal and compliance (assessing regulatory requirements), ethics teams (evaluating societal impacts), and business leadership (aligning governance with strategic objectives). Comprehensive Documentation: Organizations maintain model cards documenting purpose, training data, performance metrics, and limitations. Decision logs capture AI-generated outputs, confidence scores, and human overrides. Change management processes track all model updates with clear rationale and approval chains. Fail-Safe Mechanisms: Critical systems include confidence thresholds that trigger human review, redundant systems that cross-check AI outputs, and graceful degradation that ensures business continuity when AI systems fail. Continuous Improvement: Organizations establish incident response processes, implement feedback loops that inform system improvements, and evolve governance frameworks as new risks emerge and best practices mature. Getting Started with AI Governance Organizations beginning their AI governance journey can apply lessons from early adopters: Start with Risk Assessment: Inventory existing and planned AI systems, classifying them by risk level. Focus initial efforts on highest-risk applications where failures have the greatest impact. Adapt Existing Frameworks: Build on existing risk management, compliance, and quality assurance frameworks rather than creating entirely new processes. This accelerates implementation and leverages institutional knowledge. Invest in Monitoring Infrastructure: Implement tools for model monitoring, bias detection, and explainability early. These capabilities become harder to retrofit as deployments scale. Foster Responsible AI Culture: Educate teams on responsible AI principles and create psychological safety for raising concerns. The biggest governance challenges are often organizational, not technical The Path Forward AI governance continues evolving as technologies advance and regulations mature. Governments worldwide are developing AI regulations, making governance maturity increasingly important for compliance. Industry groups are establishing shared standards, reducing the burden on individual organizations. AI itself is being used to monitor AI systems, automating compliance and anomaly detection. The lessons from early adopters are clear: effective AI governance is not a barrier to innovation but an enabler. Organizations that establish strong governance practices build stakeholder trust, reduce operational risks, and position themselves for sustainable AI-driven growth. As AI becomes central to business operations, governance maturity will separate leaders from laggards in the AI economy. About the Author Sowjanya Pandruju is a Cloud Application Architect at Amazon Web Services, specializing in serverless architectures and enterprise AI deployments. With more than 13 years of experience in distributed systems and cloud computing, she has led the design and implementation of large-scale AI systems serving millions of users. Sowjanya holds multiple AWS certifications and has published research on serverless computing patterns, multi-agent systems, and enterprise AI architecture. She regularly speaks at industry conferences and contributes to open-source projects focused on cloud-native AI solutions. Her work bridges the gap between cutting-edge AI research and practical enterprise implementations, helping organizations successfully deploy AI systems at scale. Disclaimer: The authors are completely responsible for the content of this article. The opinions expressed are their own and do not represent IEEE's position nor that of the Computer Society nor its Leadership.

Why the future of AI governance depends on human judgment

This judgment-based AI governance sharpens with experience, with high-stakes decision-making ability peaking between 55 and 65. In the city centre of Kuala Lumpur, a senior bank manager, Diana, has her name on the decisions a machine makes. When the lending model approves or declines an application, she is the person that regulators hold responsible for mistakes. Diana is not just a reviewer who signs off - she interrogates the model, flags suspect outputs and owns the correction. Her bank can buy a better AI model next quarter, but it cannot, as easily, buy the insight Diana supplies. Executive committees and boards still treat AI governance as largely a technology problem: procure the system, place a human in the loop and then satisfy the auditors. If something in that sequence rings hollow, you're not alone. The phrase "human in the loop" has become a comfort we repeat without asking the harder question: can that person actually make wise decisions when the model and the situation disagree? The model was never the hard part of AI governance. Judgment matters more and appointing someone like Diana, who can choose when the stakes are real but the output is wrong, is crucial. The European Union's AI Act, in force since August 2024, requires under Article 14, that high-risk systems let a designated person oversee, question and override their output. Singapore's MAS has proposed risk-management guidelines, now past public consultation, that would put boards and senior management on the hook for decisions in lending, risk and fraud. South Korea's AI Basic Act, in force since January 2026, places safety and transparency duties on high-impact AI operators. Malaysia's proposed right to human review would require the reviewer to hold the authority and competence to overrule the machine. Vietnam's AI Law, in force since March 2026, goes furthest: it bans obstructing or disabling the human mechanisms that oversee and control AI, which implies that letting judgment wither through over-reliance may itself break the law. Now, notice the contrast when rules are absent. Australia has so far declined to enact a standalone AI law for the private sector, and New Zealand has taken a similarly light-touch path. That gives us a natural experiment in risk. In Singapore, Vietnam, South Korea and Malaysia, weak oversight is increasingly a legal and punitive risk: sanctions, licence loss and personal liability for named officers. In Australia and New Zealand, the absence of such measures represents a reputational risk. While in many businesses, legal risk earns a whole budget line, a board agenda item and named and trained people, reputational risk is simply assigned a policy document and a clean-up plan. One forces investment in human judgment, while the other will cross that bridge only when they come to it. So what do these laws actually demand of the individuals charged with being in the loop? The accountable person must catch the case that falls outside the pattern, question a confident output and hold the customer's interest, firm's exposure and regulator's intent at once. Then, after that, they make the call under uncertainty. This is cognitive sovereignty: the ability to stand apart from the machine and exercise higher-order situational judgment built from years of experience. It's rapidly becoming expensive brainpower in organizations. In a real-world study of more than 32,000 scans, radiologists overrode an FDA-cleared AI tool in about 2% of cases, and where they disagreed, the human call was right nearly nine times in ten. Hundreds of confirmed blood clots would have been missed by the model alone. The value was not in the routine agreement but in the rare, hard case a human caught. Here is where the familiar story about the ageing brain gets it backwards. The judgment AI governance requires can sharpen with experience rather than fade. A 2026 longitudinal study in Scientific Reports tracked nearly 4,000 adults and found no known ceiling on brain-health improvement at any age. A 2025 analysis in Intelligence found that the broad functioning behind high-stakes decisions tends to peak between 55 and 60, with those best suited to such roles rarely younger than 40 or older than 65. Dr. Sandra Bond Chapman of the Center for BrainHealth calls integrated reasoning, connecting past patterns to novel problems, our "platinum" cognitive function, often excelling between 55 and 65. I have written about this adult brain development arc in more detail. None of this is automatic. Age confers nothing on its own. These capabilities grow when trained deliberately, and they can erode under chronic overload, poor sleep and the repeated temptation to wave through an answer because it arrives polished. That last temptation has a name. Researchers call it automation bias, and decades of studies show it strikes experts as readily as novices: one review found wrong machine advice raised incorrect human decisions by about a quarter. Why? AI's polished output can create a feeling of rightness so strong that the brain sees a stop sign, and higher-order thinking never starts. Seniority is no shield. In fact, the most exposed professionals can be seasoned executives: all that experience supplies additional signals that everything looks right, so a confident output can stop hard-earned wisdom from ever becoming active judgment. This is why treating AI governance as a technology problem can be a costly mistake. Too often, organizations spend heavily on better models while underinvesting in the people meant to govern them. The best workers can be experienced professionals labelled "past their prime" and managed toward the door. We risk buying the system and starving the oversight. Diana's year looked ordinary. No headline promotion, and she has moved into a role her institution cannot operate without. She is not the person the bank replaces. Instead, she's the person they cannot lose. Boards can no longer simply ask if a human sits in the loop and check that box. They must assess whether that person can act with cognitive sovereignty and out-think the system when it matters, and whether anyone is investing so she can. You cannot govern what you depend on.

Agentic AI adoption outpaces governance in regulated industries

Agentic AI is already in your finance operations. Your governance framework is not Regulated industries are entering a turning point that many enterprise leaders have yet to fully grasp. Agentic AI tools capable of executing multi-step tasks with minimal human intervention, are now commonly embedded in audit and finance operations, automating testing, documentation, risk assessment, and reporting. But many organizations are still behind updating the governance infrastructure required to make those gains sustainable. Most organizations ask what AI can do, but neglect to evaluate whether they have operating models, governance frameworks, and human oversight capacity in place to control what AI does. In regulated environments, that gap is where exposure compounds quickly. Three Gaps Compounding at Once Validating AI output requires a different skill set than producing it. Traditional audit training doesn't develop that capability, and most firms have yet to redesign programs to account for that lack of knowledge. Junior staff are nominally in charge of reviewing AI-generated work they don't fully understand. In regulated environments, this creates easy-to-miss opportunities for exposure. Audit workflows were designed around human pacing and judgment. Agentic AI moves sequentially and at speed, silently resolving ambiguity rather than surfacing it. Layering AI tools onto processes built for human practitioners means unclear handoffs, undefined escalation paths, and audit trails that fail to document decision rationale in ways that satisfy regulators. When stewardship is a title rather than a function, organizations produce governance documentation that exists on paper, not in practice. Premature AI deployment can still look like a success even long after the foundation started to erode. Adoption metrics show usage. Cycle times improve. These ostensibly positive outcomes don't reveal whether employees can meaningfully evaluate what the system produces, whether workflows have been redesigned for how AI operates, or whether governance is anywhere close to complete. For enterprise leaders in regulated industries, the critical question is not whether the AI is working, but whether it surfaces issues early enough for teams to intervene effectively. In many organizations, AI implementation is also outpacing operational alignment. Risk, compliance, finance, and technology teams often operate with different assumptions about how agentic systems are being used and where accountability resides. Without shared oversight across those functions, governance gaps become harder to identify before they create operational or regulatory consequences. What Closing the Gap Actually Looks Like The organizations seeing sustainable results share a key characteristic: they build governance infrastructure before scaling use cases. In practice, that means establishing a centralized governance function with both business and technical representation. Successful AI governance in regulated environments requires joining stakeholders who understand operational stakes and regulatory requirements at the same table, with the authority to act on what they find. Domain stewards need real authority, with clear accountability for model performance, explicit escalation paths, and organizational backing to act accordingly. Defined rules of engagement are what separates a stewardship role from a title implying nominal ownership on an org chart. This structure must be built before deployment, not retrofitted after an incident. Starting narrow is the right instinct. Financial close, reconciliations, and anomaly detection are good initial use cases due to clean inputs, measurable outputs, and the presence of a human reviewer that evaluates what the system produced. Data flows need to be integrated across systems before models go into production. Scaling AI into fragmented processes doesn't fix fragmentation -- it accelerates it. Selecting a technology capable of bringing data integrity to the forefront is key for establishing sustained governance practices. Workforce readiness belongs on the governance roadmap alongside technical deployment. Junior staff need structured development in how to evaluate AI output including when to trust it, when to push back, and when to escalate. That capability doesn't emerge simply from exposure to AI tools. The firms getting this right are treating this part of the process as risk control. Another challenge is that many governance models remain reactive rather than adaptive. Regulatory expectations surrounding AI are evolving faster than most enterprise oversight structures, leaving organizations vulnerable to compliance gaps that may not become visible until after deployment. Companies that treat governance as an ongoing operational discipline, rather than a one-time implementation exercise, will be better positioned as both technology capabilities and regulatory scrutiny continue to advance. Governance Is the Foundation Agentic AI will continue expanding into audit and finance regardless of whether governance infrastructure is in place. The competitive pressure is too strong, and the case for efficiency is too compelling for adoption to slow. The question for enterprise leaders isn't whether to deploy AI -- it's whether they're building the operational foundation to deploy it responsibly. Accountability in regulated industries does not transfer to the algorithm. It stays with the humans who chose to deploy it, and with the organizations that decided they were ready when the evidence said otherwise. The leaders who are prepared have already answered this question: if something goes wrong, do we know exactly where judgment ended and automation began? Manage employees with the best HR software. This article was produced as part of TechRadar Pro Perspectives, our channel to feature the best and brightest minds in the technology industry today. The views expressed here are those of the author and are not necessarily those of TechRadarPro or Future plc. If you are interested in contributing find out more here: https://www.techradar.com/pro/perspectives-how-to-submit

The oversight paradox: Human control over AI may be eroding

Preserving that person, along with their skill, their practice, their judgement, will be increasing essential to governing AI well. Every framework for governing artificial intelligence (AI) rests on the same reassuring premise: the human stays in control. The EU's AI Act requires human oversight of high-risk systems; its General Data Protection Regulation (GDPR) restricts decisions taken by machines alone; corporate AI policies promise that a person reviews every output before it reaches a customer. The message is consistent: AI decides nothing on its own. But that premise hides a problem rarely stated out loud, and it is present from the very first day a system is deployed. The competence a human needs to oversee an AI system is not a fixed asset. It is built and kept alive through practice - the same practice the AI system is now doing instead of the human. The more capable the system, the more work we hand it, and the fewer occasions the overseer has to exercise the very judgement the oversight role depends on. Human oversight does not simply sit in place as a safeguard. It quietly depreciates, task by task, in step with the system it is meant to control. We call this the oversight paradox. The uncomfortable implication is that better-performing AI can make oversight weaker rather than stronger. As a system takes on more of the cognitive work, the human reviewing it has less first-hand command of that work and is therefore less able to catch the moments when the system is wrong. Capability and oversight risk rise together. This is not a worry about some future, more advanced AI. It describes what oversight is already up against today. The speed of the shift is what makes this urgent. On the GPQA benchmark -- doctoral-level questions in physics, chemistry and biology -- human PhD experts answer roughly 65% correctly. When the benchmark was released in late 2023, the strongest GPT-4 baseline managed only 39%. By 2026 the leading model scores roughly 94%. The comparison is not perfectly clean, but the broad picture holds: machines have moved from well below expert level to well above it. Software engineering shows the same curve. On SWE-bench, which asks models to fix real issues in open-source code, the best system at launch in 2023 resolved under 5% of problems. On the human-validated SWE-bench Verified set, leading systems now resolve well over 90%. The calculator took a generation to dull mental arithmetic. AI is reshaping expert performance on a timescale of months - and that compression is the whole problem for oversight. The faster a system climbs past the people meant to supervise it, the faster their hands-on familiarity with the task is left behind. Each capability jump quietly raises the bar the overseer must clear, while removing the practice that would let them clear it. The pattern is not confined to the lab. Anthropic's Economic Index finds that AI use is concentrated in mid-to-high-wage knowledge work -- software development, technical writing and similar professional tasks -- rather than at the lowest- or highest-paid ends of the labour market. Anthropic is careful not to claim that wholesale deskilling will follow; it explicitly declines to predict that outcome. But the exposure is telling: the tasks most touched by AI are precisely the expert, judgement-heavy ones on which meaningful oversight depends. Three distinct forces are at work, and they compound one another. People tend to trust an automated system when it is usually correct. Classic flight-simulator studies found that even professional pilots failed to act on problems the system missed, or followed faulty prompts without cross-checking. In scenarios where the information needed to catch the error was right there to be checked, error rates of around 55% were recorded. The same reflex transfers to knowledge work. A radiologist whose AI flags anomalies with very high accuracy will, after the hundredth scan, drift from evaluating the image to confirming the machine. The EU AI Act's Article 14 addresses this directly, requiring overseers to stay alert to automation bias. The system performs the task; the human is moved into a quality-assurance role. But quality assurance requires expertise, expertise is acquired through practice, and practice is the very thing the system has taken over. Consider a lawyer who spends two years reviewing AI-drafted contracts rather than writing them. At what point do they lose the feel for a strategic omission, or the judgement that only comes from having drafted hundreds of contracts themself? Each round of delegation makes the next round of oversight a little thinner. Article 22 of the GDPR restricts decisions based solely on automated processing where they significantly affect a person. Regulators have been explicit, though, that a controller cannot escape the rule with a token human review: the human involvement must be meaningful, carried out by someone with the authority to overrule the decision and the knowledge to weigh all the relevant data. Where that ability has eroded, what remains is compliance theatre. The EU AI Act tries to close the gap through Article 4, which requires AI literacy, and Article 14, which requires effective oversight of high-risk systems. Yet both largely assume a stable relationship between human competence and machine capability. A regulation can guarantee the possibility of intervention, but it cannot guarantee that the person intervening can still tell whether intervention is warranted. The paradox also reaches into democratic governance. AI systems are increasingly used in public administration, but the elected bodies responsible for governing them face a translation gap: the technical reality of these systems does not render easily into the language of democratic deliberation. At EU level, the AI Office must draw much of its expertise from the very organizations it regulates -- not through any conspiracy, but through a competence asymmetry built into the field. The oversight paradox is not an argument against adopting AI. It is an argument against assuming that human oversight maintains itself. Three directions deserve serious attention. The question every organization, regulator and legislature should be asking is not whether a human is in the loop. It is whether that human could still do the job without the AI - and whether that capacity is being actively kept alive. In many cases today, the honest answer is no. When the overseer can no longer do the work unaided, oversight stops being a safeguard and becomes a signature. The error that finally slips through can only be caught by a person who still knows enough to see it. Preserving that person -- their skill, their practice, their judgement -- is becoming the whole of governing AI well.

From chatbots to assistants: governance is key for AI agents

Responsible governance of AI agents means defining the extent of their capabilities according to the particular context in which they operate. After the wave of Generative AI, attention is shifting toward AI agents. These systems can plan tasks, access tools and take actions across digital environments on behalf of users. Unlike AI models that generate responses, agents can execute tasks across applications and interact with external systems. This shift from conversational tools to operational agents marks a structural change in how AI is deployed. It also introduces a new set of governance and security challenges that extend beyond model performance to entire system architectures. Early projects such as AutoGPT and LangChain-based agent prototypes demonstrated how large language models (LLM) could be chained together to plan and execute multistep tasks. Many early implementations, however, proved fragile and difficult to operate reliably. Today, the first wave of operational LLM-based agents is emerging in bounded workflows, while broader personal assistants built on emerging open-source frameworks such as OpenClaw are still evolving. The likely trajectory is a gradual expansion from narrowly scoped agents toward more capable assistants that can integrate across digital environments and act with increasing autonomy on behalf of users. What distinguishes the current wave of agentic systems is the combination of advances in memory, standardized system access and agent communication, alongside a growing ecosystem of open-source orchestration frameworks. Emerging protocols and infrastructure mechanisms such as the Model-Context Protocol (MCP), Agent2Agent (A2A) protocol, and the Agent Name Service (ANS) enable agents to access tools and external resources, communicate with other agents across systems and establish verifiable identities within distributed agent ecosystems. These developments help create the technical foundation that allows agents to interact with services such as email, messaging platforms, calendars, cloud storage and enterprise systems. Memory is a central feature that allows AI agents to transform into more advanced personal assistants. The ability to remember preferences and past interactions allows agents to anticipate needs, maintain continuity across tasks and create more personalized experiences over time. But the architectural feature that enables greater personalization also concentrates new risk. When memory is unified across surfaces such as communications, documents and productivity tools, the assistant becomes a highly integrated repository of personal or organizational data. Unlike traditional applications, where data is often siloed by function, agentic systems can reason across a range of data sources and contexts. While this cross-context capability enhances utility, weak permission structures can allow misuse or compromise that cascades across connected systems. Early deployments illustrate how powerful this unified memory model can be, but also how questions of data governance, access control and auditability have to be dealt with before broader application. Agentic systems also introduce a distinct class of security challenges. AI agents routinely process information from external sources such as web pages and documents, interpret this information and act using privileged tools and system integrations. This creates vulnerabilities that differ from those found in traditional software systems, where inputs are more structured, and actions are tightly controlled by predefined program logic. Several types of risk can emerge in practice when agents interact with external content and connected systems. Malicious instructions embedded in emails, documents or web pages can manipulate an agent's behaviour through prompt injection. Misconfigured permissions may give agents broader access than intended, and ambiguous instructions can lead an agent to take unintended actions when executing tasks across connected systems. As AI assistants evolve from experimental tools to embedded digital collaborators, security must be evaluated across the full architecture, rather than at the model level alone. The rise of AI agents highlights a broader governance challenge in which autonomy and authority have to be treated as deliberate design variables. As outlined in the World Economic Forum's work on AI agents and governance, the degree of autonomy granted to a system should be calibrated to the context in which it operates, the risks involved and the institutional maturity of the organization deploying it. This is especially important for AI assistants, which operate in highly sensitive environments with access to detailed communications, credentials and personal information. As agents become more capable, progressive governance becomes necessary, with safeguards expanding alongside their operational scope. In practice, this requires treating autonomy and authority as adjustable design parameters. Tasks that carry higher consequences should retain clear boundaries for when human approval is required, while access to critical systems should remain segmented rather than concentrated in a single agent. Visibility into agent behaviour also becomes critical, with logging, evaluation and auditability enabling organizations to monitor actions, detect failures and retain accountability as deployment expands. The emerging ecosystem associated with AI agents involves model providers, orchestration platforms, extension developers, enterprises and end users, which means that accountability can be diffuse unless roles and responsibilities are clearly defined. One key lesson from early adoption patterns is that when capability scales faster than governance, users are left to navigate complex risk trade-offs without clear institutional support. The rapid emergence of open-source projects such as OpenClaw has illustrated how quickly agent utility and autonomy are advancing, while the underlying governance architectures need to keep up and mature at the same pace. If calibrated carefully, AI agents and more capable personal assistants could become trusted components of daily digital life. Achieving this requires ecosystem-level coordination, proportionate safeguards and a clear recognition that system design and governance are inseparable in the age of agents.

AI Should Focus on Fixing Business Problems

The hardest part of adopting enterprise technology is rarely the technology itself. It's trust. If a team cannot explain how a tool works and where the human remains in control, the conversation slows down fast. With AI, corporate legal teams prioritize defensibility and explainability to avoid black box risks, often slowing, but not stopping adoption. In fact, law firm DLA Piper found 78 percent of companies see governance as the top AI barrier. This tension plays out every day in boardrooms and C-suites. Companies want the speed and scale of AI. Legal wants proof it won't blow up later. The result? Progress, but not at warp speed. And that's OK. Legal and compliance aren't the enemies here. They're not against innovation. They are against innovation without defensibility. That distinction should matter to every business leader, especially in human resources. For instance, if a hiring tool rejects candidates and a lawsuit claims bias, courts demand the full data trail. Humans are messy but hard to audit. AI creates a perfect paper trail that plaintiffs love. Several laws treat recommendations almost like decisions. For instance, the Equal Employment Opportunity Commission holds employers strictly liable for algorithmic bias in hiring and selection tools, even when vendors provide the system. Explainability fixes this. Show why Candidate X scores high: the candidate matched skills from the job description, tenure data, no ghosting history, and so on. AI shouldn't replace accountability Recent surveys show employers rapidly adopting AI to streamline hiring. When AI is used to improve fairness and transparency, many applicants report greater acceptance of AI-assisted hiring processes. People often prefer faster, clearer systems when they are treated fairly. AI can move low-value work into higher-value, more intelligent action, while freeing humans to focus on the parts of work that actually require judgment and relationship-building. It's a good reminder that automation should not replace accountability. It should remove bottlenecks so people can spend more time on the work that matters. Start with the business problem HR leaders understandably have questions about AI's role. Yet too many technology conversations begin with the tool itself when discussions should start with the business problem. What is the organization trying to fix? And what outcomes need to be improved? If those questions can't be answered clearly, no vendor demo will save you. And if HR executives can answer them clearly, they are already much closer to getting alignment across the business. Here are six points to consider before seeking approval to use AI from the general counsel's office. Leaders should explain the business result they expect, whether it's efficiency, revenue growth, or better service delivery. Before buying any new tool, define the pain point and show exactly how the solution improves performance. Ask how the vendor handles compliance, and which specific regulations the company buying the technology is responsible for. Legal slows AI to protect the company. Smart leaders flip it by building defensibility into the pitch. Legal and compliance should be partners from the start. Don't wait until the end of the buying process to involve them. Bring them in early conversations with vendors but with context. That means you have already done your homework, and you understand the use case and the risks. When that happens, legal is no longer a roadblock. It becomes a partner in making a better decision. The goal is not to ask the lawyers to bless the deal, but to bring them into the design of the process so they understand what is being built and why. Every executive project is easier when the risk team sees the logic before the purchase order is signed. Refrain from overhyping AI as something magical. If a business unit president says the system "decides" or "finds the best people" without explaining the criteria and the human review, the message can trigger distrust. Say what the system does: It surfaces information, the human reviews it, and the organization keeps accountability. Draw a sharp line between innovation and defensibility. Legal teams by and large aren't opposed to change. They are opposed to change that cannot be audited or defended. That is a useful framework for any business buyer, because it shifts the question from "Isn't this exciting?" to "Can we stand behind this in a boardroom or a lawsuit?" One of the strongest questions any buyer can ask is: What happens when a technology is wrong? Every system fails sometimes. The important thing is whether it fails safely. Can a human override it? Does it explain itself? Does it stop when it lacks enough information? Can you audit the outcome later? Those are not just HR questions. They are board-level questions. They apply anywhere technology influences decisions. 6. Trust is the gold standard Business leaders buy tech for trust and impact. If a tool helps people make better decisions and work more clearly, it earns its place. If it can't be explained, it will struggle to survive. The goal is to make work more consistent and more effective. Final thoughts That is a standard any business can understand. And it is the standard I believe every serious company should demand. AI will be judged less by what it can do than by what companies can stand behind. Successful businesses will build systems people can understand, leaders can defend, and teams can rely on. Treat explainability, oversight, and accountability as the price of entry for serious innovation. Get 1 Smart Business Story delivered straight to your inbox when you subscribe to Inc.'s free daily newsletter.

Companies Are Facing Legal Battles For Misusing AI -- Here's How to Avoid Being One of Them

There's nothing wrong with companies using AI -- it's unmanaged and unregulated adoption that is the real problem. Companies need to keep track of exactly how, when and where they use AI with AI governance documents and policies. Odds are, the government will be asking for them in the next few years. The first wave of artificial intelligence adoption was driven by speed. Companies wanted faster research, faster drafting, faster customer service, faster sales, faster decisions. In boardrooms and management meetings, AI was presented as a productivity tool, a cost-saving mechanism and, in some cases, a competitive necessity. That phrase is already giving way to something more serious. The next wave of AI will not be defined only by what companies can automate. It will be defined by what they can explain, defend and govern. That is where many businesses are dangerously unprepared. For all the excitement around AI, a basic legal question remains unanswered in many organizations: If an AI system produces a harmful, biased, false or commercially damaging outcome, who is responsible? Not theoretically. Not philosophically. Legally. The vendor? The employee? The board? The executive who approved the tool? The department that deployed it? The company that relied on it? AI has moved beyond the technology department One of the biggest mistakes companies are making is treating AI as a technology implementation issue. It is not. AI now touches contracts, employment decisions, customer communications, intellectual property, data protection, financial analysis, regulatory compliance, marketing claims, dispute resolution and board-level risk. That means AI is no longer simply a matter for IT teams. It has entered the legal and commercial architecture of the business. The EU AI Act, which entered into force in 2024 and becomes broadly applicable from 2026, is one example of how regulators are moving AI from innovation language into legal obligations. The act introduces a risk-based framework and imposes obligations depending on how AI systems are used, especially where they are classified as high-risk. That matters even for business outside Europe, because regulation in one major market often becomes a global reference point. Companies that operate internationally, serve European customers or use AI outputs in regulated environments cannot afford to treat AI governance as a local compliance footnote. The legal direction is clear: AI is moving from experimentation to accountability. The problem is not AI use, it is uncontrolled AI use There is nothing inherently wrong with companies using AI aggressively. In fact, those that refuse to engage with it may fall behind. The danger lies in unmanaged adoption. Many organizations already have employees using AI tools informally to draft documents, summarize confidential material, prepare client communications, analyze data or generate business ideas. Some of that use is productive. Some of it may also be creating legal exposure that senior leadership cannot see. This is the uncomfortable truth: Many companies do not know where AI is being used inside their own business. They cannot govern what they have not mapped. They cannot defend what they have not documented. And they cannot control risk they have allowed to spread invisibly through workflows, teams and departments. This is not a hypothetical concern. Courts are already confronting the consequences of professionals relying on AI outputs without proper verification. Reuters has reported several cases involving AI-generated fictitious legal citations and judicial scrutiny, including fresh incidents in 2026 where lawyers faced serious professional consequences for failing to verify AI-produced material. The lesson for business is wider than the legal profession: When AI produces a false output, the organization may still own the consequence. AI governance is the new corporate governance For years, corporate governance has focused on oversight, accountability, risk, ethics and transparency. AI now belongs inside that same conversation. This is not because every board member needs to become a technologist. They do not. But boards and executive teams must understand enough to ask the right questions. Where is AI being used? What data does it process? Which decisions does it influence? Is human oversight meaningful or cosmetic? Who signs off on deployment? What happens when the system fails? Can the company produce evidence that it acted responsibly? These are no longer technical questions. They are governance questions. The direction is important. Serious AI adoption requires structure. It requires accountability and a record of decision-making. In the next few years, companies will not only be asked whether they used AI. They will be asked whether they used it responsibly. That distinction will matter. The legal risk is shifting from output to process Many leaders still think of AI risk in terms of bad outputs, a hallucinated answer, an inaccurate summary, a flawed prediction or a biased recommendation. Those risks are real. But the deeper legal issue is process. If a company uses AI in hiring, customer advice, credit assessment, health, legal analysis, financial decisions or regulated services, the question is not only whether the output was correct, but also whether the process around that output was defensible. Was the tool appropriate for the task? Was the data lawful and reliable? Was there human review? Were staff trained? Was the risk classification clear? Was the decision documented? Was the customer, employee or regulator misled? This is where legal exposure grows. A company may survive an AI mistake. It may not survive evidence that it had no governance system, no ownership structure and no meaningful oversight. The future legal test will not be perfection. No technology is perfect. The test will be whether the company acted with discipline, transparency and reasonable control. What businesses should do now The first step to stop treating AI governance as a policy document that sits somewhere in a compliance folder. Governance must be operational. Every company using AI should begin with a clear internal map of where AI is being used, by whom and for what purpose. This includes formal tools approved by management and informal tools used by employees. Without that map, leadership is guessing. The second step is classification. Not all AI use carries the same risk. Using AI to brainstorm marketing ideas is not the same as using AI to screen job applicants, draft legal submissions, advise customers or influence financial decisions. High-impact use cases require stronger oversight, clearer approval and better documentation. The third step is ownership. Every AI system should have a named business owner. Not just an IT contact. Not just a vendor. Someone inside the organization must be responsible for its use, limits, monitoring and escalation. The fourth step is documentation. Companies should be able to show why a toll was selected, what risks were considered, what safeguards were introduced, who approved it and how outputs are reviewed. In the AI era, evidence of responsible process may become as important as the outcome itself. The fifth step is training. Employees do not only need to know how to use AI. They need to know when not to trust it. That requires legal literacy, commercial judgment and an understanding of where human review is essential. These steps are not designed to slow business down. They are designed to make speed defensible. Legal should not be brought in only after something goes wrong. It should be part of how AI systems are selected, structured and deployed. The best legal function in the AI era will not simply say no. It will help design the conditions under which the business can say yes safely. This is where legal and commercial meet. The companies that understand this now will not merely avoid risk. They will build trust infrastructure that the next era of business will depend on.

Governance Gives AI Agents Permission to Grow Up | PYMNTS.com

Evidence for this comes from Nubank researchers who recently reported production deployments of AI customer-support agents across card delivery, debt management, credit-limit support, card management and product explanations. The work spans a customer base of more than 100 million users and shows how AI agents are being pushed into real customer workflows, not just internal productivity tools. In one card-delivery deployment, Nubank said large-scale A/B testing produced a 37 percentage-point improvement in AI transactional Net Promoter Score and a 29 percentage-point gain in self-service rate compared with prior agent variants. That is the production proof point banks and FinTechs have been waiting for. The industry has spent the past two years testing copilots, chatbots and internal assistants. Now the focus is shifting to agents that can handle higher-volume, higher-stakes workflows with measurable impact. Other financial services players are moving in the same direction. Experian launched an Agent Operating System inside its Ascend Platform, positioning it as a way for financial institutions to scale agentic AI across the lending lifecycle with controls, auditability and human oversight. HSBC and Google Cloud also announced a multi-year AI partnership expected to support more than 200 new AI use cases over two years, including wealth management and financial crime risk management. The common thread is that AI is moving into regulated workflows. Credit, card servicing, debt support, anti-money laundering, fraud and lending operations are all areas where decisions must be explainable, monitored and auditable. That raises the bar for deployment. Banks and FinTechs will need clear escalation paths to humans, detailed call logs, data lineage, policy testing and outcome monitoring. In payments, AI customer-service records could increasingly become part of the evidence trail for chargebacks, fraud claims and consumer complaints. The story is no longer whether financial institutions will use AI agents. The story is how quickly they can scale them without losing control of the workflow. Governance Becomes the Scaling Test for Agentic AI The first wave of production AI agents shows that financial services firms can move beyond pilots. The next issue is whether companies across industries can govern those agents once they start acting on their own. That was the central theme of a recent SSON report on agentic AI governance, which argued that autonomous systems are changing the role of enterprise oversight. AI no longer just recommends an action to a human. In more advanced deployments, it can execute tasks, trigger workflows and interact with core business systems. That makes governance less of a compliance checkpoint and more of an operating requirement. The report, based on discussions at SSON's Agentic & Applied AI for the Enterprise conference, said companies are rethinking a basic management question: Who owns the agent? In older automation programs, governance often came near the end. Teams identified an opportunity, built a tool, tested it and then submitted it for approval. That sequence worked better for rules-based systems. Agentic AI creates a different problem because risk continues after launch. As LinkedIn's Bhupinder Singh Narang put it in the report, governance is no longer just a policy document. It has become an engineering problem. That means controls have to be built into the workflow from the beginning, including audit logs, scoped permissions, approval thresholds, rollback mechanisms and continuous monitoring. The lesson applies well beyond financial services. In human resources, AI agents could screen candidates, answer employee questions or trigger payroll-related workflows. In procurement, they could negotiate with suppliers or reorder goods. In shared services, they could handle finance, customer support, claims or back-office tasks. Each use case creates the same basic issue: An agent that acts independently needs boundaries. The report also reframed governance as an enabler of scale, rather than a brake on innovation. Too much control can slow experimentation. Too little can create operational and reputational risk. The practical answer is risk-based governance, where low-risk use cases move quickly inside clear guardrails, while higher-risk workflows require deeper review. For companies trying to scale agentic AI, the checklist is becoming clearer: assign a business owner, technical steward and risk sponsor for every agent; use trusted data; document actions; monitor outcomes; and make sure humans can intervene. The scaling race will not be won by the firms that deploy the most agents. It will be won by the firms that know exactly what their agents are allowed to do. For all PYMNTS AI coverage, subscribe to the daily AI Newsletter.