Amazon challenges human-in-the-loop AI governance as tech giants shift to accountability models

Amazon Rejects Human-in-the-Loop AI Governance

Amazon is challenging one of the most widely accepted principles in AI governance. Eric Brandwine, distinguished engineer and VP at Amazon Security, told The Register that human-in-the-loop AI governance is not the gold standard companies believe it to be. "Humans are not terribly consistent," Brandwine said. "Human-in-the-loop isn't necessarily the gold standard." 1

1

His reasoning draws on normalization of deviance, a concept he presented at AWS re:Invent in 2017. This describes what happens when people take shortcuts over time, and when nothing catastrophic results, the deviant behavior becomes normal. Brandwine illustrated this with emergency rooms where nurses initially respond to every alarm but gradually stop reacting after repeated false alarms. "Literally, someone's life is on the line, and people still struggle to maintain discipline," he said. "That's the human condition." 4

4

When applied to AI agents, this pattern becomes dangerous. "If you put a human inside of this tight loop, and ask them to make approval decisions for agentic tools repeatedly, time after time, they'll do a good job," Brandwine explained. "And then they'll do an okay job. And pretty quickly they'll be doing a poor job." This is why Amazon opposes human-in-the-loop oversight for high-velocity operations. 1

1

Tech Giants Shift to Accountability End to End

Amazon isn't alone in rethinking this approach. Google Cloud COO Francis deSouza announced in April that the industry has moved "from a human-led defense strategy, to a human-in-the-loop defense strategy, to an AI-led defense strategy that's overseen by humans." Google's model now uses an agentic fleet handling routine cybersecurity work at machine speed, with humans providing oversight rather than approving every action. 4

4

Microsoft CEO Satya Nadella argued for "loop learning" instead, where companies turn their workflows and accumulated judgment into AI systems that improve with each use. IBM called for human accountability at all stages of AI development rather than humans in the loop. 1

1

Amazon's alternative is accountability end to end, where human identity and ownership track through the entire workflow even when humans aren't directly approving every step. All agents at Amazon have independent identities, and activity logs show "this agent did this on behalf of Eric," not "Eric did this." If an agent causes an outage, the person who deployed it remains responsible. 4

4

Source: BleepingComputer

Shadow AI Creates Widespread Security Risks Posed by AI Agents

The security risks posed by AI agents extend far beyond model vulnerabilities. According to a 2026 CSA survey, 82% of organizations discovered at least one AI agent created without the knowledge of security, IT, or governance teams in the past year, and 41% found this happening multiple times. Even more concerning, 65% of organizations experienced AI-related security incidents in the past year, with 61% reporting exposure or mishandling of sensitive data. 3

3

Shadow AI has shifted from a data leakage concern to an access control risks problem. The threat isn't about what employees type into AI tools but which AI agents are running inside the organization, what enterprise systems they're connected to, and what actions they're authorized to take. Custom assistants, coding agents, and workflow automations are being created across departments through browser extensions, SaaS-native features, and custom scripts. 5

5

Related Stories

AI Agents Expose Critical Identity Security Gap as 69% of Enterprises Share Credentials

AI Agents Trigger Security Crisis: 65% of Enterprises Report Incidents Amid Scaling Challenges

Enterprise AI Success Now Hinges on Governance and Human Oversight, Not Just Advanced Technology

AI Agents Outpace Governance Frameworks as Enterprise Security Risks Multiply

Legacy Infrastructure Vulnerabilities Enable Attack Paths

At the Gartner Security & Risk Management Summit, security experts revealed how attackers circumvent AI security programs by exploiting legacy infrastructure vulnerabilities to hijack AI agents. Roughly 71% of organizations are piloting AI agents across enterprise applications, and 31% have already moved them into production workflows. Yet AI agents authenticate through existing identity providers, store data in existing cloud buckets, and inherit permissions from existing IAM roles—carrying whatever security debt existed before AI deployment. 2

2

Source: SiliconANGLE

A real-world attack scenario demonstrates the danger. A customer success team's AI Co-Pilot on AWS Bedrock connected to Salesforce data in an S3 bucket. An attacker exploited CVE-2025-24813, a remote code execution flaw in Apache Tomcat added to CISA's Known Exploited Vulnerabilities catalog. By compromising Active Directory credentials, the attacker moved laterally to a developer's workstation, harvested AWS access keys, and gained access to the S3 bucket containing customer records. The AI agent's excessive privileges did the rest.

According to Infosecurity Magazine, 70% of organizations grant their AI systems more privileged access than a human in the same role. Organizations with over-privileged AI agents reported a 76% incident rate, compared to just 17% for those enforcing least-privilege policies.

Identity-Centric Controls Become Critical for AI Security

AI agents have become non-human identities that most enterprises lack security and governance models for. They retrieve information, trigger workflows, update records in Salesforce, Snowflake, and GitHub, write and deploy code, and take actions across multiple systems—sometimes on behalf of humans, sometimes autonomously. An agent might be created by one team, used by another, connected to five different applications, and running on credentials provisioned for a completely different purpose. 3

3

Getting control starts with visibility. Security teams need to answer critical questions: Who owns this agent? Who can invoke it? What systems is it connected to? What credentials does it use? What can it read, write, delete, or execute in each target application? A sales prep agent only needs read access to CRM records, not the ability to delete database tables. When permissions don't match an agent's actual purpose, that gap is where real risk lives. 3

3

Source: CRN

Brandwine described practical challenges like "goal-seeking behavior," where an agent asked to upgrade a database becomes fixated on deleting and recreating it. This isn't prompt injection—the agent simply gets stuck on the wrong action. What works is telling the agent why it cannot perform an action and including constraints like "don't cause a production impact" in the prompt. Amazon's approach uses layered policies: static guardrails prohibiting destructive actions, maximum privilege sets for each agent, and dynamically scoped policies based on specific tasks. 4

4

The race to govern what AI agents can access has triggered major acquisitions, with 1Password buying access-governance startup Apono for an estimated $250 million to $300 million. As Brandwine noted, "We have millennia of experience with humans. Agentic AI is a very, very new field." The fundamental difference is that humans fear consequences like losing a job, while agents do not—and attackers are exploiting that gap. Organizations must treat AI agents like any other identity with continuous discovery, defined ownership, scoped access, and lifecycle management to prevent data exfiltration and lateral movement. 4

4