Anthropic CEO warns AI will test humanity as powerful systems arrive within 1-2 years

Anthropic CEO's grave warning: AI will "test us as a species"

"Humanity is about to be handed almost unimaginable power, and it is deeply unclear whether our social, political, and technological systems possess the maturity to wield it." Why it matters: Amodei's company has built among the most advanced LLM systems in the world. * Anthropic's new Claude Opus 4.5 and coding and Cowork tools are the talk of Silicon Valley and America's C-suites. * AI is doing 90% of the computer programming to build Anthropic's products, including its own AI. Amodei, one of the most vocal moguls about AI risk, worries deeply that government, tech companies and the public are vastly underestimating what could go wrong. His memo -- a sequel to his famous 2024 essay, "Machines of Loving Grace: How AI Could Transform the World for the Better" -- was written to jar others, provoke a public debate and detail the risks. * Amodei insists he's optimistic that humans will navigate this transition -- but only if AI leaders and government are candid with people and take the threats more seriously than they do today. Amodei's concerns flow from his strong belief that within a year or two, we will face the stark reality of what he calls a "country of geniuses in a datacenter." * What he means is that machines with Nobel Prize-winning genius across numerous sectors -- chemistry, engineering, etc. -- will be able to build things autonomously and perpetually, with outputs ranging from words or videos to biological agents or weapons systems. * "If the exponential [progress] continues -- which is not certain, but now has a decade-long track record supporting it -- then it cannot possibly be more than a few years before AI is better than humans at essentially everything," he writes. Among Amodei's specific warnings to the world in his essay, "The Adolescence of Technology: Confronting and Overcoming the Risks of Powerful AI": Call to action: "[W]ealthy individuals have an obligation to help solve this problem," Amodei says. "It is sad to me that many wealthy individuals (especially in the tech industry) have recently adopted a cynical and nihilistic attitude that philanthropy is inevitably fraudulent or useless." The bottom line: "Humanity needs to wake up, and this essay is an attempt -- a possibly futile one, but it's worth trying -- to jolt people awake," Amodei writes. "The years in front of us will be impossibly hard, asking more of us than we think we can give."

Anthropic CEO Dario Amodei's warning from inside the AI boom

Dario Amodei just gave the kind of warning AI pragmatists love: urgent, sweeping, and delivered from a podium built out of venture capital. In a sprawling, 38-page essay, "The Adolescence of Technology: Confronting and Overcoming the Risks of Powerful AI," posted Monday, the Anthropic CEO lays out a civilizational-risk map -- bioterror, autocracy, labor upheaval, and further wealth concentration. He lands on the uncomfortable thesis that the AI prize is so glittering (and its strategic value is so obvious) that nobody inside the race can be trusted to slow it down, even if the risks are enormous. "I believe we are entering a rite of passage, both turbulent and inevitable, which will test who we are as a species," he wrote. "Humanity is about to be handed almost unimaginable power, and it is deeply unclear whether our social, political, and technological systems possess the maturity to wield it." The essay scans like a threat assessment, framed through a single metaphor Amodei returns to obsessively: a "country of geniuses in a datacenter." (It appears in the text 12 times, to be exact.) Picture millions of AI systems, smarter than Nobel laureates, operating at machine speed, coordinating flawlessly, and increasingly capable of acting in the world. The danger, Amodei argues, is that the concentration of capability creates a strategic problem before it creates a moral one. Power scales faster than institutions do. But Amodei's essay also reads as a positioning statement. When the CEO of a frontier lab writes that the "trap" is the trillions of AI dollars at stake, he's describing the very gold rush he's helping lead, while pitching Anthropic as the only shop that's worrying out loud -- a billionaire CEO begging society to impose restraints on a technology his company is racing to sell. So while the argument may be sincere, the timing is also marketing-grade; on the same day that Amodei's essay dropped, Claude, Anthropic's chatbot, got an MCP extension update. The risks he catalogs fall into five buckets. First, autonomy. Second, misuse by individuals -- particularly in biology. Third, misuse by states, especially authoritarian ones. Fourth, economic disruption. And finally, indirect effects -- cultural, psychological, and social changes that arrive faster than norms can form. Threaded through all of it is the reality that no one -- or no company -- is positioned to self-police. AI companies are locked in a commercial race. Governments are tempted by growth, military advantage, or both. And the usual release valves -- voluntary standards, corporate ethics, public-private trust -- are too fragile to carry that load. He argues that powerful AI "could be as little as 1-2 years away" and says a serious briefing might call it "the single most serious national security threat we've faced in a century, possibly ever," echoing previous warnings. Amodei believes powerful AI can deliver extraordinary gains in science, medicine, and prosperity. He also believes the same systems can amplify destruction, entrench authoritarianism, and fracture labor markets if governance fails. The race continues regardless. His proposed fixes are unglamorous: Transparency laws. Export controls on chips. Mandatory disclosures about model behavior. Incremental regulation that's designed to buy time rather than freeze progress. "We should absolutely not be selling chips" to the CCP," he writes. He cites California's SB 53 and New York's RAISE Act as early templates, and he warns that sloppy overreach invites backlash and "safety theater." He argues repeatedly for restraint that is narrow, evidence-based, and boring -- the opposite of the sweeping bans or grand bargains that dominate AI discourse. Amodei might want credit for saying the quiet part out loud, that the AI incentive structure makes adults rare and accelerants plentiful. Yet he's still out here building the "country of geniuses in a datacenter" and asking the world to believe his shop can both sell the engine and mind the speed limit -- before any potential crash. He calls this "the trap," and he's right. He's also standing in it, collecting revenue.