AI leaders call for kill switch as race to superintelligence accelerates beyond control

AI leaders would like to stop racing. Let's make that possible.

Frontrunners in the race to build superintelligence say the field is moving dangerously fast. Anthropic co-founder Jack Clark and co-writers argue that machines run by artificial intelligence may soon have the ability to enhance themselves with little or no human oversight, in a fashion that could rapidly get out of control. Echoing what experts have been saying for years, they say humanity needs to establish and retain "the option to slow or temporarily pause frontier AI development." In other words, we need an "off switch" for AI. Competitor OpenAI also recently repeated its acknowledgement of a potential for catastrope. It expressed interest in global capacity for "coordinated action, including slowing frontier development when needed." In January, Google DeepMind's Nobel-laureate CEO, Demis Hassabis, shared that he would go for a readiness to pause if everyone else would. Even Elon Musk, who signed a 2023 letter calling for a six-month pause, admitted in December that AI was the stuff of his nightmares, and that he would slow it down if he could. If this is all marketing hype -- most of it from companies with trillion-dollar initial public offerings on the line -- it's a heck of a bluff, and we should call them on it. Three years ago, hundreds of experts, including several of the world's most cited living scientists, warned that general-purpose AI systems threaten humanity with extinction. Last year, scientists and policymakers from all across the political spectrum called for a global ban on the development of artificial superintelligence -- AI that broadly and substantially surpasses human abilities -- in a letter that garnered more than 100,000 signatures. Now, Anthropic has added to the mountain of evidence that AIs can now reliably complete many software tasks that would take human experts multiple hours. This time horizon is doubling every four months -- true exponential growth. About 80 percent of the code at Anthropic is now written by its AI. This frees researchers to focus on setting higher-level objectives, but the AI is also "getting better at proposing its own experiments." The era where humans meaningfully contribute to AI research may be short-lived. These revelations come on the heels of several groundbreaking milestones. A few months ago, Anthropic's Claude Mythos AI proved so good at computer hacking that banking and political figures praised the company's decision to withhold it from wide release -- while cyber defenders scrambled to address the thousands of security vulnerabilities it found, including in every major operating system and web browser. More recently, a model from OpenAI produced a novel mathematical proof on a problem that had stumped Earth's mathematicians for decades -- then another AI solved nine more. Forget about leaving superintelligence to some future generation to worry about; we can no longer safely assume this problem can be left to a future administration. The U.S. government must lead, using its position of strength to negotiate and implement a globally enforceable halt, because an AI does not need to run in an American data center to threaten an American life. The White House's recent moves to establish federal evaluation of new frontier AI models before they are released are a step in the right direction, but only a step. The models we most need to worry about won't wait for their owners to release them; existing AIs have already proven adept at breaching their training containment. There is no need to ban continued development of limited AIs specialized for purposes like revolutionizing medical research. It is only this reckless race toward superintelligence, and self-enhancing AIs, that the whole world needs to turn away from. The good news is that we won't be starting from zero. There are plenty of signs that our adversaries are willing to negotiate on AI, just as we negotiated with the Soviet Union to avert nuclear catastrophe. And any workable arrangement would tap the wealth of expertise in the diplomatic, intelligence and regulatory agencies of the U.S. and its partners. My colleagues at the Machine Intelligence Research Institute have also been studying governance and agreements to avoid the worst outcomes -- and the verification mechanisms to back them. It helps that it's a lot harder to make advanced AI chips than it is to enrich uranium. Such chips can only be manufactured by a handful of companies, using lithography machines that only one Dutch company knows how to make -- machines that cost as much as a 747 and weigh 150 tons each. Those chips must be brought together by the thousands in data centers that consume as much electricity as a mid-sized city and are visible from space. American leadership can, and must, do what the AI companies can't do themselves -- build an "off switch" that can end their mad rush to superintelligence that threatens us all. Nate Soares is president of the Machine Intelligence Research Institute and co-author of the New York Times bestselling book, "If Anyone Builds it, Everyone Dies".

Living without an AI kill switch

OXFORD, England - It has long been clear that slow-moving governments are not keeping pace with rapid AI progress. But Anthropic's announcement that its new Claude Mythos Preview model could identify and exploit vulnerabilities in every major operating system and web browser has underscored the perils of failing to regulate this technological revolution. Even enthusiastic deregulator U.S. President Donald Trump conceded that there should be a "kill switch" when news about Mythos broke. But such a simple solution no longer exists -- if it ever did. The threat is twofold: malicious humans and AI models run amok. The latest AI agents have already turbo-charged cyberattacks and bioengineering risks, accelerating the development of new weapons, including potentially catastrophic "mirror organisms." Now, with the arrival of Mythos (as well as other highly capable models that have since been made available to vetted partners), a consortium of authorized users -- including some U.S. government agencies and a handful of trusted companies -- is scrambling to secure critical software.