Anthropic's Jack Clark warns AI needs a 'brake pedal' as systems edge toward self-improvement

Anthropic co-founder Jack Clark warns AI needs a 'brake pedal'

Anthropic co-founder Jack Clark has called for the ability to slow progression of artificial intelligence (AI), warning the technology is nearing a point where it could develop without human input. "You want the option to be able to take your foot off the gas and put your foot on the brake", Clark told BBC Newsnight. "Right now, it's like the AI industry has a gas pedal, but it doesn't have a brake pedal." He stressed people, through government policy, need to keep control of AI systems, which will only get more powerful and have broader impacts on society. "The world needs to do some thinking and we need to eventually develop some new regulations that allow us to be confident in these systems," he said. Already, Anthropic's popular chatbot Claude is operating on code of which 80% the system wrote itself. Getting to 100% is possible within two years, Clark said, and "would have huge implications". Clark did not outline how a "brake pedal" for AI research and development could be created, but drew a parallel between AI and the oil boom and barons of the turn of the last century. "Society's response was to come up with a sensible policy and regulatory framework that gave people confidence in oil and the benefits that oil could provide to the world, and meant that you didn't have to worry about the personalities of the people leading the companies", Clark said. "That's clearly where we end up here." Yet, Anthropic this week welcomed an executive order on AI from US President Donald Trump that was relatively hands-off in its directives toward the companies. It did not require AI companies to submit to safety testing by the government, something that remains a voluntary effort. Major AI companies pursuing advances in the technology, including Anthropic, OpenAI and Google, have also not said they will pause their own research. Anthropic has grown so quickly since its founding five years ago that it is preparing to debut on the public stock market. It is poised to be one of the first public listings by a newer AI firm and one of the most valuable stock listings in history, as Anthropic's valuation is estimated by private investors to be nearly $1tn (£745bn). Clark said Anthropic's motivation for publicly discussing the growing capability of AI technology is not to further burnish its reputation with paying customers. He simply wants to "tell the world what we're seeing inside these companies with this unusual technology". Since its founding by chief executive Dario Amodei, Clark and a handful of other executives, Anthropic has positioned itself as outspoken about potential risks stemming from AI. It even engaged in a public dispute with the US Department of Defense over concerns that its AI tools would be used in mass surveillance of Americans and autonomous warfare. "I am worried for my kids if we as a society don't have a serious conversation about what the implications of AI's continued advances mean," Clark told Newsnight. "There are potentially great benefits. There are also risks." Clark added that one of the risks is a disruption to the economy, with fears that AI technology like "agents" - essentially individual AI bots that conduct routine tasks somewhat autonomously could take over certain jobs. Major tech companies have conducted mass layoffs over the last year, often citing the growing ability of AI tools to do the work of hundreds or even thousands of software engineers. Clark said people who are more creative and have more, better ideas may actually have an advantage over AI technology. "There are open questions about whether AI systems can be truly creative... there is not really evidence for that yet", he added. "At Anthropic, we're now limited more by the ability to generate good ideas than the ability to do the engineering to turn those ideas into reality." Nevertheless, Clark suggested a young person who may be feeling that an economy built on AI does not have a place for them should "develop a hobby" and pursue a liberal arts education. "People that are creative and can think broadly, people that read a lot, people that have interests are the ones most benefited by this," Clark said. "Indulge in curiosity and it pays back in how you can use this technology."

Anthropic co-founder warns AI could soon slip beyond our control

Anthropic co-founder Jack Clark said AI agents might soon be able to build and train models themselves and, if that happens, humans could lose control over AI systems. Anthropic co-founder Jack Clark wants the AI industry to pump the brakes before the technology starts further developing itself without human input. Speaking to the BBC, Clark said 80% of Anthropic's coding work is already being done by its AI Claude, and that it could go up to 100% in a couple of years. However, he said "it's a choice" whether AI companies let it get that far without stopping it. "We think this is a topic that the world should be talking more about," Clark said. "The AI industry right now has a gas pedal, but it doesn't have a brake pedal in the car, and we want to do some of the work to build that pedal." This process is called "recursive self-improvement," where an AI is able to improve itself without human input, according to Anthropic in a related blog post from Thursday night. In a recursive model, AI agents, the autonomous workers built by a chatbot, could "become capable enough to build and train models themselves," so Claude "could be continuously improved by Claude," Anthropic said. While recursive AI could bring some good to the fields of science and healthcare, Anthropic warns that it might mean increasing "the risks of humans losing control over AI systems." "If systems are capable of fully building their own successors, the ways we secure them, monitor them and shape their behaviour all grow much more important," the blog post reads. There is evidence within Anthropic's own model that recursion is coming sooner rather than later. It points to the fact that code correction rates by their staff have been falling steadily for the last year, which means there are fewer errors in what Claude is producing. Claude is also able to run its own research experiments when given an open-ended question, such as "Can a weaker model supervise a stronger one?" and come up with its own solutions without human input. "The evidence suggests that the human role is narrowing at each step in the AI development process," the blog reads. Anthropic said its institute will conduct research to build a system to check whether developers have actually stopped or slowed down the move towards recursive AI, it said. However, a real slowdown would require "multiple well-resourced labs at or near the frontier, in multiple countries, agreeing to stop under the same conditions."

Anthropic co-founder warns AI needs a brake pedal, calls for stronger regulations

In a recent blog post, Anthropic also warned about the possibility of "recursive self-improvement." Anthropic co-founder Jack Clark has warned that the world needs stronger regulations for AI and a way to slow its development if necessary. According to Clark, AI technology is becoming more powerful at a rapid pace, and governments need to ensure that humans remain in control. Speaking to BBC Newsnight, Clark compared the current state of AI development to a vehicle that can accelerate but cannot slow down when needed. "You want the option to be able to take your foot off the gas and put your foot on the brake," he said. "Right now, it's like the AI industry has a gas pedal, but it doesn't have a brake pedal." Clark believes policymakers should start preparing for the growing impact of AI on society. He said regulations will be needed to make sure people can trust increasingly advanced AI systems. "The world needs to do some thinking and we need to eventually develop some new regulations that allow us to be confident in these systems," he said. Also read: ChatGPT can now remember more about you with OpenAI's new memory upgrade: Details One area that concerns Clark is AI's growing ability to create and improve software. He revealed that Anthropic's AI chatbot Claude is already operating on code that is mostly written by AI itself. According to Clark, around 80 per cent of Claude's code has been generated by the system, and he believes that figure could reach 100 per cent within the next two years. He also expressed concern about AI's effect on jobs. Clark noted that AI agents, which can perform tasks with limited human involvement, could eventually replace certain types of jobs. In a recent blog post, Anthropic also warned about the possibility of "recursive self-improvement," where AI systems become capable of improving themselves. "Taken far enough, and given enough compute, that trend points to an AI system capable of fully autonomously designing and developing its own successor. This is called recursive self-improvement. We are not there yet, and recursive self-improvement is not inevitable. But it could come sooner than most institutions are prepared for," the company wrote. Also read: Google cuts more jobs in Cloud division while pouring billions into AI: Report "We believe it would be good for the world to have the option to slow or temporarily pause frontier AI development to enable societal structures and alignment research to keep up with the advance of the technology," the company added.