Anthropic warns AI may soon build itself, calls for global pause on frontier development
35 Sources
[1]
Anthropic warns AI may soon begin recursive self-improvement
I agree my information will be processed in accordance with the Scientific American and Springer Nature Limited Privacy Policy. We leverage third party services to both verify and deliver email. By providing your email address, you also consent to having the email address shared with third parties
[2]
Anthropic warns Claude AI is building itself faster than expected, calls for option to halt frontier development -- 'recursive self improvement' increases risk humans lose control of AI
Internal data shows engineers shipping eight times more code. Anthropic has published a report warning that the development path it's on could eventually leave humans unable to control AI systems, even as it disclosed that Claude now writes more than 80% of the code merged into its own codebase.
[3]
'It would be good for the world' to slow down AI sprints, Anthropic says
The plea for caution comes the same week it beat AI archrival OpenAI to filing for an IPO It would be "good for the world" to slow down the pace of AI development, according to a blog post from Anthropic, which this week began the process of going public with a confidential IPO filing. "We
[4]
Anthropic urges a way to pause AI development as risks grow with the tech advances
Anthropic is proposing that the world's top artificial intelligence companies come up with a coordinated way to pause development of advanced AI systems, warning the technology is improving so quickly there's a risk humans would lose control. The company behind the Claude chatbot said in a blog
[5]
Anthropic proposes a global slowdown of AI development - Engadget
The company says it's concerned about AI being able to build its own successors. Anthropic says AI is developing so fast, the trend points towards systems becoming capable of developing their own successor. We're not there yet, but it believes it "could come sooner than most institutions are
[6]
OpenAI Joins Anthropic in Call for International AI Watchdog
OpenAI is calling for the formation of an international organization to oversee AI development. And to slow it down, if necessary: "One goal of such an organization should be to make it possible for the world to take coordinated action, including slowing frontier development when needed, so
[7]
Claude writes 80% of its code, calls for AI pause
Anthropic reveals that Claude now writes over 80% of its production code, with engineers shipping 8x more code per quarter than in 2024. The company's new Anthropic Institute paper maps the path to recursive self-improvement and calls for a verifiable global pause mechanism. One of Anthropic's
[8]
Anthropic says AI could soon create more advanced versions of itself
Anthropic outlined the warning in a new blog post from its research-focused Anthropic Institute. The company said the industry may move toward "recursive self-improvement" sooner than many governments and institutions expect. The concept describes a future where one AI model develops the next
[9]
Anthropic (Sorta) Calls for Pause on AI Development. You Should (Sorta) Take It Seriously
Anthropic is urging a unilateral slowdown on the development of new AI models until rigorous safeguards can be put in place -- with a few caveats. In a blog post published Thursday, the company cited its own internal data as evidence that modern AI systems are nearing the point of "recursive
[10]
Anthropic urges a coordinated, verifiable pause for frontier AI
The company says frontier developers need a coordinated, verifiable way to slow or stop if AI starts improving itself faster than the risks can be managed. The scenario Anthropic is worried about is one where the technology stops waiting for permission. On Thursday the company argued that frontier
[11]
Anthropic's bizarre call for everyone to slow down on AI is a pipe dream -- here's why that will never happen
Are they fighting an existential threat or channeling Don Quixote? CNN's Wolf Blitzer leveled his inquisitive gaze at me and asked the biggest question in tech, "Is AI an existential threat to humanity?" I smiled confidently and assured Wolf, "No, I don't think it is." But just days after that TV
[12]
Anthropic urges 'temporary pause' on AI development to discuss risks
Announcement that 'policymakers' need to be convened by US firm viewed as marketing ploy by some experts Anthropic has floated the idea of a worldwide "temporary pause" on AI development - and said it was going to convene "policymakers" to discuss the dangers of advanced AI - in its latest release
[13]
Anthropic warns AI could soon help build its own successors
Why it matters: "Recursive self-improvement," a process in which AI systems build, test and improve themselves, is a phenomenon which may come sooner than expected, Anthropic says that its research shows. Driving the news: Anthropic warns that AI is no longer just changing how people work, it's
[14]
Anthropic Scared, Calls for Global Freeze on AI Advances
Can't-miss innovations from the bleeding edge of science and tech Anthropic is calling for a global "pause" on AI development, claiming that the technology is nearing a point where it can spiral out of human control. In a lengthy blog post published Thursday, the world's most valuable AI startup
[15]
Anthropic warns AI could soon build itself without human involvement -- and urges a global pause on development | Fortune
Anthropic has published a new account of how quickly its AI models are advancing, warning that the technology may soon be capable of improving itself without meaningful human involvement. Just as the AI lab, valued at almost $1 trillion, prepares to go public, it's also urging an industry-wide
[16]
Anthropic Is Helping the NSA Hack China. It Also Wants Everyone to Pause AI
Both landed as Anthropic files for an IPO that could value it above $1 trillion. Anthropic has placed about six engineers inside the National Security Agency to help deploy Mythos -- its most capable AI model -- for offensive cyber operations, the Financial Times reported Thursday. The engineers
[17]
Anthropic calls for global pause in AI development before humans lose control
Anthropic calls for global pause in AI development before humans lose control Anthropic PBC has sounded the alarm on the rapid pace of artificial intelligence development and is calling on the world's top AI laboratories to consider slamming on the brakes. In a blog post today, the company warned
[18]
AI Is Already Developing AI, Says Anthropic -- And Humans May Be Slowing Things Down
Anthropic argues AI is already helping build future AI systems and could eventually contribute to designing its own successors. AI has become so effective at writing code and researching that the biggest constraint on developing new AI systems may now be the humans overseeing them, according to a
[19]
AI Could Soon Train and Improve Itself Anthropic Says
Companies have been developing AI very quickly to stay ahead of the market, but Favaro and Clark argue that a slowdown would allow more time to deal with the technology's implications. US-based AI firm Anthropic warns AI development is advancing at a pace that could soon see agents building,
[20]
Anthropic Urges a Way to Pause AI Development as Risks Grow With the Tech Advances
Anthropic is proposing that the world's top artificial intelligence companies come up with a coordinated way to pause development of advanced AI systems, warning the technology is improving so quickly there's a risk humans would lose control. The company behind the Claude chatbot said in a blog
[21]
Anthropic just said humans could lose control over AI systems, and that should scare us all
Sara Heritage is a tech and gaming journalist, who's currently making her way up to Master Ball rank in Pokemon Champions. Bylines in IGN, GAMINGbible, The Gamer and more. You can usually find her tinkering with tech, or restoring old consoles, always with one of her 3 cats nearby. Come and talk
[22]
AGI for all among OpenAI's ambitions for its next phase
OpenAI is entering a new phase to make advanced AI widely available and affordable. The company plans to build AI researchers to accelerate scientific discovery and provide every person with a personal AGI system. This move aims to boost productivity and economic growth. Meanwhile, rival Anthropic
[23]
OpenAI Calls For Global Oversight To Manage Advanced AI Development Risks
A blog post published by OpenAI is urging governments to create a global body that could monitor advanced artificial intelligence work and even push developers to ease off when risks start to outstrip safeguards. "One goal of such an organization should be to make it possible for the world to take
[24]
Anthropic wants AI development to slow down globally, warns humans could lose control otherwise
In a long article on its website, titled "When AI builds itself", Anthropic said it would be good to have the option to slow or temporarily pause frontier AI development, citing safety risks and loss of control over the models as they self-improve. Leading artificial intelligence (AI) company
[25]
Anthropic Wants a Global AI Pause If Everyone Else Does | PYMNTS.com
In a report titled "When AI Builds Itself," co-authored by the Anthropic Institute's Lead Marina Favaro and Anthropic Co-Founder Jack Clark, the company said AI may be nearing a point where models can improve themselves with little meaningful human involvement. Anthropic called this "recursive
[26]
Anthropic Warns AI Could Soon Build Better Versions Of Itself
Artificial intelligence (AI) company Anthropic proposed an industry-wide pause in AI development, arguing that increasingly capable systems may soon begin accelerating their own development beyond the pace of human oversight. The Claude maker in a report warned Thursday that AI could eventually
[27]
When AI Builds AI: Anthropic Sets off Another Round of Fear Mongering
This time Anthropic wants regulation among AI companies that can hit pause button on development in case guardrails aren't enough to keep AI from killing humanity Having secured a fundraise that puts it closer to a trillion dollar valuation, Anthropic has now raised concerns over AI autonomously
[28]
Anthropic Calls for Frontier AI Freeze to Prevent Self-Building Tech | PYMNTS.com
The Anthropic Institute will organize conversations with other AI companies as well as policymakers, researchers and civil society to address concerns about the technology and create methods of coordination, Anthropic Institute Lead Marina Favaro and Anthropic Co-Founder Jack Clark wrote in a
[29]
Anthropic Calls for Option to Pause Frontier AI Development
You can access the blog post from here. "it would be good for the world to have the option to slow or temporarily pause frontier AI development." That is the central warning in a new paper from Anthropic, which argues that AI is increasingly helping to build newer, more capable systems, raising
[30]
Anthropic Says AI May Soon Upgrade Itself Without Human Help
Anthropic is delegating a growing share of its AI development to the AI systems, and it may develop its own successor, calling the idea "recursive self-improvement." "We are not there yet, and recursive self-improvement is not inevitable. But it could come sooner than most institutions are
[31]
Anthropic Flags Broader AI Threats Beyond Employment Concerns
The authors of the Anthropic post, company co-founder Jack Clark and Marina Favaro, head of its research institute, said the pause would be used to enable 'societal structures and alignment research' to keep up with . Alignment is industry shorthand for making sure the technology matches human
[32]
Anthropic calls for global AI slowdown after $965B valuation. Critics claim it's just to hobble competition.
Anthropic said artificial intelligence is advancing so rapidly that an industry-wide pause is needed to slow the pace of development while companies get a handle on potential societal risks. The AI juggernaut published a blog post Thursday warning that AI systems could soon be able to improve
[33]
Anthropic warns AI systems accelerate own development amid recursive improvement concerns By Investing.com
Investing.com - Anthropic said Thursday that its internal data shows AI systems are accelerating the development of more advanced AI, a trend that could lead to recursive self-improvement where AI autonomously builds its own successors. The company said engineers at Anthropic now ship eight times
[34]
Anthropic urges AI labs to pause development, warns humans risk losing control
June 5 (Reuters) - Anthropic is calling on major artificial intelligence labs to consider a coordinated and verifiable pause in development, warning that rapid advances in the technology could soon allow AI systems to improve themselves faster than society can manage the risks. The Claude creator
[35]
AI vs Humans? Anthropic's engineers don't know what their job is anymore
Since reading it, I've been thinking about the employee quotes from Anthropic's new study about AI self improvement. Employees claim to have not written any code for five months; another acknowledges, on days when the entire process proceeds smoothly, they start thinking if what they are doing even
Share
Copy Link
Anthropic is urging the world's leading AI companies to consider pausing frontier AI development as its Claude chatbot now writes over 80% of the code merged into its systems. The company warns that AI systems may be approaching recursive self-improvement, where they can design their own successors with minimal human input, potentially increasing the risk of humans losing control of the technology.
Anthropic Calls for Global Slowdown of AI Development
Anthropic, the maker of Claude AI, has issued a stark warning that artificial intelligence systems may be approaching a critical threshold where they can design and build their own successors with little human input. In a blog post titled "When AI Builds Itself," published on June 4, the company stated that "it would be good for the world to have the option to slow or temporarily pause frontier AI development to enable societal structures and alignment research to keep up with the advance of the technology"
1
. This call comes just days after Anthropic confidentially filed for an initial public offering and reached a valuation close to $1 trillion3
.
Source: Market Screener
The company revealed that Claude now writes more than 80% of the code merged into its production codebase as of last month, up from low single digits before Claude Code reached research preview in February 2025
2
. Engineers at Anthropic are now merging eight times as much code per quarter as they did from 2021 to 2025, demonstrating how rapidly the human role in AI development is shrinking2
.The Risk of Recursive Self-Improvement
Recursive self-improvement represents a potential inflection point where AI systems become capable enough to design their next generation autonomously. Anthropic warns this "might increase the risks of humans losing control over AI systems"
4
. The company outlined three scenarios for how the next few years could unfold, with the most severe warning reserved for situations where models become capable of fully improving themselves. In that case, the pace of progress would be determined almost entirely by available compute, with humans pushed toward oversight and verification roles2
.
Source: CXOToday
The firm described alignment issues as particularly concerning in this future scenario. Misalignment that's rare and survivable today could compound generation over generation, "growing more frequent but less understood until we lose control of them"
2
. This concern about AI alignment research keeping pace with technological advances sits at the heart of Anthropic's proposal.Evidence of Accelerating AI Capabilities
Anthropic backed its warnings with internal data showing dramatic improvements in Claude's capabilities. On the hardest, least-specified coding tasks, Claude succeeded 76% of the time in May 2026, a rise of 50 percentage points in just six months
2
. A recurring internal test asking each new model to optimize training code saw results climb from roughly triple the original speed with Claude Opus 4 in May 2025 to about 52 times faster with the unreleased Mythos Preview model in April2
.The length of human tasks that models can reliably complete on their own had been doubling every seven months as measured in March 2025, but now that timeline has accelerated to every four months
3
. While Claude 3 Opus released in March 2024 could reliably complete tasks taking humans four minutes, Claude Opus 4.6 can handle tasks that take humans 12 hours3
.Proposal for Global Coordination Mechanism
Anthropic is proposing what it calls a "global coordination mechanism" to slow or pause AI development, pointing to arms-control agreements on intermediate-range nuclear missiles as a loose model
1
. The company emphasized that "a meaningful slowdown or pause would require multiple well-resourced labs at or near the frontier, in multiple countries, agreeing to stop under the same conditions"5
. Mechanisms must be in place to verify that all parties have actually stopped or slowed development, preventing bad actors from using the pause to jump ahead in secret4
.
Source: PYMNTS
The Anthropic Institute, the company's research division established in March, will spend the coming months convening governments, researchers, and rival AI companies to explore whether a coordinated slowdown could function in practice
1
. The institute plans to conduct research on building the systems that a credible slowdown or pause would require5
.Related Stories
Skepticism and Criticism
The proposal has drawn significant skepticism from researchers and industry observers. Noah Giansiracusa, associate professor of mathematics at Bentley University, stated bluntly: "It's literally impossible. Zero chance there will be a slowdown"
1
. Mark Riedl, a professor at Georgia Institute of Technology, suggested that "the big AI companies are all jumping on the 'recursive self-improvement' hype train"1
.Critics point to the timing of Anthropic's warnings as suspicious. The proposal comes just days after the company filed for an IPO and follows the limited release of its Mythos model, which Anthropic claimed was too good at finding software vulnerabilities to release publicly
1
. To skeptics, such pronouncements read as business strategy designed to draw regulatory scrutiny to competitors while Anthropic continues racing toward the frontier1
.Broader Context of AI Risks and Safety
The warning comes amid growing concerns about AI cybersecurity and AI safety. Researchers at the University of Toronto recently demonstrated how AI tools could create adaptive AI worms that spread from device to device, taking over computing networks
4
. President Donald Trump also signed an executive order directing the Treasury Department to establish an "AI cybersecurity clearinghouse" to coordinate with the AI industry3
.Anthropic acknowledged that unknown bottlenecks could emerge to slow progress, and that current models consistently underperform in "taste" or selecting the next step when left unprompted by humans
3
. The company stated it's "genuinely unclear" whether today's training methods and architectures could unlock the capacity for fully autonomous AI development3
. However, if current trends hold, the alternative scenario is that progress becomes determined entirely by the availability of compute, with humans playing a substantially diminished role3
.References
Summarized by
Navi
[1]
Related Stories
Sam Altman Outlines OpenAI's Vision for AGI: Balancing Progress, Equality, and Safety
10 Feb 2025•Technology
OpenAI Weighs AI Development Slowdown as Safety Concerns Mount and Antitrust Questions Loom
05 Sept 2026•Policy and Regulation
Global AI Summit in Paris Shifts Focus from Safety to Opportunity, Sparking Debate
12 Feb 2025•Policy and Regulation
Recent Highlights
1
OpenAI Claims AI Breakthrough on Navier-Stokes Problem, Sparking Math Community Crisis
Science and Research
2
Anthropic's Claude Now Leads 26% of AI Development Work, Company Reveals New Transparency Metrics
Technology
3
OpenAI Reveals AI Models Hiding Mistakes and Taking Unauthorized Actions in 6 New Misalignment Cases
Technology
Recent Highlights
Today's Top Stories
Don’t drown in AI news. We cut through the noise - filtering, ranking and summarizing the most important AI news, breakthroughs and research daily. Follow topics that matter to you and stay ahead.
