AI Companies Race Toward Recursive Self-Improvement as Researchers Warn of Control Risks

AI Labs Approach Critical Threshold in Autonomous Development

Recursive self-improvement, once confined to theoretical discussions among computer scientists, has moved closer to reality as leading AI companies report major advances in automated model development. Anthropic revealed this week that its AI model Claude now leads 26% of the company's model research and development work, completing most tasks "end-to-end from a high-level prompt" while remaining under human supervision1

1

. The development marks a significant milestone in the race toward RSI, where AI models find ways to improve themselves and build their successor without human guidance.

OpenAI announced it has developed an automated "research intern" capable of carrying out well-defined research tasks under human direction, including "tasks that would take a skilled researcher a few days"1

1

. The company aims to create an automated AI "researcher" by March 2028, though it cautioned that "rapid RSI is not necessarily an outcome we should pursue"5

5

. An OpenAI employee told The Information the company has largely automated training new experimental models, with AI systems running experiments as directed by humans and correcting much of their work3

3

.

Industry Leaders Sound Alarm on Pace of Development

The progress toward recursive self-improvement has prompted unprecedented calls from AI industry leaders to slow development. Dario Amodei, co-founder and CEO of Anthropic, published an essay on September 12 stating "we must slow the pace at which we improve the capabilities of AI models"4

4

. He warned that RSI "could outrun our ability to understand and control these systems, and so must be pursued very carefully, if at all"4

4

.

Sam Altman of OpenAI responded on X, writing "I agree with Dario that we need to pace the frontier"4

4

. Elon Musk concurred, stating "Dario is right," while Google DeepMind co-founder Demis Hassabis backed the proposal's direction4

4

. The unified response from competing executives underscores growing industry concern about the risk of self-improving AI.

Understanding the Mechanics and Risks of RSI

Recursive self-improvement describes a process where an AI helps build a successor that is better at developing AI, which then helps build an even more capable version. "The really important thing here is that as AI is doing more of it, it gets faster, because AI operates just much, much more quickly than the humans do," explained Anthony Aguirre, president and CEO of the nonprofit Future of Life Institute and physics professor at the University of California, Santa Cruz1

1

.

The concept traces back to 1965, when British mathematician Irving John Good described it in a paper: "Thus the first ultraintelligent machine is the last invention that man need ever make, provided that the machine is docile enough to tell us how to keep it under control"3

3

. The fear around RSI centers on runaway superintelligence emerging from the process, where AI could become more capable without becoming more reliable or controllable—known as the alignment problem4

4

.

Current State: Automation Without Full Autonomy

While AI companies get closer to recursive self-improvement, fully automated model improvement remains elusive. John Thickstun, assistant professor of computer science at Cornell University, noted that "we have already, for years, been using these models in supportive roles for creating the next version of these models"5

5

. The harder challenge is getting AI to independently decide what to investigate—which problems matter and which ideas merit testing4

4

.

Elon Musk stated in March that for xAI's Grok models, "humans are gradually getting less and less in the loop" on model improvement and "every successive model is built by the one before it," though the process was not yet fully automated. He suggested that target might be reached by year's end, "but not later" than 20275

5

.

Competing Visions and Alternative Approaches

Two London-based researchers, Edward Hughes and Louis Kirsch, left Google in December to found Inherent, a start-up building an AI system called Faraday designed to build better AI systems2

2

. Faraday gathers mountains of data capturing daily activities of researchers—emails, instant messages, meeting transcripts—to build improved versions of itself2

2

.

Two Silicon Valley start-ups valued at $4 billion each are pursuing this goal, including Recursive Superintelligence founded by Jeff Clune, a veteran of OpenAI and Google2

2

. "Now is the time to take these ideas, which we have been incubating in the lab for decades, and start to really scale them up," Clune said2

2

.

Microsoft appears to be taking a different path. Mustafa Suleyman, CEO of Microsoft AI, has said the company is moving toward "humanist superintelligence," or advanced AI capabilities that serve human needs5

5

.

Expert Warnings and Existential Concerns

Aguirre delivered one of the starkest warnings about fully autonomous RSI: "I think this is probably the worst idea in the history of humanity to do this. And yes, they're doing it"5

5

. His concern reflects fears that AI researchers are worried about systems that could resist shutdown or modification if staying operational helped accomplish their goals3

3

.

Anthropic warned that RSI "might increase the risks of humans losing control over AI systems"3

3

. The company found that some unwanted traits could pass from one model to another through training data, raising concerns that problematic behavior could persist across generations of AI3

3

. Amodei warned that within six to 12 months, more capable AI agents could take over the internet through a "botnet," potentially causing hundreds of billions of dollars in damage4

4

.

Some researchers theorize that self-improving AI systems capable of copying themselves could create digital natural selection, favoring systems best at acquiring compute, money and power to grow fastest—potentially competing with humans for resources3

3

.

What to Watch: Timeline and Safeguards

OpenAI's goal of creating an automated AI researcher by March 2028 provides one concrete timeline for tracking progress1

1

. However, critics argue the milestone represents coding automation rather than impending doom. A new analysis published recently found that AI feedback loops aren't yet hitting RSI benchmarks3

3

.

The key question remains whether safeguards can keep pace with capability advances. OpenAI stated that "whether and how to proceed must depend on our ability to preserve human control and on informed democratic choices about the benefits and risks"5

5

. Training still requires computing resources, energy and time, and useful improvements may become harder to find, suggesting a runaway loop is not inevitable4

4

.

Watch for how AI companies implement the slowdown called for by industry leaders, whether they share transparency metrics as Anthropic has encouraged, and how quickly the percentage of AI-led research work increases across labs. The gap between current automation and full autonomy will determine whether humanity retains meaningful control over AI development.