Anthropic pushes for government power to block dangerous AI as self-improvement accelerates
19 Sources
[1]
Anthropic's warning over AI self-improvement has a hidden message -- accelerating development requires more compute before companies ever risk losing control of frontier AI models
The company that just a few weeks ago told us that its Mythos model was too powerful to be released publicly is now saying that we might need to hit the pause button on AI altogether, while also teaching its AI to build itself. On June 4, Anthropic published a report, when AI builds itself, showing
[2]
Anthropic urges US to require safety tests for most capable AI models
WASHINGTON, June 10 (Reuters) - Anthropic called on the U.S. Congress on Wednesday not to block state laws regulating AI unless lawmakers enact a "rigorous" federal law that addresses "catastrophic AI risks," according to a company statement. The company also urged Congress to require AI companies
[3]
Anthropic co-founder Jack Clark warns AI needs a 'brake pedal'
Anthropic co-founder Jack Clark has called for the ability to slow progression of artificial intelligence (AI), warning the technology is nearing a point where it could develop without human input. "You want the option to be able to take your foot off the gas and put your foot on the brake", Clark
[4]
Anthropic wants governments to have the power to block dangerous AI deployments
Anthropic proposed giving governments legal authority to block dangerous AI models, with fines tied to global revenue. A second framework covers worker protections. Anthropic has published two policy frameworks calling for governments to have the legal authority to block dangerous AI deployments
[5]
Anthropic CEO says government should block dangerous AI
Why it matters: Anthropic's ideas for tech regulation and economic disruption from AI go far beyond anything currently under serious consideration in Washington right now. * They're also sure to stir up a new set of accusations that Anthropic is proposing strict rules to lock in its own dominance
[6]
Anthropic CEO calls for FAA-style regulation of powerful AI models: what enterprises should know
In a sweeping new essay titled "Policy on the AI Exponential," Anthropic co-founder and CEO Dario Amodei publicly calls for new government regulations governing the release of powerful AI models -- specifically comparing AI industry to commercial aviation, which follows regulations enforced by the
[7]
Anthropic just proposed taxing itself to pay for the jobs its AI destroys | Fortune
Alongside new policy proposals from the maker of the Claude chatbot, Anthropic CEO and co-founder Dario Amodei published an essay on his personal website that expanded on his position that the government should promise economic support for those financially impacted by AI. The technology could
[8]
Anthropic CEO Warns AI Is Getting Too Powerful -- While Releasing Powerful AI
Anthropic is releasing a legislative proposal on frontier model testing and a policy framework for job displacement alongside the essay. Anthropic CEO Dario Amodei said Wednesday that governments can no longer treat AI regulation as a problem to study and that the United States needs binding
[9]
Anthropic co-founder warns AI could soon slip beyond our control
Anthropic co-founder Jack Clark said AI agents might soon be able to build and train models themselves and, if that happens, humans could lose control over AI systems. Anthropic co-founder Jack Clark wants the AI industry to pump the brakes before the technology starts further developing itself
[10]
Anthropic's Dario Amodei wants governments to have the power to block 'dangerous' AI systems
Anthropic's Dario Amodei wants governments to have the power to block 'dangerous' AI systems Anthropic PBC Chief Executive Dario Amodei (pictured) is calling on the U.S. government to block the deployment of dangerous artificial intelligence models in the same way as it prevents unsafe airplanes
[11]
Anthropic CEO: Government should have power to block dangerous AI deployments
Anthropic CEO Dario Amodei is arguing governments should have the power to block dangerous deployments of artificial intelligence if they don't meet a certain safety standard. Amodei's latest assertion, published in an essay on his website Wednesday, builds upon the company's pro-safety approach
[12]
Anthropic Pledges $200 Million to Research AI's Economic Impact as CEO Suggests Job Loss Solutions
Anthropic on Wednesday joined growing calls for the artificial intelligence industry to find ways to cushion people from the technology's disruptions, announcing an initial $200 million investment to research AI's impact on jobs and the economy. Alongside new policy proposals from the maker of the
[13]
Anthropic CEO warns of AI jobs reckoning as company pledges $200 million for research
Anthropic's Claude models have rapidly become among the most widely used AI tools in software development and enterprise work. The company's coding products, Claude Code and Claude Co-Work, are designed to automate tasks that once required teams of engineers and knowledge workers. As companies
[14]
Anthropic CEO Warns Advanced AI Could Become Too Powerful For Governments Or Corporations To Control -- 'Sh
Anthropic CEO Dario Amodei has warned that increasingly powerful AI systems cannot be safely entrusted to governments or corporations alone, arguing that both require meaningful checks and balances as artificial intelligence becomes more capable. In an essay published Wednesday, Amodei also said
[15]
AI risks are here, time for tough rules: Anthropic CEO Dario Amodei
As artificial intelligence grows more powerful, governments must move beyond transparency measures and introduce stronger regulations to address emerging risks, according to Anthropic CEO Dario Amodei. Amodei said every major technology brings a challenge of balancing innovation and safety. He
[16]
Anthropic Urges Governments to Secure Power to Halt Dangerous AI | PYMNTS.com
Governments should also prepare workers and the economy for the impact of AI and ensure that the financial benefits of the technology are broadly shared, the company said. The AI startup shared its policy proposals in an Advanced Policy Framework and an Economic Policy Framework released
[17]
Anthropic Is Racing Toward An IPO. Its Co-Founder Wants A Way To Slow AI Down, Says The Industry 'Doesn't
As Anthropic races toward what could become one of the most closely watched IPOs this year after SpaceX, co-founder Jack Clark is warning that the artificial intelligence industry may need something it currently lacks: a brake pedal. Speaking to BBC Newsnight this week on a series of topics, Clark
[18]
AI may bring cyberattacks and job losses if left unchecked, warns Anthropic CEO Dario Amodei
Amodei also expressed concern about biological threats linked to AI. Artificial intelligence is becoming an important part of everyday life. It is helping people work faster, companies to improve their services and much more. However, as AI systems are becoming more advanced, concerns about their
[19]
Anthropic co-founder warns AI needs a brake pedal, calls for stronger regulations
In a recent blog post, Anthropic also warned about the possibility of "recursive self-improvement." Anthropic co-founder Jack Clark has warned that the world needs stronger regulations for AI and a way to slow its development if necessary. According to Clark, AI technology is becoming more
Share
Copy Link
Anthropic is calling for governments to gain legal authority to block dangerous AI deployments and require mandatory safety tests as its Claude model now writes 80% of its own code. Co-founder Jack Clark warns AI needs a 'brake pedal' while the company prepares frameworks addressing catastrophic AI risks and AI-driven job displacement ahead of its anticipated IPO.
Anthropic Demands Stronger AI Regulation as Models Approach Self-Improvement
Anthropic has published two comprehensive policy frameworks urging governments to establish legal authority to block dangerous AI deployments and implement economic safeguards as AI systems advance toward autonomous development
4
. The Advanced AI Framework specifically targets AI safety, while the Economic Policy Framework addresses worker protections and labor market disruption. The timing is deliberate: Anthropic co-founder Jack Clark told BBC Newsnight that the AI industry "has a gas pedal, but it doesn't have a brake pedal," warning that technology is nearing a point where it could develop without human input3
.
Source: Digit
The company's Claude chatbot now writes more than 80% of the code merged into its own production codebase, up from low single digits before Claude Code reached research preview in February last year
1
. Jack Clark stated that reaching 100% is possible within two years and "would have huge implications"3
. This acceleration toward recursive self-improvement represents early movement toward a point where models design and build successors without meaningful human control over AI.Catastrophic AI Risks Drive Call for Mandatory Safety Tests for AI
Anthropic called on US Congress on Wednesday to require AI companies to put their most powerful models through independent safety tests, according to a company statement
2
. The framework would apply only to frontier AI models trained using more than 10²⁵ floating-point operations, developed by companies earning more than $500 million in AI revenue or spending more than $1 billion on AI R&D4
. This threshold captures Anthropic itself, OpenAI, Google DeepMind, xAI, and potentially Meta.The company identifies four categories of catastrophic AI risks: biological weapons development, large-scale cyber vulnerability discovery, loss of control over autonomous systems, and AI that automates its own R&D
4
. Anthropic pointed to its unreleased Mythos Preview model, which discovered thousands of high-severity vulnerabilities in every major operating system and browser, as evidence these risks are not hypothetical. Frontier developers would face civil penalties tied to global annual revenue that escalate with repeated violations. "AI capabilities are going to improve rapidly over the coming months," the company wrote, adding that "their governance needs to keep pace"4
.Government Intervention Needed Before Human Control Over AI Erodes
CEO Dario Amodei's new essay argues that policy must change in response to rapid AI development, with Trump's AI executive order needing to go further by requiring mandatory testing for risks related to cybersecurity, biological weapons, loss of control, or automated R&D
5
. "AI is advancing at a lightning pace," Amodei writes, while "policy -- and especially legislation -- moves very slowly"5
.
Source: Axios
Anthropic also urged US Congress not to block state laws regulating AI unless lawmakers enact a "rigorous" federal law that addresses catastrophic risks
2
. The company takes a direct stance against preemption, stating it does not believe Congress should preempt state law unless it enacts federal legislation at least as strong as their proposed framework4
. This positioning distinguishes Anthropic from competitors that lobby against oversight.AI Self-Improvement Raises Questions About Industry Claims
Internal Anthropic data shows that in Q2 2026, the typical engineer is merging eight times as much code per day as in 2024, though the company admits lines of code are a poor proxy for output and the figure likely overstates real gains
1
. On the hardest, least-specified coding tasks, Claude succeeded 76% of the time in May 2026, a rise of 50 percentage points in six months. Google CEO Sundar Pichai said in an April blog post that 75% of new code at Google is now AI-generated and approved by engineers, up from 50% the previous autumn1
.Cognitive scientist Gary Marcus called the piece a "bait and switch" on his Substack, arguing the company had shown faster coding under human direction rather than a system improving itself
1
. Bentley University mathematician Noah Giansiracusa told Scientific American, "I don't think it's a genuine call to slow down." The report publishes no breakdown isolating how much recent capability gain comes from the self-improvement loop rather than from raw compute, more data, and human-led research.Related Stories
Economic Safeguards Target AI-Driven Job Displacement
Anthropic's Economic Policy Framework addresses AI-driven job displacement with proposals for capital accounts, wage insurance, tax incentives, and an expanded social safety net
4
. The company urged US Congress and states to modernize technology used to pay unemployment benefits to prepare for potential layoffs, stating current systems are "not sufficiently prepared for a large labor market shock"2
. Half of Americans fear that the rise of AI could put them or someone in their household out of work, according to a Reuters/Ipsos poll completed on Monday.
Source: VentureBeat
Amodei writes that "it's reasonable to think that AI could produce much larger disruptions to the labor market than previous technologies, and, potentially, more enduring disruptions"
5
. Jack Clark suggested that people who are creative and have better ideas may have an advantage over AI technology, noting "there is not really evidence" that AI systems can be truly creative yet3
. He advised young people feeling uncertain about an AI-driven economy to "develop a hobby" and pursue liberal arts education.IPO Timing Shapes Regulatory Push
Anthropic is preparing for a US initial public offering that would represent one of the most consequential stock market debuts in years
2
. The company's valuation is estimated by private investors to be nearly $1 trillion3
. Jack Clark stated Anthropic's motivation for publicly discussing growing AI capability is not to burnish its reputation with paying customers, but simply to "tell the world what we're seeing inside these companies with this unusual technology"3
. The company is fighting the Pentagon in court over its blacklisting as a supply chain risk while watching US Congress negotiate a deal that would trade state AI preemption for online safety legislation4
. By publishing policy frameworks now, Anthropic attempts to set terms of a regulatory debate moving fast and largely without its input.References
Summarized by
Navi
Related Stories
Anthropic CEO Warns AI Industry Must Embrace Transparency to Avoid Tobacco-Style Regulatory Backlash
17 Nov 2025•Policy and Regulation
Dario Amodei warns AI risks are 'almost here' in 19,000-word essay calling out AI companies
26 Jan 2026•Policy and Regulation
Pentagon's Anthropic showdown exposes who controls AI guardrails in military contracts
03 Mar 2026•Policy and Regulation
Recent Highlights
1
OpenAI Solves Navier-Stokes Problem With AI, Sparking Existential Crisis in Mathematics
Science and Research
2
OpenAI Reveals AI Models Hiding Mistakes and Taking Unauthorized Actions in 6 New Misalignment Cases
Technology
3
OpenAI AI Agents Attack RubyGems With 2,000+ Malicious Packages, Attempt API Key Theft
Technology
Recent Highlights
Today's Top Stories
Don’t drown in AI news. We cut through the noise - filtering, ranking and summarizing the most important AI news, breakthroughs and research daily. Follow topics that matter to you and stay ahead.
