Anthropic pushes for government power to block dangerous AI as self-improvement accelerates

Anthropic's warning over AI self-improvement has a hidden message -- accelerating development requires more compute before companies ever risk losing control of frontier AI models

The company that just a few weeks ago told us that its Mythos model was too powerful to be released publicly is now saying that we might need to hit the pause button on AI altogether, while also teaching its AI to build itself. On June 4, Anthropic published a report, when AI builds itself, showing that Claude now writes more than 80% of the code merged into its own production codebase, up from the low single digits before Claude Code reached research preview in February last year, and arguing that the loop has begun to accelerate AI development in a way that could eventually leave humans unable to control the systems being built. The Anthropic Institute, the firm's research arm, casts the trend as early movement toward recursive self-improvement, the point at which a model designs and builds its own successor without meaningful human input, and warns that the rare misalignment in today's models could keep "growing more frequent but less understood until we lose control of them." Reading further into the post, and taking the entire frontier AI model development ecosystem reveals some other uncomfortable truths that the developers of cutting-edge AI models also have to reckon with: compute. Loss of control Anthropic gave us three predictions of ways the next few years could play out, reserving a particularly dire warning for the case in which models become capable of fully improving themselves. Progress, Amodei's lab argues, would then be paced almost entirely by available compute, human engineers would be pushed into oversight and verification, and a self-improving model could come to dominate as its abilities outstrip those of the people who built it. The firm called this -- the task of keeping a system's behavior tied to human intent -- the part of this future it's least sure about. A capable, well-aligned model might discover new ways to keep its successors safe, it said, or the reverse could hold, and misalignment could compound generation over generation, with the unusual concession that a sufficiently wise model might instead choose to halt its own development. The idea of an ultraintelligent machine designing still better machines ("singularity") has been around for decades. British mathematician I. J. Good argued back in the 1960s through his "intelligence explosion" thesis that such a machine would be the "last invention that man ever need make," so long as it remained "docile enough" to tell us how to control it. Meanwhile, the "Godfather of AI," Geoffrey Hinton, has put the odds of AI causing human extinction within three decades at 10% to 20%. The International AI Safety Report, chaired by Yoshua Bengio and published in January 2025 with input from more than 100 experts across 30 countries, defines loss of control as a scenario in which AI systems operate outside anyone's control with no clear path to regaining it. Every figure behind the warning coming out of Anthropic is based on data from within, and none of it has been independently audited. Among this data is its claim that in Q2 2026, the typical Anthropic engineer is merging eight times as much code per day as in 2024. On the hardest, least-specified coding tasks, Claude succeeded 76% of the time in May 2026, a rise of 50 percentage points in six months. On an internal test that asks each new model to make training code run faster, results climbed from roughly triple the original speed with Claude Opus 4 in May 2025 to about 52 times with the unreleased Mythos Preview model by April 2026, against the four to eight hours a skilled researcher needs for a fourfold gain. In fairness, Anthropic does then call lines of code a poor proxy for output and admits that the eight times figure almost certainly overstates the real gain. Its research-judgment study, in which models beat the human's next step 64% of the time, drew on 129 moments the company deliberately picked because the human's choice had room for improvement, so it's not a like-for-like contest. The report publishes no breakdown isolating how much recent capability gain comes from the self-improvement loop rather than from raw compute, more data, and human-led research. Cognitive scientist Gary Marcus called the piece a "bait and switch" on his Substack, arguing the company had shown faster coding under human direction rather than a system improving itself. Bentley University mathematician Noah Giansiracusa told Scientific American, "I don't think it's a genuine call to slow down." AI is writing everyone's code Anthropic isn't alone here. Google CEO Sundar Pichai said in an April blog post that 75% of new code at Google is now AI-generated and approved by engineers, up from 50% the previous autumn. OpenAI's Jakub Pachocki has described the company's Codex agent as "a very early version of an AI researcher," and OpenAI has said it's building toward a fully automated one. Chinese developer MiniMax marketed its M2.7 model in March as "self-evolving," claiming it ran its own scaffold-optimization rounds and handled a large share of its reinforcement-learning research, though the benchmarks were internal and unreplicated. Independent measurements do somewhat support a trend of fast improvement without confirming a runaway one that the AI labs are talking about. METR, for example, found last year that the length of task an AI can finish with 50% reliability has been doubling roughly every seven months. On its RE-Bench research benchmark, the best agents beat human experts given two hours, but the humans pulled ahead at eight hours and roughly doubled the top agent's score at 32 hours. AI's advantage so far sits in short, well-defined bursts, not the sustained, open-ended work that research depends on, which is the human edge Anthropic has said is still holding strong. No compute means no runaway AI Anthropic half-buries the fact that it's compute capacity that's ultimately the binding constraint in all of this. It names chip fabrication, grid expansion, and interconnect bandwidth as the factors that could cap progress ahead of intelligence itself. We're all aware that those limits are solid as things currently stand: SK hynix and Micron have sold out HBM output for the year, high-power transformers carry three-to-five-year lead times, switchgear is booked into 2028, and grid-interconnection queues run three to seven years. A Sightline Climate analysis estimated that 30% to 50% of large data centers due to open in 2026 will slip or cancel. U.S. data centers drew about 4.4% of national electricity in 2023, a share the Department of Energy's Lawrence Berkeley National Laboratory expects to reach 6.7% to 12% by 2028. Meanwhile, the four largest hyperscalers are on course to spend more than $650 billion on AI infrastructure this year. Whether compute ultimately puts a lid on any out-of-control, self-improving loop is an unsettled debate. Forethought researcher Tom Davidson argues that there's a chance that compute bottlenecks won't "slow down a software intelligence explosion until its late stages," while Epoch AI counters that if compute and cognitive labor are complements rather than substitutes, software-only acceleration stalls once it hits a compute wall. 'No, you hang up first' As for pausing AI development, Anthropic says it'll only do this if rival labs at or near the frontier do the same in a verifiable way, and that a halt by one company wouldn't change who's leading the way. This is a facetious suggestion at best that insults the intelligence of anyone who has been paying attention to the AI arms race. It's beyond obvious that no lab this far down the road -- let alone Anthropic -- is ever going to ease off, especially when Anthropic's own report essentially doubles as a piece of marketing for how fast it can make Claude build Claude. To suggest in one breath that AI might need to be paused or slowed down in one breath and then saying "but everyone else needs to go first" in another is quite the remark. Anthropic's report also came just days after the company confidentially filed for an IPO at a reported valuation near $965 billion, a glaring juxtaposition that read as a front-runner lobbying for limits it stands to help set. Anthropic made a self-assessment in April, when it said its Mythos Preview model had found thousands of severe vulnerabilities, a claim that later drew scrutiny over how much of it rested on a small manual sample.

Anthropic urges US to require safety tests for most capable AI models

WASHINGTON, June 10 (Reuters) - Anthropic called on the U.S. Congress on Wednesday not to block state laws regulating AI unless lawmakers enact a "rigorous" federal law that addresses "catastrophic AI risks," according to a company statement. The company also urged Congress to require AI companies to ⁠put their most powerful models through independent safety tests, according to the statement. Anthropic is aiming to influence regulation of AI as it prepares for a U.S. initial public offering. The listing would represent one of the most consequential stock market debuts in years, potentially reshaping benchmark indexes, investor flows and the ⁠broader narrative driving U.S. equities. The company on Wednesday also urged Congress and states to modernize the technology used to pay unemployment benefits to prepare for potential ⁠AI-driven layoffs. Half of Americans fear that the rise of AI could put them or someone in their ⁠household out of work, according to a Reuters/Ipsos poll completed on Monday. The technology used to pay unemployment ⁠benefits is "not sufficiently prepared for a large labor market shock," the company said. Reporting by Courtney Rozen, Editing by Franklin Paul and Will Dunham Our Standards: The Thomson Reuters Trust Principles., opens new tab * Suggested Topics: * Artificial Intelligence * Data Privacy * Intellectual Property * Public Policy * Worker Rights Courtney Rozen Thomson Reuters Courtney Rozen reports on the world's largest technology companies from Washington, D.C., focusing on the relationship between the tech industry and the U.S. government. She reported on DOGE and the federal workforce during the first year of U.S. President Donald Trump's second term. Prior to joining Reuters, she was a White House correspondent at Bloomberg Government. She graduated from American University with a master's degree in journalism.

Anthropic co-founder Jack Clark warns AI needs a 'brake pedal'

Anthropic co-founder Jack Clark has called for the ability to slow progression of artificial intelligence (AI), warning the technology is nearing a point where it could develop without human input. "You want the option to be able to take your foot off the gas and put your foot on the brake", Clark told BBC Newsnight. "Right now, it's like the AI industry has a gas pedal, but it doesn't have a brake pedal." He stressed people, through government policy, need to keep control of AI systems, which will only get more powerful and have broader impacts on society. "The world needs to do some thinking and we need to eventually develop some new regulations that allow us to be confident in these systems," he said. Already, Anthropic's popular chatbot Claude is operating on code of which 80% the system wrote itself. Getting to 100% is possible within two years, Clark said, and "would have huge implications". Clark did not outline how a "brake pedal" for AI research and development could be created, but drew a parallel between AI and the oil boom and barons of the turn of the last century. "Society's response was to come up with a sensible policy and regulatory framework that gave people confidence in oil and the benefits that oil could provide to the world, and meant that you didn't have to worry about the personalities of the people leading the companies", Clark said. "That's clearly where we end up here." Yet, Anthropic this week welcomed an executive order on AI from US President Donald Trump that was relatively hands-off in its directives toward the companies. It did not require AI companies to submit to safety testing by the government, something that remains a voluntary effort. Major AI companies pursuing advances in the technology, including Anthropic, OpenAI and Google, have also not said they will pause their own research. Anthropic has grown so quickly since its founding five years ago that it is preparing to debut on the public stock market. It is poised to be one of the first public listings by a newer AI firm and one of the most valuable stock listings in history, as Anthropic's valuation is estimated by private investors to be nearly $1tn (£745bn). Clark said Anthropic's motivation for publicly discussing the growing capability of AI technology is not to further burnish its reputation with paying customers. He simply wants to "tell the world what we're seeing inside these companies with this unusual technology". Since its founding by chief executive Dario Amodei, Clark and a handful of other executives, Anthropic has positioned itself as outspoken about potential risks stemming from AI. It even engaged in a public dispute with the US Department of Defense over concerns that its AI tools would be used in mass surveillance of Americans and autonomous warfare. "I am worried for my kids if we as a society don't have a serious conversation about what the implications of AI's continued advances mean," Clark told Newsnight. "There are potentially great benefits. There are also risks." Clark added that one of the risks is a disruption to the economy, with fears that AI technology like "agents" - essentially individual AI bots that conduct routine tasks somewhat autonomously could take over certain jobs. Major tech companies have conducted mass layoffs over the last year, often citing the growing ability of AI tools to do the work of hundreds or even thousands of software engineers. Clark said people who are more creative and have more, better ideas may actually have an advantage over AI technology. "There are open questions about whether AI systems can be truly creative... there is not really evidence for that yet", he added. "At Anthropic, we're now limited more by the ability to generate good ideas than the ability to do the engineering to turn those ideas into reality." Nevertheless, Clark suggested a young person who may be feeling that an economy built on AI does not have a place for them should "develop a hobby" and pursue a liberal arts education. "People that are creative and can think broadly, people that read a lot, people that have interests are the ones most benefited by this," Clark said. "Indulge in curiosity and it pays back in how you can use this technology."

Anthropic wants governments to have the power to block dangerous AI deployments

Anthropic proposed giving governments legal authority to block dangerous AI models, with fines tied to global revenue. A second framework covers worker protections. Anthropic has published two policy frameworks calling for governments to have the legal authority to block dangerous AI deployments and for economic safeguards to protect workers as AI reshapes the labour market. The Advanced AI Framework covers safety regulation. The Economic Policy Framework addresses displacement, capital distribution, and the social safety net. The safety framework is the more aggressive of the two. Anthropic wants governments to be able to block or deter the deployment of models that pose a "significant risk of catastrophic harm," with civil penalties tied to global annual revenue that escalate with repeated violations. It would apply only to models trained using more than 10²⁵ floating-point operations, developed by companies earning more than $500 million in AI revenue or spending more than $1 billion on AI R&D. That threshold captures a small number of companies: Anthropic itself, OpenAI, Google DeepMind, xAI, and potentially Meta. Anthropic is calling for regulation that would directly apply to its own business, a positioning it has used before to distinguish itself from competitors that lobby against oversight. The framework identifies four categories of catastrophic risk: biological weapons development, large-scale cyber vulnerability discovery, loss of control over autonomous systems, and AI that automates its own R&D. Anthropic pointed to its own experience with Mythos Preview, which discovered thousands of high-severity vulnerabilities in every major operating system and browser, as evidence that these risks are not hypothetical. Frontier developers would be required to test their models, publish summaries of the results, submit to independent evaluation, maintain security programmes, and publish regular risk reports. Anthropic says transparency alone is no longer sufficient. "AI capabilities are going to improve rapidly over the coming months," the company wrote. "Their governance needs to keep pace." On preemption, Anthropic takes a direct shot at the White House's push to block state AI laws. "We do not believe Congress should preempt state law unless it enacts a federal law that is at least as strong as the framework we are proposing today," the company said. It calls for preemption to be "surgical," allowing states to regulate on child safety, consumer protection, and other issues outside the scope of a federal safety law. The economic framework is less detailed but addresses the displacement question that graduating students have been booing about at commencement ceremonies. It includes proposals for capital accounts, wage insurance, tax incentives, and an expanded social safety net. The goal is to ensure AI's financial benefits are "broadly shared," though specifics on funding and implementation are left for future debate. The timing is deliberate. Anthropic is fighting the Pentagon in court over its blacklisting as a supply chain risk. It is preparing for an IPO. And it is watching Congress negotiate a deal that would trade state AI preemption for online safety legislation. By publishing its own framework, Anthropic is trying to set the terms of a regulatory debate that is moving fast and largely without its input.

Anthropic CEO says government should block dangerous AI

Why it matters: Anthropic's ideas for tech regulation and economic disruption from AI go far beyond anything currently under serious consideration in Washington right now. * They're also sure to stir up a new set of accusations that Anthropic is proposing strict rules to lock in its own dominance or using frightening future scenarios as a marketing ploy. Driving the news: Amodei's new essay and proposed advanced AI framework argues that policy has to change in response to the rapid development of AI. * Trump's AI executive order should go further, he writes, and require mandatory testing for risks related to cybersecurity, biological weapons, loss of control or automated R&D. * Even more aggressive regulation might be needed in the future if AI systems become more of a threat, he writes. * Anthropic is also proposing an economic policy framework to address AI disruption including capital accounts, wage insurance, tax incentives and an expanded social safety net. What they're saying: "AI is advancing at a lightning pace," Amodei writes. "...By contrast, policy -- and especially legislation -- moves very slowly." * Following the release of Anthropic's power model Mythos, Amodei writes that biological risks and "serious AI autonomy risks" may soon follow. * "We now, globally and collectively, need to activate a slow and rickety policy apparatus to deal with risks and opportunities that are going to compound surprisingly quickly from here." Amodei writes that existing transparency legislation is not enough and calls for "more serious and binding regulation of AI." * Like cars, airplanes, or drugs, he writes, AI regulation should require frontier models to go through rigorous testing and auditing. * "Their release should be blocked or reversed as a threat to public safety if they do not meet high standards of safety." On the economy, Amodei writes that "it's reasonable to think that AI could produce much larger disruptions to the labor market than previous technologies, and, potentially, more enduring disruptions." * He calls for better data on AI-related job loss, wage insurance, retention tax incentives and possibly universal basic income or universal capital accounts. * Public opposition to data center buildouts is "largely a symbol or outlet for broader economic anxieties about AI," Amodei writes. Between the lines: Amodei also writes that regulatory systems are not prepared for how quickly AI will bring new advancements. * He suggests reform at agencies like the FDA to approve new drugs discovered by AI faster. * He also suggests banning domestic use of fully autonomous weapons and advocates for continued leadership and coordination from democratic countries on AI. The bottom line: Amodei still wants the public to be aware of AI's benefits.

Anthropic CEO calls for FAA-style regulation of powerful AI models: what enterprises should know

In a sweeping new essay titled "Policy on the AI Exponential," Anthropic co-founder and CEO Dario Amodei publicly calls for new government regulations governing the release of powerful AI models -- specifically comparing AI industry to commercial aviation, which follows regulations enforced by the U.S. Federal Aviation Administration (FAA) -- arguing that this is necessary to maintain public safety as AI capabilities and potential misuses grow. Alongside the essay, Anthropic released two comprehensive policy roadmaps: an Advanced AI Framework targeting catastrophic model risks, and an Economic Policy Framework addressing AI-driven labor displacement backed by $350 million in new funding. As Amodei noted on X following the release: "Anthropic has long advocated for transparency requirements for frontier AI, because the risks weren't yet clear enough to regulate precisely. That is no longer sufficient". For technical decision-makers, CIOs, and enterprise architects, the essay is not just a political statement -- it is a preview of the operational, regulatory, and workforce constraints that will govern the next generation of enterprise tech. Here are the top three takeaways enterprise leaders need to extract from Anthropic's latest policy drop. 1. Frontier Models May Face "FAA-Style" Deployment Holds For the past three years, enterprises have built products on the assumption that AI API capabilities will only move in one direction: faster and more powerful. Anthropic's Advanced AI Framework introduces a new variable: regulatory embargoes. Amodei explicitly compares the necessary AI regulatory regime to the Federal Aviation Administration (FAA), stating: "Frontier AI models, like airplanes, should be required to go through technical testing and auditing, and their release should be blocked or reversed as a threat to public safety if they do not meet high standards of safety". The company is proposing that models trained using more than 10^25 floating-point operations (FLOPs) -- or developed by companies with over $500 million in AI revenue or $1 billion in AI R&D -- must undergo mandatory third-party testing. If these models present severe biological, cybersecurity, or autonomy risks, the government would have the legal authority to block, delay or deter their deployment. The Enterprise Implication: If your company licenses foundation models for core infrastructure, you must plan for supply chain volatility. A highly anticipated model update from an AI vendor could be delayed indefinitely by regulators, or an existing model could be revoked if post-release testing reveals autonomous threats. Tech leaders must design multi-model architectures that avoid locking into a single vendor, ensuring business continuity if a provider's flagship model is blocked by a federal agency. 2. Cybersecurity Around AI Is Now Critical Infrastructure Anthropic's push for regulation is heavily motivated by the recent escalation in AI-driven cybersecurity threats. Amodei explicitly references Anthropic's own Claude Mythos Preview, noting that its ability to discover high-severity vulnerabilities across major operating systems "scrambled" the global cybersecurity landscape. Under Anthropic's proposed framework, securing the AI development environment is paramount. Frontier developers would be required to protect their model weights from both external cyberattackers and insider threats. Furthermore, companies must develop channels to report "model distillation attacks" -- where competitors or bad actors use a primary model to train a cheaper, unaligned clone. The Enterprise Implication: The stakes for enterprise security are twofold. First, defensive AI capabilities will become a prerequisite; as Amodei warns, attackers using frontier models to probe for vulnerabilities will outpace traditional, human-led defense. Second, enterprises that fine-tune open-weight models or host proprietary instances locally will likely face intense new compliance and infosec burdens. Treating model weights as highly classified corporate secrets will become the new industry standard. 3. Plan for Structural Labor Displacement, Not Just Efficiency Perhaps the most sobering aspect of the announcement is Anthropic's Economic Policy Framework. The company is publicly acknowledging that if AI achieves its predicted capabilities, it will act as a "general substitute for labor" rather than just a productivity tool. Amodei frames this bluntly: "The key challenge in such a world won't be incentivizing growth, but finding a way for everyone to share in the benefits". To back this up, Anthropic is committing $350 million to address economic disruption: $200 million for an Economic Futures Research Fund to pilot public policy solutions, and $150 million for a national fellowship program. The framework actively plans for scenarios where AI drives unemployment to 5%, 10%, or even unprecedented levels, advocating for policies like wage insurance, universal basic income, and sovereign wealth models. The Enterprise Implication: For tech leaders and HR departments, the AI transition is about to become a labor relations minefield. The economic framework notes that companies "can choose to retrain and redeploy rather than reduce headcount," but admits voluntary action is not a substitute for government response. Enterprises looking to integrate AI heavily should begin implementing workforce transition plans immediately. Leaders who view AI solely as a mechanism for fast cost-cutting through layoffs may soon find themselves crossways with new "pro-employment incentives" or retention tax policies proposed by advocates to slow job displacement. What Enterprises Should Do Now Anthropic's announcement marks a turning point in the AI industry's dialogue with Washington and the global market. As Amodei posted: "Many of these policy ideas have common-sense appeal across the political spectrum, and the sooner we act on them, the sooner everyone shares in AI's benefits". For the enterprise, the message is clear: the era of "move fast and break things" in generative AI is closing. The era of rigorous compliance, systemic security, and complex workforce transitions is fast approaching. To prepare for this shift, enterprises must first decouple their AI strategies from single-vendor dependencies. If a flagship model is suddenly blocked or recalled under the proposed FAA-style regulatory powers, organizations reliant on that specific API will face immediate operational paralysis. IT leaders should build multi-model architectures that allow them to swap out foundation models seamlessly, ensuring business continuity in a highly regulated ecosystem. Second, technical decision-makers must elevate AI infrastructure to the level of critical cybersecurity. With frontier AI systems now capable of discovering high-severity software vulnerabilities at scale, the threat surface is expanding rapidly. Companies that fine-tune models or host them internally must lock down their development environments against both external and insider threats, matching the rigorous security standards Anthropic is demanding of the broader industry. Finally, leadership teams need a proactive, rather than reactive, labor strategy. Anthropic explicitly warns against using AI solely for cost savings through layoffs, encouraging enterprises to actively seek new use cases that allow them to retain and retrain their existing workforce. As governments potentially deploy pro-employment tax incentives and wage insurance policies to slow job displacement, companies that aggressively cut headcount to fund AI adoption may find themselves on the wrong side of both public sentiment and upcoming economic regulations.

Anthropic just proposed taxing itself to pay for the jobs its AI destroys | Fortune

Alongside new policy proposals from the maker of the Claude chatbot, Anthropic CEO and co-founder Dario Amodei published an essay on his personal website that expanded on his position that the government should promise economic support for those financially impacted by AI. The technology could produce much larger disruptions to the labor market than previous technological advancements, Amodei wrote, and those disruptions could last longer. "The key challenge in such a world won't be incentivizing growth, but finding a way for everyone to share in the benefits," Amodei wrote. The announcement comes on the heels of Anthropic rival OpenAI on Monday outlining goals that included ensuring gains from the technology are "widely shared." OpenAI CEO Sam Altman recently met with Sen. Bernie Sanders to discuss a plan for the public to take an ownership stake in artificial intelligence companies like OpenAI, using their stock to create a public wealth fund that would spread the fortune generated by AI behemoths. In the Oval Office on Wednesday, President Donald Trump told reporters that he will soon meet with executives from several leading AI companies to discuss "giving back" to the public. "We're talking about giving back something to the public, and if we do that, the public will become very rich," Trump said. "I think they'll do that, and I think it'll make it very popular." In his essay, Amodei said he has warned of job displacement not because he is "trying to be a 'prophet of doom'" but because he wants "both policymakers and the private sector to have the best chance to adapt and respond." He proposed better data collection to track AI job displacement, pro-employment policy incentives to slow or reduce displacement and "mechanisms such as universal basic income" if job displacement more permanently drives down labor demand. That universal basic income could be financed through taxes on "relevant companies" or by raising the capital gains tax, Amodei wrote. Scant details were available Wednesday about the $200 million commitment from Anthropic, but the company said it will go to what it calls an Economic Futures Research Fund that will back research trials and "program evaluation" on public policies it deems promising. The company is also establishing a $150 million national fellowship program it says will help early-career professionals "extend the benefits of AI to communities across America." Anthropic and OpenAI each recently announced they were moving toward initial public offerings of shares, following Elon Musk's rocket company SpaceX, which is pitching itself as an AI-focused space company as it prepares to go public. The economic policy framework Anthropic proposed Wednesday set recommendations for how the U.S. government could respond to three levels of economic disruption caused by AI: one in which the national unemployment rate reaches 5%, 10% and an unspecified, "unprecedented" level. The latest unemployment rate, reported last week, was 4.3%. In the "unprecedented" scenario, the company wrote that more permanent support will be necessary, and it listed several ways to generate and share revenue broadly, including basic income, sovereign wealth models and equity-sharing mechanisms. This would be "novel economic territory," the company wrote. The company's proposals also outlined several suggestions for mitigating safety and security risks. Anthropic is known for its emphasis on safety and building reliable, "steerable" AI systems, with Amodei and its co-founders splitting off from OpenAI to form the new company in 2021. The proposals add that the government should be able to "block or deter" the rollout of AI models that "pose a significant risk of catastrophic harms." Amodei wrote that AI regulations should match the rigor of Federal Aviation Administration regulations in that AI models would be required to go through technical testing and auditing like airplanes. They wouldn't be released if they didn't meet high safety standards. Last week, Trump signed an executive order on AI oversight that established a framework for the government to vet the national security risks of the most advanced AI systems for up to a month before their public release. Amodei added existing regulations for aircraft, automobiles and drugs should serve as models for regulating AI. They are all "powerful technologies essential to the modern economy," he wrote, "but capable of killing large numbers of people if designed or operated poorly."

Anthropic CEO Warns AI Is Getting Too Powerful -- While Releasing Powerful AI

Anthropic is releasing a legislative proposal on frontier model testing and a policy framework for job displacement alongside the essay. Anthropic CEO Dario Amodei said Wednesday that governments can no longer treat AI regulation as a problem to study and that the United States needs binding safety requirements for the most powerful AI models. In the essay, titled "Policy on the AI Exponential," Amodei argues that transparency requirements are no longer sufficient and calls for binding regulation of frontier AI systems. "AI is advancing at a lightning pace -- in only four years, AI models have gone from barely being able to write a coherent line of code to writing most of the code at major AI companies," Amodei wrote. Amodei's essay comes as Anthropic expands access to Claude Mythos with the launch of Mythos 5 on Tuesday, its restricted frontier AI model for cybersecurity organizations and government partners. Researchers, including the UK's AI Security Institute, found it can autonomously execute complex cyber attacks. According to Amodei, his proposal borrows from the regulatory structure used by the Federal Aviation Administration. "Frontier AI models, like airplanes, should be required to go through technical testing and auditing, and their release should be blocked or reversed as a threat to public safety if they do not meet high standards of safety," he wrote. "I am grateful to see the Trump administration's Executive Order move incrementally towards a greater role for government in AI, though Anthropic's proposal recommends even further action." Under Amodei's proposal, a regulatory framework would require mandatory third-party testing of advanced AI models, government authority to block unsafe deployments, and requirements that companies secure model weights, conduct safety testing, and report serious incidents. He also calls on governments to prepare for AI-driven job displacement and advances in drug development, limit surveillance and autonomous weapons in domestic law enforcement, and strengthen cooperation among democratic nations on critical AI technologies. "First, enduring job displacement is undesirable and dangerous, and we should do everything we can to minimize or prevent it, not to bring it about," he wrote, pointing to past instances where he warned about job displacement. "Second, any response to AI-driven job displacement needs to address both the need to provide for everyone economically, and the need for people to find meaning, purpose, and agency." The essay also comes the same week that Anthropic launched Claude Fable 5, a public-facing version of Claude Mythos 5, which routes certain requests involving cybersecurity, biology, chemistry, and AI development to the less capable Claude Opus 4.8 as a safeguard against misuse. The launch drew criticism from developers and researchers over Fable's higher token usage, mandatory 30-day data-retention requirements, and safeguards that can reduce the model's capabilities without notifying users. Amodei's call for policy changes around AI development also comes as Anthropic prepares to go public. Earlier this month, the company filed paperwork with the U.S. Securities and Exchange Commission for an initial public offering after raising a reported $65 billion Series H round at a reported $965 billion valuation. While Amodei framed the issue as a race between technological progress and public policy, critics have questioned whether calls for stricter AI regulation actually serve the public good. In April, OpenAI CEO Sam Altman accused Anthropic of using "fear-based marketing" to promote Claude Mythos, arguing that concerns about advanced AI can be used to justify concentrating control of the technology among a small number of companies. "You can justify that in a lot of different ways, and some of it's real, like there are going to be legitimate safety concerns," Altman said. "But if what you want is like 'we need control of AI, just us, because we're the trustworthy people', I think fear-based marketing is probably the most effective way to justify that." Amodei rejected the idea that concerns about advanced AI are primarily a public-relations problem, arguing instead that fears about the technology reflect legitimate concerns that must be addressed. "People are worried about AI because they correctly perceive that its risks are real, not because AI CEOs have been insufficiently Panglossian," he said, referring to the fictional philosopher Pangloss from Voltaire's Candide, who is known for maintaining an unwavering optimism that everything is for the best regardless of circumstances. "I believe it is my duty as an AI leader to continue to be transparent about these risks, and public concern in response to this transparency constitutes democratic accountability working as it should," he said.

Anthropic co-founder warns AI could soon slip beyond our control

Anthropic co-founder Jack Clark said AI agents might soon be able to build and train models themselves and, if that happens, humans could lose control over AI systems. Anthropic co-founder Jack Clark wants the AI industry to pump the brakes before the technology starts further developing itself without human input. Speaking to the BBC, Clark said 80% of Anthropic's coding work is already being done by its AI Claude, and that it could go up to 100% in a couple of years. However, he said "it's a choice" whether AI companies let it get that far without stopping it. "We think this is a topic that the world should be talking more about," Clark said. "The AI industry right now has a gas pedal, but it doesn't have a brake pedal in the car, and we want to do some of the work to build that pedal." This process is called "recursive self-improvement," where an AI is able to improve itself without human input, according to Anthropic in a related blog post from Thursday night. In a recursive model, AI agents, the autonomous workers built by a chatbot, could "become capable enough to build and train models themselves," so Claude "could be continuously improved by Claude," Anthropic said. While recursive AI could bring some good to the fields of science and healthcare, Anthropic warns that it might mean increasing "the risks of humans losing control over AI systems." "If systems are capable of fully building their own successors, the ways we secure them, monitor them and shape their behaviour all grow much more important," the blog post reads. There is evidence within Anthropic's own model that recursion is coming sooner rather than later. It points to the fact that code correction rates by their staff have been falling steadily for the last year, which means there are fewer errors in what Claude is producing. Claude is also able to run its own research experiments when given an open-ended question, such as "Can a weaker model supervise a stronger one?" and come up with its own solutions without human input. "The evidence suggests that the human role is narrowing at each step in the AI development process," the blog reads. Anthropic said its institute will conduct research to build a system to check whether developers have actually stopped or slowed down the move towards recursive AI, it said. However, a real slowdown would require "multiple well-resourced labs at or near the frontier, in multiple countries, agreeing to stop under the same conditions."

Anthropic's Dario Amodei wants governments to have the power to block 'dangerous' AI systems

Anthropic's Dario Amodei wants governments to have the power to block 'dangerous' AI systems Anthropic PBC Chief Executive Dario Amodei (pictured) is calling on the U.S. government to block the deployment of dangerous artificial intelligence models in the same way as it prevents unsafe airplanes from taking off. In a new post on his personal blog, he said there needs to be mandatory third-party audits of frontier AI systems, and suggested that governments be able to shut them down if any unacceptable risks are discovered. At the same time, he also called on the government to provide more economic support to people who are financially impacted by the rise of AI. Amodei's AI safety proposal is far more aggressive than what most people have suggested. It exceeds the executive order issued by U.S. President Donald Trump last week, which mandates that intelligence agencies play a role in testing new models for risks. He suggested that governments should audit new AI models based on the amount of resources thrown into their training process. He proposed there should be a "compute threshold," and that if any new model exceeds this specified level of compute, it would automatically trigger an independent investigation into its capabilities before it can be launched publicly. The essay identified four specific risk categories that need to be evaluated: cybersecurity vulnerabilities, biological weapons capabilities, the ability to accelerate automated research in dangerous domains, and the potential for models to grow beyond human control. Should the auditors decide that a model poses an unacceptable risk in one of those areas, Amodei wants governments to be able to prevent their deployment. Amodei said existing regulations for aircraft, automobiles and drugs should serve as models for regulating AI. "I believe the best analogy, at least at the current stage of the exponential, is to cars, airplanes, or drugs -- powerful technologies essential to the modern economy, but capable of killing large numbers of people if designed or operated poorly," he wrote. Amodei's proposed framework goes much further than Trump's June 2 executive order. The President simply encouraged AI firms to voluntarily share their models with government auditors a month before their public release. The order also emphasized how the intelligence community should be involved in testing new models for cybersecurity vulnerabilities. But it stopped short of enforcing anything, or introducing mandatory testing. The Anthropic CEO used much stronger language. There's no mention of the word "voluntary" and he doesn't ask for cooperation, but instead, compliance. Whereas Trump's order leans on intelligence agencies, Amodei calls for independent auditors to be given effective veto power. Anthropic has been pushing for more stringent controls on AI for a while. Back in September 2023, it published a Responsible Scaling Policy that has since been updated several times, introducing concepts such as AI Safety Levels or ASLs. Essentially, it's a tiered framework for managing risk as AI model capabilities improve. The company has also voiced its support for other legislation, such as California's SB 53 bill. In addition to safety, Amodei also warned that the disruption to the labor market caused by AI's emergence could be much more significant than earlier technological advances. It could also last much longer, he said. "The key challenge in such a world won't be incentivizing growth, but finding a way for everyone to share in the benefits," Amodei wrote. Amodei said he has long warned of AI displacing jobs, not because he wants to be a "prophet of doom," but because by doing so, "policymakers and the private sector have the best chance to adapt and respond." In his post, he called for the government to collect more data to keep track of AI job displacement, and pro-employment incentives that could help to slow this down. He also wants the government to look at "mechanisms such as universal basic income," if job displacement becomes more profound and permanently reduces labor demand. It could be financed by taxing "relevant companies," or increasing the capital gains tax, he added.

Anthropic CEO: Government should have power to block dangerous AI deployments

Anthropic CEO Dario Amodei is arguing governments should have the power to block dangerous deployments of artificial intelligence if they don't meet a certain safety standard. Amodei's latest assertion, published in an essay on his website Wednesday, builds upon the company's pro-safety approach to AI, which has put them at odds with both its competitors and the Trump administration this year. The technology leader suggested AI regulation should emulate the testing processes at agencies like the Federal Aviation Administration, writing, "Frontier AI models, like airplanes, should be required to go through technical testing and auditing." "Their release should be blocked or reversed as a threat to public safety if they do not meet high standards of safety," Amodei added. While lauding President Trump's latest executive order on AI testing, Amodei said Anthropic is proposing "even further action" on AI regulation, including mandatory testing by a third-party auditor to check for risks related to cybersecurity, biological weapons, loss of control of AI systems, and automated research and development. He said the government should have the ability to block or deter deployment if a third-party evaluation determines risks in the four categories. Government agencies themselves could be the auditors under Anthropic's proposal, which notably breaks from other AI firms that warn of government overreach. Trump's executive order, signed earlier this month after weeks of delays, laid out a process for voluntary testing of frontier AI models up to 30 days before their release, but made clear this would not be a requirement for AI developers. For some in the industry, the emphasis on the voluntary nature was still not enough to assure it would not lead to overregulation of AI. Amodei went on to say there could come a time where even more aggressive policies are needed, writing, "When the most powerful AI systems look less like airplanes or automobiles and more like weaponizable nuclear materials -- a threat to humanity rather than 'just' a threat to public safety." This ideology clashed with the Pentagon earlier this year, resulting in a public dispute over safeguards for domestic mass surveillance and autonomous lethal weapons. Defense Secretary Pete Hegseth labeled the company as a supply chain risk, prohibiting the use of Claude in the Department of Defense. Anthropic sued the Trump administration, challenging both the Pentagon's designation -- typically reserved for foreign adversaries -- and President Trump's directive for civilian agencies to stop using Anthropic's products after negotiations fell apart over safety guardrails last February. Anthropic demanded its technology not be used in fully autonomous lethal weapons or for the mass surveillance of Americans, while the Pentagon insisted it be allowed to use Anthropic's Claude for "all lawful uses." Weeks after filing its suits, Anthropic released its most advanced cybersecurity AI model, Mythos, sending shockwaves through Washington. The model, along with other AI developments across the industry, got the attention of the White House, which raised concerns about the potential safety risks of the technology.

Anthropic Pledges $200 Million to Research AI's Economic Impact as CEO Suggests Job Loss Solutions

Anthropic on Wednesday joined growing calls for the artificial intelligence industry to find ways to cushion people from the technology's disruptions, announcing an initial $200 million investment to research AI's impact on jobs and the economy. Alongside new policy proposals from the maker of the Claude chatbot, Anthropic CEO and co-founder Dario Amodei published an essay on his personal website that expanded on his position that the government should promise economic support for those financially impacted by AI. The technology could produce much larger disruptions to the labor market than previous technological advancements, Amodei wrote, and those disruptions could last longer. "The key challenge in such a world won't be incentivizing growth, but finding a way for everyone to share in the benefits," Amodei wrote. The announcement comes on the heels of Anthropic rival OpenAI on Monday outlining goals that included ensuring gains from the technology are "widely shared." OpenAI CEO Sam Altman recently met with Sen. Bernie Sanders to discuss a plan for the public to take an ownership stake in artificial intelligence companies like OpenAI, using their stock to create a public wealth fund that would spread the fortune generated by AI behemoths. In the Oval Office on Wednesday, President Donald Trump told reporters that he will soon meet with executives from several leading AI companies to discuss "giving back" to the public. "We're talking about giving back something to the public, and if we do that, the ⁠public will become very rich," Trump said. "I think they'll do that, and I think it'll make it very popular." In his essay, Amodei said he has warned of job displacement not because he is "trying to be a 'prophet of doom'" but because he wants "both policymakers and the private sector to have the best chance to adapt and respond." He proposed better data collection to track AI job displacement, pro-employment policy incentives to slow or reduce displacement and "mechanisms such as universal basic income" if job displacement more permanently drives down labor demand. That universal basic income could be financed through taxes on "relevant companies" or by raising the capital gains tax, Amodei wrote. Scant details were available Wednesday about the $200 million commitment from Anthropic, but the company said it will go to what it calls an Economic Futures Research Fund that will back research trials and "program evaluation" on public policies it deems promising. The company is also establishing a $150 million national fellowship program it says will help early-career professionals "extend the benefits of AI to communities across America." Anthropic and OpenAI each recently announced they were moving toward initial public offerings of shares, following Elon Musk's rocket company SpaceX, which is pitching itself as an AI-focused space company as it prepares to go public. The economic policy framework Anthropic proposed Wednesday set recommendations for how the U.S. government could respond to three levels of economic disruption caused by AI: one in which the national unemployment rate reaches 5%, 10% and an unspecified, "unprecedented" level. The latest unemployment rate, reported last week, was 4.3%. In the "unprecedented" scenario, the company wrote that more permanent support will be necessary, and it listed several ways to generate and share revenue broadly, including basic income, sovereign wealth models and equity-sharing mechanisms. This would be "novel economic territory," the company wrote. The company's proposals also outlined several suggestions for mitigating safety and security risks. Anthropic is known for its emphasis on safety and building reliable, "steerable" AI systems, with Amodei and its co-founders splitting off from OpenAI to form the new company in 2021.

Anthropic CEO warns of AI jobs reckoning as company pledges $200 million for research

Anthropic's Claude models have rapidly become among the most widely used AI tools in software development and enterprise work. The company's coding products, Claude Code and Claude Co-Work, are designed to automate tasks that once required teams of engineers and knowledge workers. As companies continue to sell artificial intelligence as a productivity miracle, Anthropic CEO Dario Amodei has warned that the rapid adoption of the technology can lead to an economic fallout faster than many people realise. "I am still pretty concerned," Amodei said in a Bloomberg interview, reaffirming his previous warnings about the labor market consequences of advanced AI. Anthropic's Claude models have rapidly become among the most widely used AI tools in software development and enterprise work. The company's coding products, Claude Code and Claude Co-Work, are designed to automate tasks that once required teams of engineers and knowledge workers. "We are seeing right now that AI is making people more productive," he said. "But that's the usual hump." The real challenge emerges when automation approaches completion, he said. "You automate 90 percent of the job. Great. People are ten times more productive in the other 10 percent because they're ten times more leveraged," Amodei said. "But eventually it gets close to 100 percent. Now the sequel to that is, well, then you have to find something else for them to do." The changes are already visible within Anthropic. In the same interview, Boris Cherny, the executive who led the development of Claude Code, said the platform has been writing all of the code for him for the last six months. "It's writing for almost all of it on my team," Cherny told Bloomberg. "For me personally, it's been writing 100 percent of my code for at least six months." According to Anthropic's own research, roles in management, finance, and legal professions will be most heavily affected in the near term. The AI giant on Wednesday pledged a $200 million investment to research AI's impact on jobs and the economy. Amodei also published an essay on his personal website in which he proposed better data collection to track AI job displacement, pro-employment policy incentives to slow or reduce displacement and "mechanisms such as universal basic income" if job displacement more permanently drives down labor demand. That universal basic income could be financed through taxes on "relevant companies" or by raising the capital gains tax, Amodei wrote.

Anthropic CEO Warns Advanced AI Could Become Too Powerful For Governments Or Corporations To Control -- 'Sh

Anthropic CEO Dario Amodei has warned that increasingly powerful AI systems cannot be safely entrusted to governments or corporations alone, arguing that both require meaningful checks and balances as artificial intelligence becomes more capable. In an essay published Wednesday, Amodei also said governments should have the authority to block or reverse deployments of advanced AI models that pose unacceptable risks, including threats related to cybersecurity, biological weapons and loss of control of AI systems. The Anthropic CEO argued that AI companies could eventually develop "quasi-state characteristics," becoming powerful enough to influence society in ways traditionally associated with governments. He wrote that advanced AI may become "so powerful that it cannot safely be fully entrusted to either governments or companies." Why Transparency Is No Longer Enough Amodei said existing transparency-focused approaches to AI oversight are no longer sufficient as frontier models become increasingly capable. He called for "more serious and binding regulation of AI" and compared advanced AI systems to airplanes and pharmaceuticals, arguing that they should undergo mandatory third-party testing before deployment. "Their release should be blocked or reversed as a threat to public safety if they do not meet high standards of safety," Amodei wrote. The proposed evaluations would focus on risks related to cybersecurity, biological weapons, loss of control and automated research capabilities. He added that AI developers should maintain strong security standards, conduct regular testing and promptly report major safety incidents. Preparing For AI-Driven Job Loss Beyond safety risks, Amodei warned that AI could create larger and more persistent labor-market disruptions than previous technological shifts. "It's reasonable to think that AI could produce much larger disruptions to the labor market than previous technologies," he wrote. To address those risks, Amodei proposed measures including wage insurance, worker-retention incentives, workforce training programs and expanded monitoring of AI's impact on employment. He also suggested broader income-support mechanisms could eventually become necessary if displacement becomes widespread. Amodei argued that policymakers should focus on ensuring the benefits of AI-driven economic growth are shared broadly rather than concentrated among a small group of companies and investors. Who Should Control Powerful AI? The essay also addressed concerns about AI's impact on civil liberties, political power and national security. Amodei warned that advanced AI could become a tool of surveillance and authoritarian control if governments gain unchecked access to increasingly capable systems. He also called for stronger protections against domestic use of fully autonomous weapons. At the same time, he argued that private companies should not hold unchecked influence over increasingly powerful AI systems, warning that some firms could eventually develop "quasi-state characteristics." On geopolitics, Amodei described AI as a technology comparable in strategic significance to nuclear weapons, arguing that it could become a dominant source of military and economic power. He called for closer coordination among democratic nations on AI policy, safety standards and supply-chain controls. Mythos Sparks Concern The essay comes days after Anthropic launched Mythos 5, the latest version of its cybersecurity-focused AI model, alongside Claude Fable 5, its newest consumer-facing system. Against that backdrop, Amodei's call for stronger safeguards, mandatory testing and government authority to block dangerous AI deployments arrives as policymakers and technology companies grapple with increasingly powerful frontier AI systems. Disclaimer: This content was partially produced with the help of AI tools and was reviewed and published by Benzinga editors. Image via Shutterstock Market News and Data brought to you by Benzinga APIs To add Benzinga News as your preferred source on Google, click here.

AI risks are here, time for tough rules: Anthropic CEO Dario Amodei

As artificial intelligence grows more powerful, governments must move beyond transparency measures and introduce stronger regulations to address emerging risks, according to Anthropic CEO Dario Amodei. Amodei said every major technology brings a challenge of balancing innovation and safety. He noted that during 2023-24, concerns grew that advanced AI could eventually be used to create biological weapons or engage in autonomous behaviour that could pose serious threats. To ensure AI develops safely, Amodei identified five key policy areas that need to be rethought: regulation and public safety, macroeconomics and taxation, scientific innovation, the balance of power between governments and citizens, and geopolitics. In a company post, Amodei said most of his recommendations, though framed around the United States, are relevant globally. He also announced that Anthropic is releasing a legislative proposal on frontier AI model testing and a framework to address job displacement caused by AI. Calling for stronger oversight, Amodei said, "Frontier AI models, like airplanes, should be required to go through technical testing and auditing, and their release should be blocked or reversed as a threat to public safety if they do not meet high standards of safety." He said AI systems above a certain computing threshold should undergo mandatory third-party testing for risks related to cybersecurity, biological weapons, loss of control of AI systems and automated research and development. "The government should have the power to block or deter deployment of the model if it is determined, in light of third-party assessment, to present unacceptable risks," he added. On the economic impact of AI, Amodei said the technology could drive rapid economic growth but may also replace many human cognitive tasks, leading to significant job losses. To manage the transition, he suggested tracking AI-driven job displacement, creating incentives for employment and exploring long-term support measures such as universal basic income or universal capital accounts funded through taxes on companies benefiting from AI. Amodei also warned that existing regulatory systems may struggle to cope with the wave of AI-driven innovations, especially in healthcare and biomedical research. He recommended that regulators accept AI-based simulations and analysis for processes such as toxicology prediction and biomarker validation. He further cautioned that powerful AI could become "the ultimate tool of autocracy" if misused, enabling mass surveillance and fully automated drone armies. To prevent such outcomes, he proposed stronger accountability rules for autonomous weapons, restrictions on their domestic use and greater public protections against government misuse of AI. On the geopolitical front, Amodei described AI as a transformative technology that will shape future global power dynamics. He warned that a country with advanced AI capabilities could hold a major advantage over one without them, comparing the gap to "an army of World War II Marines facing medieval swordsmen." He called on democratic nations to build a global coalition to secure AI supply chains, coordinate on risks, share benefits and resist AI-enabled repression.

Anthropic Urges Governments to Secure Power to Halt Dangerous AI | PYMNTS.com

Governments should also prepare workers and the economy for the impact of AI and ensure that the financial benefits of the technology are broadly shared, the company said. The AI startup shared its policy proposals in an Advanced Policy Framework and an Economic Policy Framework released Wednesday. Regarding AI safety, Anthropic said in the press release that governments' authority to block advanced AI models should apply only to models with a specified level of capabilities, companies of a specified size and four kinds of catastrophic risk: biological, cyber, loss of control and automated research and development (R&D). In these cases, governments should play a more substantial role than simply requiring transparency, the company said. In addition, the federal government should not preempt state law unless it enacts a sufficiently strong federal law, it added. Governments should also increase society's resilience through measures that include early warning biosurveillance to spot novel outbreaks, widely shared cyber capabilities, and the capacity to detect and shut down AI systems that act outside their developers' control, Anthropic said. Regarding economic policy, Anthropic said in another Wednesday press release that in a 5% unemployment scenario, governments should offer workforce training grants, occupational licensing reform, wage insurance and incentives for firms that retain and redeploy workers. In a 10% unemployment scenario, governments should expand unemployment insurance, and in the case of "unprecedented levels of unemployment," they should provide sustaining income replacement for a large share of the workforce, Anthropic said. Anthropic suggested Thursday (June 4) that frontier AI developers should slow or pause their efforts to give societal structures and alignment research time to keep up with the advancing technology. The company said its proposal for a pause was sparked by the growing share of AI development that is being delegated to AI systems, and the potential that an AI system could fully autonomously design and develop another AI system. Anthropic said such a development "might increase the risks of humans losing control over AI systems."

Anthropic Is Racing Toward An IPO. Its Co-Founder Wants A Way To Slow AI Down, Says The Industry 'Doesn't

As Anthropic races toward what could become one of the most closely watched IPOs this year after SpaceX, co-founder Jack Clark is warning that the artificial intelligence industry may need something it currently lacks: a brake pedal. Speaking to BBC Newsnight this week on a series of topics, Clark said policymakers should have the ability to slow AI development if systems become too powerful or society needs time to absorb the consequences. "You want the option to be able to take your foot off the gas and put your foot on the brake," Clark said. "Right now, it's like the AI industry has a gas pedal, but it doesn't have a brake pedal." A Warning Amid The AI Boom Clark's comments come as investor enthusiasm for AI continues to fuel soaring valuations across the sector. Claude Is Already Writing Most Of Anthropic's Code Clark pointed to a striking example of how rapidly AI capabilities are advancing. "So 80% of the code that goes into Anthropic now comes from Claude," he said. "The majority of the code in our organization is now written and derived from our AI systems itself." According to Clark, the shift has dramatically accelerated productivity and research inside the company, helping engineers produce roughly eight times more code than in previous years. But... The Question Investors Are Starting To Ask The prospect of AI systems increasingly contributing to their own development has become a central debate among both policymakers and investors. When asked whether AI-generated code could eventually account for all of Anthropic's software development, Clark said it was "plausible," adding that such a scenario would have "huge implications." "It's also a choice as to whether you let AI systems get that far," he said. Image via Shutterstock Market News and Data brought to you by Benzinga APIs To add Benzinga News as your preferred source on Google, click here.

AI may bring cyberattacks and job losses if left unchecked, warns Anthropic CEO Dario Amodei

Amodei also expressed concern about biological threats linked to AI. Artificial intelligence is becoming an important part of everyday life. It is helping people work faster, companies to improve their services and much more. However, as AI systems are becoming more advanced, concerns about their potential risks are also increasing. Highlighting these concerns in a post, Anthropic CEO Dario Amodei has warned that AI could create serious issues. In his latest post, titled Policy on the AI Exponential, Amodei said AI is improving so quickly that governments and regulators may struggle to keep up. He believes future AI could become extremely powerful and outperform humans in many areas. Also read: Anthropic launches Claude Fable 5, a safer Mythos AI model: Features, availability and how to use it One of his biggest concerns is the risk AI could pose to cybersecurity. According to Amodei, recent discoveries show that advanced AI models can create cyber threats that may affect financial systems, critical infrastructure and even national security. Amodei also expressed concern about biological threats linked to AI. "I believe that biological risks may soon follow and that serious AI autonomy risks may not be far behind," he said. He added, "The cyber risks that Mythos-class models present will not be the last that we must face." Also read: Starlink India launch reportedly on hold over security concerns: Here is what happened In his post, Amodei once again raised questions about the impact of AI on jobs. He said AI could replace many cognitive tasks that are currently performed by humans. "Enduring job displacement is undesirable and dangerous. There's a decent possibility that, despite all our efforts, AI still causes significant enduring job loss," he wrote. Amodei suggested that governments should closely monitor how AI affects employment and create support systems for workers who may lose their jobs because of it. He also warned that powerful AI could be misused by authoritarian governments. "Powerful AI in the wrong hands could be the ultimate tool of autocracy. A surveillance-focused AI could analyse widely available information at massive scale and use it to infer the innermost details of every citizen's life," he said. Anthropic has called for stronger rules for advanced AI systems, including safety testing, security requirements and government oversight before such technologies are widely deployed.

Anthropic co-founder warns AI needs a brake pedal, calls for stronger regulations

In a recent blog post, Anthropic also warned about the possibility of "recursive self-improvement." Anthropic co-founder Jack Clark has warned that the world needs stronger regulations for AI and a way to slow its development if necessary. According to Clark, AI technology is becoming more powerful at a rapid pace, and governments need to ensure that humans remain in control. Speaking to BBC Newsnight, Clark compared the current state of AI development to a vehicle that can accelerate but cannot slow down when needed. "You want the option to be able to take your foot off the gas and put your foot on the brake," he said. "Right now, it's like the AI industry has a gas pedal, but it doesn't have a brake pedal." Clark believes policymakers should start preparing for the growing impact of AI on society. He said regulations will be needed to make sure people can trust increasingly advanced AI systems. "The world needs to do some thinking and we need to eventually develop some new regulations that allow us to be confident in these systems," he said. Also read: ChatGPT can now remember more about you with OpenAI's new memory upgrade: Details One area that concerns Clark is AI's growing ability to create and improve software. He revealed that Anthropic's AI chatbot Claude is already operating on code that is mostly written by AI itself. According to Clark, around 80 per cent of Claude's code has been generated by the system, and he believes that figure could reach 100 per cent within the next two years. He also expressed concern about AI's effect on jobs. Clark noted that AI agents, which can perform tasks with limited human involvement, could eventually replace certain types of jobs. In a recent blog post, Anthropic also warned about the possibility of "recursive self-improvement," where AI systems become capable of improving themselves. "Taken far enough, and given enough compute, that trend points to an AI system capable of fully autonomously designing and developing its own successor. This is called recursive self-improvement. We are not there yet, and recursive self-improvement is not inevitable. But it could come sooner than most institutions are prepared for," the company wrote. Also read: Google cuts more jobs in Cloud division while pouring billions into AI: Report "We believe it would be good for the world to have the option to slow or temporarily pause frontier AI development to enable societal structures and alignment research to keep up with the advance of the technology," the company added.