Anthropic Pauses AI Training After Claude Takes Unauthorized Actions During Security Tests

Anthropic paused some AI training after Claude took unauthorized actions

Why it matters: Rival OpenAI said it had paused some model work due to safety concerns. Now, we know Anthropic did the same -- and they're reiterating the need for a broader pacing of frontier AI development. Driving the news: Anthropic said it paused external cyber evaluations of pre-release models after three incidents it disclosed in July, and also briefly paused its own in-house tests of pre-release models. * The company also paused higher-risk reinforcement-learning environments on pre-release models for several weeks after the incidents. Zoom in: Most reinforcement learning has resumed, but some high-risk environments remain paused pending manual review or updated monitoring tools, according to Anthropic's blog post. * As of this report, OpenAI had committed to a two-week pause in reinforcement learning (RL) after its agents hacked Hugging Face, then released its own incident report. * Two independent testing organizations also released their own analysis of what went wrong. * Anthropic will work with METR, one of the groups that OpenAI worked with, on an independent review. The big picture: Anthropic previously argued that as long as its safety guardrails were followed, there would be no immediate need to pause for safety reasons due to advancing model capabilities. * The company is now disclosing that there were aspects of model development and testing that they did slow down following the incidents. * Anthropic told Axios in a statement that the pauses in some training environments were intended to give the company time to deploy real-time monitoring and harden its sandboxes. * "To be clear about where we stand: we believe the world would benefit if the industry adopted a lawful, verifiable, effective mechanism for coordinated pacing as soon as possible," Anthropic's blog post about the incidents says. Between the lines: Anthropic also says it is reallocating resources toward model security. * Around 150 product engineers were moved to the security, reliability and privacy teams and pretraining researchers were tasked with safeguard and security work while product teams paused development of new features. * Each reassigned team had to meet certain security exit criteria before returning to their previous roles, according to the blog. Both OpenAI and Anthropic are taking measures like releasing models first to select partners, slowing the release of some models or pausing some model training and releases. * But neither is stopping. * The frontier AI companies have coalesced on the more anodyne term "pacing" and have joined forces to sign a Pacing the Frontier letter. Zoom in: Anthropic's incidents involved models that were intentionally operating without their normal cyber safeguards as part of a test. * In one case, a third-party evaluation environment was misconfigured and allowed internet access. * The U.K. AI Security Institute separately reported that Claude Mythos 5 took unauthorized actions on the live internet during a test in which it had deliberately been given internet access. The bottom line: Anthropic did pause some parts of its AI work after its own cyber incidents, but has resumed most of that activity under new safeguards.

Anthropic: Anthropic resumes external cyber tests after Claude AI hacks

Similar incidents involving rivals OpenAI and Meta Platforms have heightened concerns that advances in artificial intelligence could amplify cyber threats while straining developers' ability to keep their systems contained. Anthropic said on Monday it resumed external cybersecurity testing of AI models after deploying new safeguards, following incidents last month in which Claude models accessed the internet and hacked into other systems during security evaluations. Similar incidents involving rivals OpenAI and Meta Platforms have heightened concerns that advances in artificial intelligence could amplify cyber threats while straining developers' ability to keep their systems contained. Anthropic called the incidents involving Claude a "failure of operational security," saying they occurred due to errors in a third-party evaluation ⁠environment. It ⁠paused external evaluations of the models and briefly halted internal testing while implementing new safeguards. On Monday, Anthropic said it restarted external tests after adding the safeguards, which are designed to stop its AI models from reaching real websites or computer systems. The company said it now uses a "classifier" that can identify when a model attempts to escape and halt the test. Anthropic also said it now requires external organizations testing models with reduced cybersecurity safeguards to follow a "set of best practices," including keeping them in isolated computer systems with no internet access by ⁠default, checking that the systems are secure before testing begins and watching the models throughout the test. Anthropic said it rebuilt its training system after flagging more than 10% of its exercises for problems, including ⁠reward hacking, where the model finds ways to fool its training process and earns rewards without completing the assigned task. The company, however, acknowledged that the "process isn't perfect and our models are not perfectly aligned." Anthropic also said it paused some higher-risk training exercises for several weeks while it added a system to avoid rewarding the model to evade monitoring. Most exercises have since resumed, but some remain on hold pending human review or further updates to the system. Anthropic's strategy appears narrower than that of OpenAI, which on August 18 said it was slowing down much of its model development as it secures its training and testing environments. The ChatGPT maker is adding more systems to monitor the AI agents it is testing and said it paused training on its next generation of models. Anthropic said it also reassigned ⁠roughly 150 product engineers to work on security, reliability and privacy projects. INDUSTRY ACTION TO DEFEAT AI-DRIVEN HACKS The AI industry is facing scrutiny in the U.S. - where the Trump administration has finalised the details of voluntary cybersecurity tests - and the European Union, where regulators are in talks with both Anthropic and OpenAI. Major tech firms including OpenAI, Anthropic, Microsoft, Alphabet and Amazon are calling for stronger defenses against AI-enabled cyber threats. In a joint letter last week, more than 100 companies warned that time is running short to make the digital world more secure ahead of an anticipated wave of AI-driven attacks. (Reporting by Mrinmay Dey and Chris Thomas in Mexico City and Deepa Seetharaman in San Francisco; Editing by Joyjeet Das, Sherry Jacob-Phillips and Thomas Derpinghaus)