Anthropic Strengthens AI Safety Measures with Updated Responsible Scaling Policy

Anthropic just made it harder for AI to go rogue with its updated safety policy

The policy, originally introduced in 2023, has evolved with new protocols to ensure that AI models, as they grow more powerful, are developed and deployed safely. This revised policy sets out specific Capability Thresholds -- benchmarks that indicate when an AI model's abilities have reached a point where additional safeguards are necessary. The thresholds cover high-risk areas such as bioweapons creation and autonomous AI research, reflecting Anthropic's commitment to prevent misuse of its technology. The update also brings new internal governance measures, including the appointment of a Responsible Scaling Officer to oversee compliance. Anthropic's proactive approach signals a growing awareness within the AI industry of the need to balance rapid innovation with robust safety standards. With AI capabilities accelerating, the stakes have never been higher. Why Anthropic's Responsible Scaling Policy matters for AI risk management Anthropic's updated Responsible Scaling Policy arrives at a critical juncture for the AI industry, where the line between beneficial and harmful AI applications is becoming increasingly thin. The company's decision to formalize Capability Thresholds with corresponding Required Safeguards shows a clear intent to prevent AI models from causing large-scale harm, whether through malicious use or unintended consequences. The policy's focus on Chemical, Biological, Radiological, and Nuclear (CBRN) weapons and Autonomous AI Research and Development (AI R&D) highlights areas where frontier AI models could be exploited by bad actors or inadvertently accelerate dangerous advancements. These thresholds act as early-warning systems, ensuring that once an AI model demonstrates risky capabilities, it triggers a higher level of scrutiny and safety measures before deployment. This approach sets a new standard in AI governance, creating a framework that not only addresses today's risks but also anticipates future threats as AI systems continue to evolve in both power and complexity. How Anthropic's capability thresholds could influence AI safety standards industry-wide Anthropic's policy is more than an internal governance system -- it's designed to be a blueprint for the broader AI industry. The company hopes its policy will be "exportable," meaning it could inspire other AI developers to adopt similar safety frameworks. By introducing AI Safety Levels (ASLs) modeled after the U.S. government's biosafety standards, Anthropic is setting a precedent for how AI companies can systematically manage risk. The tiered ASL system, which ranges from ASL-2 (current safety standards) to ASL-3 (stricter protections for riskier models), creates a structured approach to scaling AI development. For example, if a model shows signs of dangerous autonomous capabilities, it would automatically move to ASL-3, requiring more rigorous red-teaming (simulated adversarial testing) and third-party audits before it can be deployed. If adopted industry-wide, this system could create what Anthropic has called a "race to the top" for AI safety, where companies compete not only on the performance of their models but also on the strength of their safeguards. This could be transformative for an industry that has so far been reluctant to self-regulate at this level of detail. The role of the responsible scaling officer in AI risk governance A key feature of Anthropic's updated policy is the creation of a Responsible Scaling Officer (RSO) -- a position tasked with overseeing the company's AI safety protocols. The RSO will play a critical role in ensuring compliance with the policy, from evaluating when AI models have crossed Capability Thresholds to reviewing decisions on model deployment. This internal governance mechanism adds another layer of accountability to Anthropic's operations, ensuring that the company's safety commitments are not just theoretical but actively enforced. The RSO will also have the authority to pause AI training or deployment if the safeguards required at ASL-3 or higher are not in place. In an industry moving at breakneck speed, this level of oversight could become a model for other AI companies, particularly those working on frontier AI systems with the potential to cause significant harm if misused. Why Anthropic's policy update is a timely response to growing AI regulation Anthropic's updated policy comes at a time when the AI industry is under increasing pressure from regulators and policymakers. Governments across the U.S. and Europe are debating how to regulate powerful AI systems, and companies like Anthropic are being watched closely for their role in shaping the future of AI governance. The Capability Thresholds introduced in this policy could serve as a prototype for future government regulations, offering a clear framework for when AI models should be subject to stricter controls. By committing to public disclosures of Capability Reports and Safeguard Assessments, Anthropic is positioning itself as a leader in AI transparency -- an issue that many critics of the industry have highlighted as lacking. This willingness to share internal safety practices could help bridge the gap between AI developers and regulators, providing a roadmap for what responsible AI governance could look like at scale. Looking ahead: What Anthropic's Responsible Scaling Policy means for the future of AI development As AI models become more powerful, the risks they pose will inevitably grow. Anthropic's updated Responsible Scaling Policy is a forward-looking response to these risks, creating a dynamic framework that can evolve alongside AI technology. The company's focus on iterative safety measures -- with regular updates to its Capability Thresholds and Safeguards -- ensures that it can adapt to new challenges as they arise. While the policy is currently specific to Anthropic, its broader implications for the AI industry are clear. As more companies follow suit, we could see the emergence of a new standard for AI safety, one that balances innovation with the need for rigorous risk management. In the end, Anthropic's Responsible Scaling Policy is not just about preventing catastrophe -- it's about ensuring that AI can fulfill its promise of transforming industries and improving lives without leaving destruction in its wake.

Anthropic updates policy to address AI risks

Anthropic has developed a framework for assessing different AI capabilities to be better able to respond to emerging risks. Anthropic, the AI safety and research start-up behind the chatbot Claude, has updated its scaling policy, which develops a more flexible approach to assessing and managing AI risks. To help with this new approach, the start-up is hiring a number of roles focused on risk management, including a head of responsible scaling. The risks associated with AI and its capabilities are growing exponentially as the technology develops at a staggering speed. Last year, the Nobel Laureate known as the "godfather of AI" Geoffrey Hinton quit Google to speak openly about the dangers of AI. "Given the rate of progress, we expect things to get better quite fast," Hinton told the BBC at the time. "So we need to worry about that." First announced in September 2023, Anthropic's Responsible Scaling Policy is a framework that looks to manage risks from increasingly "capable" AI systems. The framework proposes increased security and safety measures depending on the AI model's capability - the higher its capability, the higher the security measures. In the announcement yesterday (15 October), Anthropic said it maintains its commitment not to train or deploy AI models "unless we have implemented safety and security measures that keep risks below acceptable levels", while making updates to how it perceives and addresses emerging risks. Examples of the lowest risk AI safety level-1 (ASL), the start-up said, include old large language models (LLM), while a step up to ASL-2 include most current LLMs, including Anthropic's own Claude, that have the ability to provide dangerous information - however, not more than what a search engine could. The higher risk ASL-3 includes models that show low-level autonomous capability while the higher ASL-4 and up is reserved for future advances, with Anthropic saying this technology could have "catastrophic misuse potential and autonomy". Anthropic has now updated its methodology for assessing specific capabilities of AI models and their associated risks to include a focus on capability thresholds, that is "specific AI abilities that, if reached, would require stronger safeguards than our current baseline" and required safeguards, which are "the specific ASL standards needed to mitigate risks once a capability threshold has been reached". Anthropic said that all of its current models meet the ASL-2 standard. However, if a model can conduct complex AI research tasks usually requiring human expertise, this would meet a capability threshold and require the greater security of ASL-4 or higher, the company said. Also, if a model can "meaningfully assist someone with a basic technical background" in creating or deploying chemical, biological or nuclear weapons, this would meet another capability threshold and require ASL-3 standards of security and deployment safeguards. The AI start-up said will conduct routine evaluations of its AI models to ensure its currently applied safeguards are appropriate. Jared Kaplan, the Anthropic co-founder and chief science officer, who previously worked as a research consultant at OpenAI, will take over as the start-up's responsible scaling officer, a role that was previously held by co-founder and CTO Sam McCandlish. Anthropic, founded in 2021 by former employees of ChatGPT-creator OpenAI, positions itself as a safety-oriented AI company. Earlier this year, the company announced the opening of an office in Dublin, saying that it will hopefully be its main establishment in the EU market. Don't miss out on the knowledge you need to succeed. Sign up for the Daily Brief, Silicon Republic's digest of need-to-know sci-tech news.