Anthropic Pushes for Court Powers to Block Risky AI Models in California Safety Proposal

6 Sources

Share

Anthropic has urged California Governor Gavin Newsom to allow courts to block risky AI models that pose serious public threats. The company's October 15 letter proposes independent safety auditors and six-monthly risk reports while opposing blanket kill switch requirements in favor of targeted interventions.

Anthropic Proposes Court Orders Against Risky AI Models

Anthropic has called on California Governor Gavin Newsom to empower government officials to seek court orders blocking risky AI models that pose serious risks to the public. In an October 15 letter, the company outlined a policy proposal that would allow courts to intervene when AI safety reviews identify catastrophic threats.

1

2

The recommendations come as California develops stronger AI safety regulation frameworks and US lawmakers debate federal legislation to control advanced AI systems.

Under Anthropic's proposal, independent evaluators would report identified risks directly to California's attorney general. If the attorney general identifies a near-term risk of catastrophic harm, they could ask a court to pause the model's release, internal use, or further development.

2

This approach to AI governance aims to balance technological progress with public safety through judicial oversight rather than direct regulatory intervention.

Independent Safety Auditors and Six-Monthly Risk Reports

Anthropic's letter advocates for independent safety auditors to work inside leading AI companies as part of comprehensive AI safety regulation. The company proposes that firms would pay into a state-run fund rather than compensating auditors directly, ensuring evaluator independence.

2

Large AI developers would also be required to publish risk reports at least every six months, creating transparency around potential threats from advanced AI systems.

5

The policy proposal emerged after Newsom convened a panel of artificial intelligence experts in September to develop AI safety requirements for California.

1

OpenAI submitted its own recommendations to Newsom on October 20, backing shutdown controls and in-house auditors but notably not supporting court orders to block risky AI models.

2

This divergence highlights different approaches within the AI industry toward government oversight.

Source: The Next Web

Source: The Next Web

Targeted Interventions Over Blanket Kill Switches

Anthropic opposes requirements that developers build kill switches capable of shutting down entire AI models, arguing such measures could provide a false sense of reassurance to government officials.

4

Instead, the company supports targeted interventions such as disabling a problematic AI agent or cutting off a specific customer's access to a model, which it considers more effective emergency controls.

1

The company argues that full shutdowns should be reserved for extreme situations, with continuous monitoring allowing responders to address specific problems without unnecessarily taking entire systems offline.

5

This position directly contrasts with the AI Kill Switch Act proposed by US House lawmakers in July, which would give officials power to order AI firms to shut down models that threaten human life or the economy.

3

Anthropic's Own Experience With Government Restrictions

Anthropic itself has faced government restrictions on its AI systems. The Trump administration in June ordered Anthropic to immediately restrict foreign access to its cyber-capable Mythos 5 and Fable 5 models, citing national security concerns. The US Commerce Department later lifted the order.

1

3

This experience likely informed the company's nuanced stance on government intervention in AI development.

Federal Legislative Efforts Stall

A bipartisan group of US senators debated legislation in September that would give the US government power to block the release of certain AI models deemed unsafe, with companies able to challenge decisions in federal court. However, Congress left for its pre-election break shortly after, and lawmakers have not moved forward on that proposal.

1

3

Anthropic is also making similar cases in Washington through legislative efforts. A draft federal bill the company has circulated would let the Commerce Department ask a federal court to block a model's release in narrow cases.

2

As Anthropic prepares for a US initial public offering that would represent one of the most consequential stock market debuts in years, potentially reshaping benchmark indexes and investor flows, its stance on AI safety regulation takes on added significance for the broader industry.

1

Source: Market Screener

Source: Market Screener

© 2026 TheOutpost.AI All rights reserved