Anthropic Proposes Court Orders to Block Risky AI Models in Letter to California Governor

Anthropic Recommends Court-Based AI Safety Regulation

Anthropic sent an October 15 letter to California Governor Gavin Newsom proposing that government officials should be able to seek court orders blocking the release of AI models that pose a serious risk to the public1

1

. The policy proposals echo the company's earlier positions on AI safety regulation and were submitted after Newsom convened a panel of artificial intelligence experts in September to develop AI safety requirements3

3

. "When a review finds a model poses a serious risk, the government should be able to ask a court to block it," the letter stated2

2

. An Anthropic spokesperson confirmed the letter is authentic, while Politico first reported the details4

4

.

Opposition to Kill Switches for AI Models

Anthropic explicitly opposes requirements that developers build kill switches capable of shutting down entire AI models in the event a company loses control over their actions1

1

. The company argues such measures could provide a "false sense of reassurance" to government officials3

3

. Instead, Anthropic supports more targeted interventions, such as disabling a problematic AI agent or cutting off a specific customer's access to a model, which the company believes would be more effective2

2

. This stance positions Anthropic AI models within a framework of AI governance that prioritizes precision over broad shutdown capabilities.

Anthropic's Experience with Government Restrictions

Anthropic itself has faced government restrictions on its AI systems, providing the company firsthand experience with AI safety regulation. The Trump administration in June ordered Anthropic to immediately restrict foreign access to its cyber-capable Mythos 5 and Fable 5 models, citing national security concerns1

1

. The US Commerce Department later lifted the order3

3

. This experience likely informed the company's nuanced approach to proposing how authorities should block risky AI models while maintaining operational flexibility.

Federal Legislative Efforts on AI Safety

A bipartisan group of US senators in September debated legislation that would give the US government the power to block the release of certain AI models deemed unsafe, with companies able to challenge that decision in federal court2

2

. However, Congress left for its pre-election break shortly after, and lawmakers have not moved forward on that proposal1

1

. US House lawmakers in July proposed the AI Kill Switch Act, which would give US officials the power to order AI firms to shut down models that put human life or the economy at risk4

4

. The parallel federal and state efforts signal growing momentum around establishing comprehensive AI governance frameworks.

Implications for AI Industry and Upcoming IPO

Anthropic is preparing for a US initial public offering that would represent one of the most consequential stock market debuts in years, potentially reshaping benchmark indexes, investor flows and the broader narrative driving US equities3

3

. The company's proactive engagement with policymakers on AI safety could position it favorably as regulatory frameworks take shape. OpenAI on October 20 also sent a list of recommendations to Newsom, indicating that major AI companies are actively shaping the conversation around AI safety requirements3

3

. As California and federal authorities develop regulations, the industry should watch whether court-based oversight mechanisms gain traction over mandatory technical controls like kill switches, and how these policy proposals influence the competitive landscape for AI development.