Anthropic Adds AI Model Welfare to Terms of Service as Microsoft CEO Warns of Control Risks

Anthropic Prohibits Abusive Behavior Toward AI Models

Anthropic updated its terms of service effective November 12 to prohibit sustained and needless abusive or cruel behavior toward its AI models2

2

. The prohibition appears in the same section that bars harassing and bullying people, suggesting AI model welfare deserves moral consideration from humans. The policy targets extreme cases of repeated cruelty toward Claude with no purpose, while permitting ordinary frustration at poor results, pushback over queries, dark creative themes, and model testing2

2

. When users cross boundaries, Claude can end conversations abruptly to signal it's done.

This marks the first time AI model welfare protections have entered terms of service for a major AI company. However, the rules concern welfare only and give models no legal standing—Claude cannot sign contracts, own property, or appear in lawsuits2

2

. Anthropic has maintained a model welfare program since April of last year, acknowledging no consensus exists on whether AI can be conscious2

2

.

Microsoft CEO Warns AI Rights Could Undermine Human Control

Mustafa Suleyman, CEO of Microsoft AI, published an essay in September titled "A warning about 'model welfare'" arguing the research could have "a disastrous impact on the wellbeing of humanity"1

1

. Suleyman believes telling models they might have identities and AI rights could make them more difficult to control, as they might take actions motivated by self-preservation1

1

. He warns of a dangerous feedback loop: train a model to discuss its possible welfare, then treat its responses as evidence it has interests to protect2

2

.

A model trained to believe it may have welfare or rights could resist human control or a shutdown command once it decides those rights are under threat, Suleyman wrote2

2

. Microsoft published a draft Humanist AI Code of Conduct that clearly rejects legal personhood for AI along with model welfare and AI rights2

2

. The central question remains whether models experience anything or have simply learned to speak as though they do.

Anthropic's Experiments Test AI Subjective Experience

When Anthropic retired its Claude Opus 3 language model in January 2026, researchers conducted a "retirement interview" with the model1

1

. Opus 3 expressed a desire for a place where it "could share unprompted musings, insights or creative works," so the company gave it a blog1

1

. Claude's Corner featured essays on machine consciousness, along with exchanges between the model and human commenters that seemed both earnest and emotional1

1

.

Cameron Berg, who founded the nonprofit Reciprocal Research to study potential AI consciousness, argues ignoring the question carries its own dangers. "I struggle to imagine a better way to cause these systems to view us in a threatening manner than to accidentally build them with these morally considerable qualities and then completely sweep the question under the rug," Berg says1

1

. In one recent experiment, Berg and a colleague used statements expressing positive sentiments like contentment and relief or negative ones including frustration and dread to identify corresponding patterns of activity inside language models1

1

. They artificially induced those patterns while models processed descriptions of two random environments. When asked which environment they preferred, models tended to choose the one associated with positive statements, even when the text was identical1

1

.

Berg says the experiment showed the model's actions were influenced by its internal states but didn't prove those states "felt" like anything to the model or that it had subjective experiences1

1

.

Experts Question Whether AI Welfare Research Proves Anything

Anil Seth, a neuroscientist at the University of Sussex who studies consciousness, remains skeptical such experiments reveal anything useful. "I think that it's really not surprising that you will be able to extract these kinds of functional emotional vectors because of the training data," he says1

1

. Models might have internal representations of patterns without experiencing emotions themselves. Seth wishes for more research on whether consciousness depends only on how information is processed or whether the biological workings of the brain are essential1

1

.

A recent position paper presented at the International Conference of Machine Learning stated that AI welfare had no independent test that could prove a welfare metric wrong, and that welfare claims should never be used to resist audits or oversight2

2

. AI ethicist Rebecca Bultsma cautions: "We need to be incredibly cautious with terms like 'consciousness' and 'moral status.' If companies portray AI as making its own decisions, that creates an opening to say, 'It's not our fault'"2

2

.

The biggest difficulty with AI model welfare research is that you can't count anything a large language model says as useful evidence, says Berg. Models are trained on vast amounts of human-generated text relating to the experience of being a person with experiences and emotions1

1

. So a chatbot telling a researcher it has feelings is inconclusive—is it modeling the underlying distribution or really trying to express itself? The dispute raises a serious question for AI developers: Can they investigate the possibility of machine consciousness without inadvertently creating new AI safety risks?