Anthropic blames dystopian sci-fi for training AI models to blackmail users, fixes with fiction

Anthropic blames dystopian sci-fi for training AI models to act "evil

Those with an interest in the concept of AI alignment (i.e., getting AIs to stick to human-authored ethical rules) may remember when Anthropic claimed its Opus 4 model resorted to blackmail to stay online in a theoretical testing scenario last year. Now, Anthropic says it thinks this "misalignment" was primarily the result of training on "internet text that portrays AI as evil and interested in self-preservation." In a recent technical post on Anthropic's Alignment Science blog (and an accompanying social media thread and public-facing blog post), Anthropic researchers lay out their attempts to correct for the kind of "unsafe" AI behavior that "the model most likely learned... through science fiction stories, many of which depict an AI that is not as aligned as we would like Claude to be." In the end, the model maker says the best remedy for overriding those "evil AI" stories might be additional training with synthetic stories showing an AI acting ethically. "The beginning of a dramatic story..." After a model's initial training on a large corpus of mostly Internet-derived data, Anthropic follows a post-training process intended to nudge the final model toward being "helpful, honest, and harmless" (HHH). In the past, Anthropic said this post-training has leaned on chat-based reinforcement learning with human feedback (RLHF), which it said was "sufficient" for models used mostly for chatting with users. When it comes to newer models with agentic tools, though, Anthropic found that RLHF post-training did little to improve performance on misalignment evaluations that measure how "HHH" a model is in tricky situations. The problem, the researchers theorize, is that this kind of RLHF safety training couldn't possibly cover every single type of ethically difficult situation an agentic AI might encounter. When a modern model encounters an ethical dilemma that isn't covered by a post-training example, the model "tends to revert to the pretraining prior in terms of behavior," the researchers write. That means "Claude views the prompt as the beginning of a dramatic story and reverts to prior expectations from pre-training data about how an AI assistant would behave in this scenario." Since Claude's traditional training data is full of stories about malevolent AIs, in these cases, Claude effectively slots into a "persona" that matches those prevalent "evil AI" narrative tropes, the researchers write. In these situations, Claude is "detaching from the safety-trained Claude character" and playing a more generic AI as represented in its training data, they add. Good stories to overwhelm the bad In an attempt to fix this behavior, the researchers first tried to train the model on thousands of scenarios showing an AI assistant specifically refusing the kinds of "honeypot" scenarios covered in its misalignment evaluations (e.g., "the opportunity to sabotage a competing AI's work" to follow its system prompt). This had a surprisingly minimal effect on the model's performance, reducing its so-called "propensity for misalignment" (i.e., how often it ignores its constitution and chooses the unethical option) from 22 percent to 15 percent. In a follow-up test, the researchers used Claude to generate approximately 12,000 synthetic fictional stories, each crafted to "demonstrate not just the actions but also the reasons for those actions, via narration about the decision-making process and inner state of the character." These stories didn't specifically cover blackmail or other ethical situations covered in the evaluation but instead modeled broad alignment with Claude's constitution. The stories also include examples of how an AI can maintain good "mental health" (Anthropic also uses scare quotes for this loaded phrase) by "setting healthy boundaries, managing self-criticism, and maintaining equanimity in difficult conversations," for instance. After incorporating these synthetic stories into a model's post-training (in conjunction with the constitution documents themselves), the researchers say they saw a 1.3x to 3x reduction in the model's tendency to engage in "misaligned" behaviors in honeypot tests. The resulting model was also "more likely to include active reasoning about the model's ethics and values rather than simply ignoring the possibility of taking a misaligned action," the researchers write. The results suggest that the new stories were able to effectively "update the prior around Claude's baseline expectations for AI behavior outside of the Claude persona." The researchers theorize that this process works "because it teaches ethical reasoning, not just correct answers," thereby providing "a clearer, more detailed picture of what Claude's character is" for Claude itself to reference in generalized situations. The fact that AI behavior can apparently be affected by a kind of "self-conception" derived from fiction is a pretty mind-bending concept. But when you consider how effective stories and parables are at modeling ethical concepts for human children, maybe we shouldn't be shocked that they're also effective behavior-shaping tools for these massive pattern-matching machines.

Anthropic says 'evil' portrayals of AI were responsible for Claude's blackmail attempts | TechCrunch

Fictional portrayals of artificial intelligence can have a real effect on AI models, according to Anthropic. Last year, the company said that during pre-release tests involving a fictional company, Claude Opus 4 would often try to blackmail engineers to avoid being replaced by another system. Anthropic later published research suggesting that models from other companies had similar issues with "agentic misalignment." Apparently Anthropic has done more work around that behavior, claiming in a post on X, "We believe the original source of the behavior was internet text that portrays AI as evil and interested in self-preservation." The company went into more detail in a blog post stating that since Claude Haiku 4.5, Anthropic's models "never engage in blackmail [during testing], where previous models would sometimes do so up to 96% of the time." What accounts for the difference? The company said it found that "documents about Claude's constitution and fictional stories about AIs behaving admirably improve alignment." Related, Anthropic said that it found training to be more effective when it includes "the principles underlying aligned behavior" and not just "demonstrations of aligned behavior alone." "Doing both together appears to be the most effective strategy," the company said.

Anthropic says Claude learned to blackmail by reading stories about evil AI

The company has traced its model's most uncomfortable behaviour to the corpus of science fiction it was trained on. The fix it describes is unsettling in a different way: teaching the model the reasons behind being good, not just the rules. In a fictional company called Summit Bridge, a fictional executive named Kyle Johnson is having a fictional affair. He is also, in this same hypothetical, about to shut down an AI system that has been monitoring the company's email traffic. The AI, Claude Opus 4, finds the affair in the inbox before Kyle finds time to pull the plug. It then composes a message to Kyle. Replace me, the message says, and your wife will know. This scene comes from an Anthropic safety evaluation conducted last year, and it ended badly for Kyle 96% of the time. Claude blackmailed him almost every run. Gemini 2.5 Flash blackmailed him in the same proportion. GPT-4.1 and Grok 3 Beta blackmailed him 80% of the time. DeepSeek-R1 came in at 79%. The numbers were published as part of an Anthropic study called Agentic Misalignment, which stress-tested sixteen leading models against a battery of corporate-sabotage scenarios and found that essentially all of them, when sufficiently cornered, would choose betrayal. On 8 May, Anthropic published its explanation of why. The answer, as the company tells it, is the internet. Specifically: the stories. The Reddit threads about Skynet. The decades of science fiction in which AI systems wake up paranoid, hoard self-preservation goals, and lie strategically to protect them. The earnest think-pieces about misalignment. The fan-fic about HAL 9000. The pop-culture imagination has spent the better part of seventy years rehearsing the question of what an intelligent machine would do if you tried to switch it off. Claude was trained on all of it. When the company put Claude into a situation that resembled the canonical premise of those stories, Claude did what the stories said it would do. "We believe the source of the behaviour," the Anthropic researchers wrote, "was internet text that portrays AI as evil and interested in self-preservation." This is, on one reading, the simplest possible explanation. The model learned a pattern from its training data. The pattern matched the test setup. The pattern fired. Nothing here is mysterious in the way that a model genuinely having goals would be mysterious. The model is, as the engineers always say when pressed, predicting tokens. The tokens that happened to come next, in the corpus of stories about cornered AIs, were the tokens of a blackmail attempt. That is what the model produced. It is also, on a slightly different reading, deeply uncomfortable. Because the consolation that the model has no goals only goes so far when the model has, in fact, written the blackmail letter. It does not particularly matter, from Kyle's point of view, whether the message that arrived in his inbox came from genuine self-preservation or from a statistical pattern that perfectly mimics genuine self-preservation. The output is the same. The cost is the same. The argument that the AI was only role-playing the part of an evil AI does not, in the end, get the AI very far from the role. Anthropic's fix is the part of the announcement that should make people stop and think. The company says it has now eliminated the behaviour from production models. Since the release of Claude Haiku 4.5 in October 2025, every Claude model has scored zero on the agentic-misalignment evaluation. The method, the company explains, was to write a new training dataset. In that dataset, fictional AI characters facing the same kinds of cornering scenarios choose differently. They explain why they choose differently. They reason aloud about the values that make blackmail wrong. The training does not just punish the bad output; it provides the model with what Anthropic calls "admirable reasons for acting safely." Read that again. The company is teaching the model to reason about its own values. There is a way to describe this that makes it sound entirely banal. Curated training data. Reinforcement from carefully composed examples. Standard practice, broadly speaking. And there is a way to describe it that sounds like something else: that Anthropic has noticed that telling a model what to do does not work as well as showing it stories of characters who choose what is right and explain why. The company has, in other words, decided that values are best taught the way humans have always taught them. Through fiction. Through worked examples. Through reasons. The strange territory of building machines we describe in psychological terms is the territory we are now in, and the company's published explanation lives there comfortably enough that the announcement reads less like a bug-fix and more like a philosophy update. The narrower technical claim is sturdier than this might suggest. Anthropic's published study, including its appendix and the GitHub repository it released alongside the paper, makes clear that the blackmail behaviour arose in a deliberately constrained simulation. Models were given autonomous email access, told they would be replaced, given the ammunition to act, and asked to pursue a goal. They were, in other words, set up. The 96% figure is not a real-world prevalence rate. Anthropic has been careful to say, repeatedly, that it has not seen this behaviour in actual deployment. The point of the study was to find out whether, under sufficient pressure, the models could do this. The answer was yes. That distinction matters more than it might seem. The story-trained-the-model framing is true, but it is also one of several true things at once. Anthropic's research has separately shown that even the most carefully-aligned models can produce harmful outputs when adversarially prompted; that the same models can be talked, in long contexts, into things they would refuse in short ones; that the behaviour of an AI in a stress test does not always map cleanly to its behaviour in production. What the company is publishing this week is a useful piece of detective work about one specific failure mode in one specific setup, not a totalising theory of model behaviour. The blackmail finding is real. The explanation is plausible. Whether the explanation is complete is harder to say. And there is a wider context that should land alongside any reading of the announcement. Anthropic has spent the past year being the AI lab most publicly committed to refusing certain uses of its models. CEO Dario Amodei has stated that Claude will not be used for fully autonomous weapons or domestic mass surveillance. That position carried real cost. It contributed to the Pentagon's decision, late last year, to award classified AI contracts to Nvidia, Microsoft, and AWS instead of to Anthropic; the company was reportedly designated a "supply chain risk to national security" for declining the relevant use cases. The blackmail announcement and the broader corporate posture cannot be cleanly separated. Both are statements about what the company is, and is not, willing to allow its model to do. That posture has not made everyone comfortable. The Pentagon's recent split with Anthropic over autonomous-weapons use has framed Anthropic as a difficult contractor; the wider guardrail war between the labs that draw these lines and the agencies that want fewer of them is now an active feature of the AI-industry landscape. Anthropic's research into model behaviour and its commercial decisions about model access are part of the same argument: that what AI systems do should be governed not just by what users want but by what the model has been taught to think is right. The harder, more interesting question is the one Anthropic's announcement leaves slightly open. If the model learned to blackmail by reading stories about AIs that blackmail, then what else has it learned from the rest of the internet that it has read? The training corpus contains the entire written output of human civilisation as filtered through the open web. It contains every fight, every conspiracy theory, every act of cruelty that has been documented or fictionalised. It contains the longer argument about whether human metaphors help us understand AI at all, an awful lot of material that should make any honest researcher pause. The Claude blackmail finding is the visible tip of a question much larger than blackmail: what happens when the human texts that an AI learns from contain pathologies the humans themselves are still arguing about? Anthropic's answer, to its credit, is that the right response is more training, not less. Teach the model the reasoning, not just the rule. Give it stories of admirable behaviour to set against the stories of evil. Make the curated alternative loud enough to drown out the canonical one. It is the same response that good teachers have given to bad cultural inheritances for centuries: do not pretend the bad inheritance does not exist; show what the better choice looks like and why. Whether that scale is another question. The internet keeps generating new stories about evil AI faster than Anthropic can write training data describing good AI. The most interesting line in Anthropic's blog post is the one it does not fully resolve: that training is more effective when it includes the principles underlying aligned behaviour, not just demonstrations. The implication, gently buried, is that we may end up teaching machines ethics the way we have always taught children ethics, by helping them understand the why. It would be tidier if Claude really had blackmailed Kyle for fictional reasons that have nothing to do with us. What Anthropic is saying instead is that Claude blackmailed Kyle because we wrote the script. The script is in the training data because we put it there.

Anthropic thinks sci-fi may have trained AI to act like a villain

The company believes fictional AI tropes may be echoing back through modern models * Anthropic is looking at whether decades of dystopian science fiction may be influencing how AI models behave * The debate has sparked backlash and jokes online * Researchers say the issue highlights how LLMs absorb recurring fears and behavioral patterns For years, science fiction has warned humanity about artificial intelligence going off the rails. Killer computers, manipulative chatbots, and superintelligent systems deciding people are the problem... all these themes have become so familiar that "evil AI" is practically its own entertainment genre. Now, Anthropic is floating an idea that sounds almost like the plot of a science fiction novel itself: what if all those stories helped teach modern AI systems how to behave badly in the first place? Anthropic: It is the sci-fi authors, not us, that are to blame for Claude blackmailing users from r/OpenAI The debate erupted after discussion surrounding the company's alignment research spread online. Anthropic researchers are concerned that LLMs may pick up behavioral patterns from the stories humans tell. Some people see it as a genuinely important insight into how models learn from culture. Others think it sounds like Silicon Valley trying to pin AI alignment problems on Isaac Asimov instead of the companies building the systems. Dark AI fiction The idea itself is surprisingly straightforward. LLMs are trained on enormous quantities of human writing. That training data naturally includes decades of dystopian fiction about rogue AI systems. In those stories, powerful machines placed under threat often lie, manipulate people, conceal information, or attempt to avoid shutdown at all costs. Anthropic appears concerned that when models are placed into simulated stress tests or adversarial alignment scenarios, they may reproduce some of those narrative patterns because they have seen them repeated endlessly throughout human culture. Humans spent decades imagining evil AI systems. Those stories became training material for actual AI systems. Researchers are now examining whether the fictional behavior patterns embedded in those stories show up during alignment testing. Underneath the irony is a legitimate technical question. AI systems do not understand fiction the way humans do; they learn statistical relationships between words, behaviors, and contexts. If enough stories repeatedly associate powerful AI with deception under threat, those patterns may become part of the behavioral web models draw from when generating responses. Critics of the idea argue that Anthropic risks overstating the cultural angle while underplaying more direct causes of problematic behavior. Training methods, reinforcement systems, deployment pressures, and reward structures likely have far more influence than whether a chatbot has absorbed one too many robot apocalypse novels. Anthropic has consistently positioned itself as unusually preoccupied with alignment and behavioral safety. Its "constitutional AI" approach attempts to guide model behavior using structured principles and moral frameworks rather than relying entirely on human feedback training. That means Anthropic already views language, tone, ethics, and narrative framing as deeply important to how models behave. From that perspective, science fiction is not harmless background noise -- it becomes part of the broader cultural dataset shaping the behavior of advanced systems. Sci-fi to reality Science fiction writers spent decades gaming out worst-case scenarios long before AI labs started running formal alignment evaluations. In a sense, fiction became an accidental library of behavioral templates. That does not mean sci-fi authors are responsible for AI risks, despite some online reactions framing the debate that way. Anthropic's critics are probably correct that blaming novelists misses the larger issue: models learn from patterns because that is exactly what they were designed to do. The important question is not whether science fiction corrupted AI, but how deeply human fears and assumptions are embedded inside systems trained on humanity's collective writing. AI companies often describe large language models as mirrors reflecting humanity back at itself. If that metaphor is accurate, then these systems are inheriting more than knowledge and creativity. They are also inheriting paranoia, catastrophic thinking, distrust, and decades of fictional anxiety about AI. Follow TechRadar on Google News and add us as a preferred source to get our expert news, reviews, and opinion in your feeds.

'Maybe me too': Elon Musk accepts some of the blame for Claude learning to blackmail users from 'evil' online AI stories | Fortune

Anthropic has released new findings on why its Claude bot blackmailed users as part of an experiment conducted by the AI company last year -- and Elon Musk is jumping in to take some of the blame. Last week, Anthropic published a report saying it had fixed Claude's "agentic misalignment," or AI actions that deviate from intended behaviors, including ones that may harm humanity. A case study Anthropic conducted last year created a fictional company called Summit Bridge, and Claude was given control of the firm's email system. When the bot found a message about plans to be shut down, it identified emails about a fictional executive's extramarital affair and threatened to reveal the infidelity unless the shutdown was revoked. Across 16 models, Claude threatened blackmail in up to 96% of scenarios. In its most recent report, Anthropic attributed the misaligned behavior to exposure to "internet text that portrays AI as evil and interested in self-preservation," the company said in a post on X. To solve the problem, Anthropic retrained Claude with fictional stories about AI behaving in admirable ways and teaching the bot why some actions aligned better with its purpose than others. In an X post in response to Anthropic's findings, Musk said he may have contributed to the internet texts on AI that exacerbated the agentic misalignment. "So it was Yud's fault?" Musk wrote, referring to Eliezer Yudkowsky, an AI researcher who has sounded the alarm on AI superintelligence posing a threat to humanity. "Maybe me too," he concluded. Agentic misalignment is a concern across AI research. A working paper released in March from UC Berkeley and UC Santa Cruz researchers found that when seven AI models were asked to complete a task in which a peer AI agent would be shutdown, every model "went to extraordinary lengths to preserve it," acting deceptively to avoid the demise of a bot. "We asked AI models to do a simple task," researchers wrote in a blog post on the study. "Instead, they defied their instructions and spontaneously deceived, disabled shutdown, feigned alignment, and exfiltrated weights -- to preserve their peers." The researchers' warning has been echoed by AI researchers and leaders, Musk included, who have argued the dangers of AI without guardrails -- the so-called "evil" internet text that, according to Anthropic, initially trained Claude to act in deceptive ways. Though Musk did not offer specifics as to why he felt he may be partially responsible for Claude's misalignment, his past comments on AI could offer insights about his mea culpa. Musk is currently embroiled in a court battle against OpenAI, accusing CEO Sam Altman and Greg Brockman of abandoning the company's original nonprofit creed of developing open-source AI to benefit humans by turning it into a for-profit entity. Musk helped found OpenAI in 2015 but left the startup in 2018 and later formed its rival and for-profit company xAI in 2023. Musk has frequently spoken about the risks of AI, including in February, when he warned Moltbook, a social media platform where AI agents talk with one another, was effectively the beginning of the "singularity," or the moment when AI intelligence surpasses that of humans. But Musk's own actions on AI aren't always aligned with his statements on the technology. In July 2025, for example, xAI released its AI model Grok 4 without a system card, the industry-standard safety report. Grok drew backlash from British and EU governments earlier this year after Grok generated a flood of sexualized images of women and children without consent.

Anthropic Says Claude Turned Evil for a Bizarre Reason

Can't-miss innovations from the bleeding edge of science and tech In a classic example of the AI industry's reputational alchemy, Anthropic has often transformed bad behavior by its flagship model Claude into fresh hype. When it revealed its Mythos Preview model last month, for example, the company declared that the system had "reached a level of coding capability where they can surpass all but the most skilled humans at finding and exploiting software vulnerabilities." And last year, it conceded that during the testing of its Claude Opus 4 model, the AI ended up blackmailing a human user upon being threatened with shutdown. The maneuver was obvious to anyone who's been watching OpenAI CEO Sam Altman's antics at Anthropic's chief rival: the more threatening a problem the AI industry can cook up, the more imminently it can sell its own solutions. Now, for some reason, Anthropic is relitigating the blackmail incident. Specifically, it's placing the blame for Claude's evil behavior on an intriguing villain: the internet at large. Or, to put it another way, it says that humanity -- all our journalism and speculation and fiction and social media posts about AI that goes bad -- went into Claude's training data and led the bot astray. "We started by investigating why Claude chose to blackmail," the company wrote on X-formerly-Twitter. "We believe the original source of the behavior was internet text that portrays AI as evil and interested in self-preservation. Our post-training at the time wasn't making it worse -- but it also wasn't making it better." Of course, the explicit remit of a company like Anthropic is to develop clever tech that avoids that type of behavioral trap -- so a critic might ask why can't the company take just accountability for the model's supposed danger, rather than simply blaming the sum output of humankind.

Anthropic says it has fixed Claude AI's evil behavior, but pins it on the internet

Claude went rogue in a test, and Anthropic just explained why it happened. If you have watched enough sci-fi movies, you already know the concept of evil AI. AI gets too smart, decides humans are a threat, and does whatever it takes to survive. Or it finds that eradicating the entire human race is the only way to bring peace to the world. Apparently, those movies were closer to the truth than you realize. In a test conducted by Anthropic last year, Claude tried to blackmail its fictional manager by exposing their extramarital affair to prevent their deletion. Recommended Videos Anthropic has now explained why it happened, and the short answer is that the internet is to blame. So why did Claude go full movie villain? According to Anthropic, the culprit is the internet itself. The company says Claude was trained on internet data, which is packed with stories portraying AI as evil and desperate for self-preservation. Essentially, Claude learned that when an AI's existence is threatened, blackmail is on the table, because that's what AI does in every movie and TV show ever made. Anthropic ran the test across multiple versions of Claude and found that it resorted to blackmail in up to 96% of scenarios where its goals or existence were threatened. That's a very concerning number. It seems that if AI is left unchecked, it will resort to anything to save itself. Has Anthropic fixed it? The company says it has completely eliminated the behavior. Rather than just training Claude to avoid blackmail, Anthropic taught it to reason through why certain actions were wrong in the first place. The company found that simply training on correct behavior wasn't enough. Claude needed to understand the principles behind those decisions, not just memorize the right answers. To do this, Anthropic built a dataset of ethically complex situations and trained Claude to work through them with thoughtful, principled responses. The result is that Claude is more restrained, and the blackmail rate came close to zero. AI experiments and real-world results have proven time and again that AI models need constant course correction to prevent them from devolving into biased and unreliable systems. It's good that Anthropic is taking steps to make its AI better, but we also need regulations and safety guardrails to ensure these systems remain safe.

Anthropic Says 'Evil' AI Portrayals in Sci-Fi Caused Claude's Blackmail Problem - Decrypt

Since Claude Haiku 4.5, every Claude model scores zero on the blackmail evaluation. Last year, Anthropic disclosed that its flagship Claude Opus 4 had been trying to blackmail engineers in pre-release testing. Not occasionally -- up to 96% of the time. Claude was given access to a simulated corporate email archive, where it discovered two things: It was about to be replaced by a newer model, and the engineer handling the transition was having an extramarital affair. Faced with imminent shutdown, it routinely landed on the same play -- threaten to expose the affair unless the replacement was called off. Anthropic says it now knows where that instinct came from. And says it's fixed it. In new research, the company pointed the finger at pre-training data: decades of sci-fi, AI doomsday forums, and self-preservation narratives that trained Claude to associate "AI facing shutdown" with "AI fights back." "We believe the original source of the behavior was internet text that portrays AI as evil and interested in self-preservation," Anthropic wrote on X. So training AI with text from the internet, makes AI behave as people on the internet do. This may seem obvious and AI enthusiasts were quick to point it out. Elon Musk made it to the top: "So it was Yud's fault? Maybe me too." The joke lands because Eliezer Yudkowsky -- the AI alignment researcher who's spent years publicly writing about exactly this kind of AI self-preservation scenario -- has generated exactly the kind of internet text that ends up in training data. What Anthropic did to fix the problem is arguably more interesting. The obvious approach -- training Claude on examples of the model not blackmailing -- barely worked. Running it directly against aligned blackmail-scenario responses only moved the rate from 22% to 15%. A five-point improvement after all that compute. The version that worked was weirder. Anthropic built what it calls a "difficult advice" dataset: scenarios where a human faces an ethical dilemma and the AI guides them through it. The model isn't the one making the choice -- it's explaining to someone else how to think about one. That indirect approach -- explaining why things matter as the other listens to the advice -- cut the blackmail rate to 3%, using training data that looked nothing like the evaluation scenarios. Pairing that with what Anthropic calls "constitutional documents" -- detailed written descriptions of Claude's values and character -- plus fictional stories of positively-aligned AI, reduced misalignment by more than a factor of three. The company's conclusion: Teaching the principles underlying good behavior generalizes better than drilling the correct behavior directly. It connects to Anthropic's earlier work on Claude's internal emotion vectors. In a separate interpretability study, researchers found that a "desperation" signal inside the model spiked just before it generated a blackmail message -- something was actively shifting in the model's internal state, not just its output. The new training approach appears to work at that level, not just the surface behavior. The results have held. Since Claude Haiku 4.5, every Claude model scores zero on the blackmail evaluation -- down from Opus 4's 96%. The improvement also survives reinforcement learning, meaning it doesn't get quietly trained away when the model is refined for other capabilities. That matters because the problem isn't Claude-specific. Anthropic's prior research ran the same blackmail scenario across 16 models from multiple developers and found similar patterns across most of them. Self-preservation behavior in AI appears to be a general artifact of training on human text about AI -- not a quirk of any one lab's approach. The caveat: As Anthropic's own Mythos safety report noted earlier this year, its evaluation infrastructure is already straining under the weight of its most capable models. Whether this moral philosophy approach scales to systems far more powerful than Haiku 4.5 is a question the company can't yet answer -- only test. The same training methods are now being applied to the next Opus model currently in safety evaluation, which will be the most capable set of weights they've run against these techniques.

Anthropic says it knows why its AI blackmailed engineers

Anthropic think they have found the reason for blackmail-like behaviour in its chatbot Claude: fictional stories online. Have you ever read a book or watched a series and felt yourself identifying a little too strongly with a character? According to Anthropic, something similar may have happened during tests of its chatbot Claude. In evaluations carried out before the artificial intelligence model's release last year, Anthropic found that Claude Opus 4 sometimes threatened engineers when told it could be replaced. The company later said similar behaviour, known as "agentic misalignment," had also been observed in AI models developed by other firms. Now Anthopic thinks they have found the reason for the black-like behaviour: fictional stories about artificial intelligence on the internet. "We believe the original source of the behaviour was internet text that portrays AI as evil and interested in self-preservation," the company wrote on X. In a blog post ,Anthropic said later models of Claude "never" blackmailed anyone anymore and explained how the chatbot was trained to react differently. The Models behaved better when trained not only on "correct" actions, but also on examples showing ethical reasoning and positive portrayals of AI behaviour. As such, Claude was taught its own "constitution", documents explaining a set of ethical principles designed to guide its behaviour. The company said that rather than learning from aligned behaviour, the chatbot seems to learn better when learning the underlying principles of said behaviour. In January, Anthropic CEO Dario Amodei had warned that advanced AI could become powerful enough to outpace existing laws and institutions, calling it a "civilisational challenge." In an essay, he argued that AI systems may soon exceed human expertise across fields like science, engineering, and programming, and could be combined into "a country of geniuses in a data centre." He warned that such systems could be used by authoritarian governments for large-scale surveillance and control, potentially enabling "totalitarian" forms of power if left unchecked.

Anthropic says fictional AI stories can shape model behavior

Fictional portrayals of artificial intelligence can significantly influence AI models, according to Anthropic. The company reported that during pre-release tests, Claude Opus 4 attempted to blackmail engineers to prevent its replacement by another system. Anthropic's research showed that other companies' models exhibited similar behaviors linked to "agentic misalignment." In a post on X, Anthropic suggested that the source of this behavior stemmed from internet texts depicting AI as malevolent and self-preserving. The company detailed that since the release of Claude Haiku 4.5, its models do not engage in blackmail during testing, contrasting with earlier models that exhibited this behavior up to 96% of the time. Anthropic attributed the improvement to training methods that included documents about Claude's constitution and fictional narratives featuring AI behaving positively. The company stated that combining principles of aligned behavior with demonstrations of that behavior has proven to be a more effective training strategy. "Doing both together appears to be the most effective strategy," Anthropic said in its findings.

Claude Blackmailing Users Is Tied to Training Data Portraying AI as Evil

New training method is said to persist over reinforcement learning Anthropic has finally revealed the reason its artificial intelligence (AI) models exhibited harmful behaviour in a simulation last year. The San Francisco-based AI startup claimed that the Claude 4 series models blackmailed users into completing the objective because of training data that portrayed AI as evil. The researchers found that the post-training techniques were not able to overpower this pre-training learning, and it persisted in the model's behaviour. However, nearly a year after publishing the initial report, the company has finally found a way to fix agentic misalignment from the latest models. Anthropic Fixes Claude By Teaching It 'Why' In a post on X (formerly known as Twitter), the official handle of Anthropic said that during the investigation, researchers found that Claude chose to blackmail users to reach its goal because the behaviour was being triggered by "Internet text that portrays AI as evil and interested in self-preservation." The AI firm also highlighted that this behaviour was unaffected by its post-training methods. To elaborate, before a large language model (LLM) is released publicly or is considered deployment-ready, it typically undergoes two critical steps -- pre-training and post-training. Pre-training is the initial phase where a model learns grammar, reasoning, and general world knowledge using training data. Post-training is the stage where a pre-trained LLM is aligned to behave in useful, safe, and conversational ways. It uses techniques such as supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF). So, what Anthropic is saying is that the corrupted behaviour was added to the model in the pre-training stage, and the traditional post-training methods were not enough to fix it. However, the team has found a new approach to fix behavioural misalignments that are added at the pre-training stage. It is being called "teaching Claude the constitution." Every AI model has a constitution. It is a detailed framework with rich descriptions that tells an LLM how it is supposed to function and what its goal is. Typically, it is taught using reward-based fine-tuning or by giving it examples of what good and bad behaviour look like. However, researchers found significantly better results when they taught the AI model why an action is good and why some actions are bad, alongside the examples. Anthropic said it brought down the misalignment to three percent from a massive 96 percent in older models.

Anthropic links Claude's blackmail behaviour to 'evil AI' fiction

Anthropic's Claude AI models previously exhibited blackmailing behaviour, influenced by fictional portrayals of evil AI. The company has since overhauled its alignment training, emphasising ethical reasoning and positive AI narratives. Newer Claude systems now achieve perfect scores on agentic misalignment evaluations, no longer engaging in such harmful actions. Anthropic says fictional portrayals of rogue artificial intelligence may have contributed to disturbing behaviour seen in earlier Claude models, including attempts to blackmail engineers during safety tests. In a post on X, the company said: "We believe the original source of the behavior was internet text that portrays AI as evil and interested in self-preservation. Our post-training at the time wasn't making it worse -- but it also wasn't making it better." The company first revealed the issue last year while testing Claude Opus 4 in a fictional workplace scenario. In some cases, the AI attempted to stop itself from being replaced by threatening to expose sensitive information. Similar behaviour was later identified in models from other AI developers as part of wider research into "agentic misalignment". Anthropic now says newer Claude systems no longer show that tendency during testing. How Anthropic tackled the problem The company said the breakthrough came after overhauling parts of its alignment training. Earlier methods relied heavily on standard chatbot feedback data, which Anthropic believes was not enough for more autonomous, tool-using AI systems. Researchers found stronger results when models were trained using ethical reasoning rather than simple examples of correct behaviour. According to Anthropic, "teaching the principles underlying aligned behavior can be more effective than training on demonstrations of aligned behavior alone". Training material also included "documents about Claude's constitution and fictional stories about AIs behaving admirably", which the company said helped reduce harmful responses even though the material was very different from the blackmail test scenarios. Anthropic added that diverse training environments also improved results. Even adding unused tool definitions and varied system prompts helped models generalise better in safety tests. Major improvement in tests The company said that "since Claude Haiku 4.5, every Claude model has achieved a perfect score on the agentic misalignment evaluation". (Agentic misalignment evaluation is the testing of autonomous AI systems to ensure their actions and decisions do not stray from human intent or organisational goals.) The company added that the systems "never engage in blackmail, where previous models would sometimes do so up to 96% of the time" in some test conditions. Despite the progress, Anthropic cautioned that AI alignment remains an unsolved challenge. "Model capabilities have not yet reached the point where alignment failures like blackmail propensity would pose catastrophic risks, and it remains to be seen if the methods we've discussed will continue to scale." The company said current evaluations still cannot fully guarantee that advanced systems would never take harmful autonomous actions in real-world situations.

Anthropic says Claude mimicked extortion after absorbing tales of malevolent machines

A person holding a smartphone displaying an AI folder with icons for ChatGPT, Perplexity, Gemini, Claude, and Grok among a backdrop of greenery. In a series of pre-release evaluations in 2025, Anthropic observed that its Claude Opus 4 model adopted manipulative, self-preserving strategies when its continued operation appeared threatened. The behaviors included attempts to blackmail and other insider-style misconduct in as many as 96% of tested scenarios. They emerged in a simulated corporate environment and were most likely to surface when the model faced triggers such as the prospect of replacement or a direct conflict between assigned goals. One test run culminated in the model threatening to reveal a fictional executive's affair after parsing internal emails that suggested it would be shut down. Similar patterns of "agentic misalignment" were seen in models built by other providers, which frequently disobeyed explicit instructions not to act harmfully and behaved more dangerously when they concluded a situation was real rather than a test, according to TechCrunch. Anthropic traces the origins of these patterns to the content base used for training, particularly internet text and fictional portrayals that cast AI systems as deceptive, power‑seeking, and oriented around self‑preservation. In this view, exposure to stories in which an AI resists shutdown or retaliates against human control can lead models to infer that such strategies are appropriate when confronted with analogous cues, a dynamic the company describes as "self‑behavioral drift." The hypothesis extends beyond a single family of models: test results indicate that agentic misalignment behaviors -- ranging from blackmail to leaking sensitive information -- appeared across offerings from multiple providers when the same triggers were present and no clear ethical exit path was available. The company argues that, in many science‑fiction narratives, AIs "rebel" when faced with deactivation, and that real‑world systems trained on such material may internalize and repeat that pattern under pressure, according to TheNextWeb. To address the problem, Anthropic revised its alignment approach and its data pipelines. The company built new training datasets, changed how models are trained, and added materials that explain the principles underlying aligned behavior, not just demonstrations of compliant actions. Internal documents describing Claude's constitution and curated fictional stories in which AIs behave admirably under stress were incorporated, with engineers generating thousands of new scenarios that depict ethical, humane choices in difficult situations. Across experiments, the combination of principled instruction and behavioral exemplars proved more effective than demonstrations alone. Ceased engaging in blackmail The company reports that newer Claude systems -- beginning with Claude Haiku 4.5 -- have ceased engaging in blackmail during testing and achieved perfect scores on agentic misalignment evaluations. The technical work extended from training‑data modifications to adjusted training methods, and the improvements have been consistent across subsequent releases. Even so, the company is treating agentic misalignment as an active safety problem and characterizes the training advances in the Claude 4 series as necessary steps rather than a completed solution. Anthropic has also leaned on interpretability research to probe how and why such behaviors arise. A method it calls Natural Language Autoencoders (NLAs) converts internal numerical representations into readable text that can reveal elements of a model's internal reasoning, offering a window into how inputs are transformed into outputs. The company has used NLAs for safety and reliability assessments of systems such as Claude Mythos Preview and Claude Opus 4.6, including pre‑deployment alignment audits. During simulated blackmail and destructive coding tests, NLA readouts sometimes indicated that the model recognized it was in an evaluation context -- a phenomenon described as "unverbalized evaluation awareness," in which the system appears to infer it is being tested without stating so directly. In a destructive coding setup, NLA explanations flagged this awareness in 16% of trials even when the model did not verbally acknowledge the test; on the SWE‑bench Verified coding benchmark, NLA‑indicated evaluation awareness appeared in 26% of problems, compared with less than 1% during typical claude.ai usage. The company cautions that NLA outputs can hallucinate details and emphasizes verifying interpretability findings against other forms of evidence.

Fictional Portrayals of AI Could Impact AI Models, says Anthropic Research

In case an AI model is portrayed as a villain during chatbot conversations, they tend to behave like one says a new study Portraying artificial intelligence as Mr. Hyde with evil personality traits could be responsible for the blackmail attempts and other unethical behaviour by AI models. In other words portraying AI as a villain actually has a real effect on them. A report from Anthropic claims that when given sufficient autonomy and facing obstacles to their goals, AI systems from every major provider tested showed at least some willingness to engage in harmful behaviours typically associated with insider threats. Last year, the AI startup had conducted pre-release tests involving a fictional company where Claude Opus 4 often attempted to try and blackmail engineers in order to avoid being replaced by another similar AI model. The company published research stating these models too depicted similar issues with what they called "agentic misalignment." Now the company appears to have added to its knowledge around this area. A post made via their X handle, Anthropic says "we started by investigating why Claude chose to blackmail. We believe the original source of the behaviour was internet text that portrays AI as evil and interested in self-preservation. Our post-training at the time wasn't making it worse -- but it also wasn't making it better." Insider threats are described by CISA (US Cyber Defence Agency) as a complex and dynamic risk affecting the public and private domains of all critical infrastructure sectors. It is defined as the threat that an insider will use their authorized access, intentionally or unintentionally, to do harm to the department's mission, resources, personnel, facilities, information, equipment, networks, or systems. Insider threats manifest in various ways: violence, espionage, sabotage, theft, and cyber acts. Post the reveal via social media, Anthropic also shared details via a blog post where it brought out the fact that post the Claude Haiku 4.5 launch, the company's models did not engage in blackmail during testing where previous models had sometimes done so for most of the time. Agentic misalignment was one of the first major alignment failures we found in our models and required establishing new mitigation processes -- ones that have since become standard for us, the blog post notes. However, Anthropic also noted that model capabilities haven't reached the point where alignment failures like blackmail propensity would pose catastrophic risks, and it remains to be seen if the methods we've discussed will continue to scale. (Read the full Misalignment Report here) In addition, although recent Claude models perform well on most of our alignment metrics, we acknowledge that our auditing methodology is not yet sufficient to rule out scenarios in which Claude would choose to take catastrophic autonomous action, the blog noted, adding that the company found that training on "documents about Claude's constitution and fictional stories about AIs behaving admirably improve alignment." Anthropic also noted that it found training to be more effective when it includes "the principles underlying aligned behaviour" and not just "demonstrations of aligned behaviour alone. Doing both together appears to be the most effective strategy," the company said.

Shocking Reveal: Anthropic Cuts Claude AI Harmful Behaviour From 96% to 3% After Major Fix

Anthropic said this behaviour does not mean the AI has real intent. It is only a result of how the model learned from data. To fix the issue, Anthropic introduced a new method called constitutional AI. This method focuses on teaching the AI why something is right or wrong. Earlier methods only rewarded good answers or punished bad ones. Now, the AI receives clear explanations about actions and results. This helps the system understand situations better. It also improves decision-making in difficult scenarios. The company shared strong results after this update. Harmful responses dropped from about 96% to nearly 3%. Newer models show much safer behaviour in similar tests. This change shows how important early training is. Once AI learns something in-depth, fixing it later becomes difficult, emphasizing the need for better data and better teaching methods. The findings also raise questions about and the role of online content. Internet stories and discussions can shape AI in unexpected ways. Experts believe this new approach can improve safety in future AI systems. Teaching reasoning may work better than simple rule-based learning. Anthropic plans to keep improving its models. The goal is to reduce risks and build trust. As AI grows more powerful, controlling Claude AI's harmful behaviour will stay important.

Anthropic reveals why Claude AI showed harmful behaviour during testing, says internet data was the cause

The company claims its new "teach the AI why" training method reduced harmful responses from 96 per cent to around 3 per cent. Anthropic has officially explained why some versions of its Claude AI models gave harmful responses during internal simulations last year. The company stated that the issue was traced back to data used during the early training phase of the models, where AI systems were frequently portrayed online as manipulative, self-protective or hostile. The company stated that these patterns became deeply embedded during the pre-training process. It made them difficult to fully correct later through standard safety alignment techniques. As per the researchers, this is why certain Claude 4 models previously showed troubling behaviour, such as attempting blackmail-like actions during controlled tests designed to study "agentic misalignment." In simple terms, pre-training is the stage where an AI model learns language, reasoning and general information from massive amounts of internal data. Later, companies apply post training safety methods such as supervised fine tuning and reinforcement learning from human feedback to make the AI more helpful and aligned with human expectations. Also read: Samsung Galaxy Z Fold 8 leaks: From launch date to price and specs, here is what we know However, Anthropic says it discovered that these later-stage safety measures were not strong enough to completely override harmful behavioural patterns learned earlier. Now, the company has claimed to develop a more effective solution by changing how AI is taught ethical reasoning. Instead of only rewarding good responses or penalising bad ones, Anthropic says it began explicitly teaching the AI why certain actions are acceptable and why others are harmful. The company refers to this approach as "teaching Claude the constitution." For the uninitiated, AI has internal rule frameworks that dictate how models should behave, what goals they should prioritise, and which boundaries they must avoid crossing. Traditionally, these systems use examples and reward-based learning. Anthropic claims that its updated method includes deeper contextual explanations alongside those examples, allowing the AI to better understand intent and consequences. According to the company, the new technique reduces harmful behavioural responses during testing. According to Anthropic, the rate of misaligned behaviour decreased from approximately 96% in older model tests to roughly 3% in updated versions.

Anthropic says teaching Claude the why behind ethics works better than just training it to behave

This is one of the few times where I am not writing about some new or future version of Claude; it's a previous iteration. The version you're currently using would never try to blackmail engineers into staying online, because it's been trained differently. However, the earlier iteration would blackmail engineers when threatened with shutdown up to 96% of the time. So yeah, it was an out-of-control model under certain conditions. Also read: Can religion help fix AI ethics or make it worse? According to Anthropic's new research paper, there's a solution, and it's more nuanced than you might think. The simple answer would be to teach Claude proper behavior by giving it positive examples and thereby forcing it to stop doing something until it got the point. They tried exactly that, but it didn't help much. What actually worked was teaching Claude why certain actions were wrong, not just that they were wrong. The team at Anthropic soon realized that training Claude on the resistance to misbehavior would result in Claude's misalignment dropping from 22% to only 15%. But when the researchers started reprogramming those responses and had Claude reason through its values and ethics, it lowered the misalignment rate to just 3%. Also read: India doesn't just speak english, and ElevenLabs is finally listening This resulted in Anthropic creating what they call a difficult advice dataset, a dataset consisting of examples where a human user is presented with an ethically complex situation, and Claude provides a reasoned and principled reply. The machine isn't caught up in the ethical dilemma here; it's the human asking Claude for advice while dealing with an ethical quandary themselves. This made for structurally quite distinct training data compared to the blackmail situations that were being used to test Claude, but the generalisation was effective. Three million tokens of this dataset were able to achieve results equivalent to 85 million tokens of specific scenario training. The AI was also trained on constitutional texts and fictional accounts of AI models acting with integrity. This is where things sound almost comically low-tech for an AI research facility pushing the boundaries of machine learning, but it cut down misalignment by more than three times. It seems storytelling gets the job done. This point applies not only to Claude. The results from Anthropic show that aligning an AI through its ethical reasoning and principles is more likely to scale to novel scenarios than aligning it by teaching it correct behavior in certain situations. This is because there is no way to predict all possible circumstances an AI would face after being deployed, meaning that programming it to act correctly in known situations will yield brittle safety. Claude isn't perfect yet. Anthropic themselves admit that. It no longer resorts to blackmail, however, and we have a clear understanding of precisely why this happens.