Anthropic's 'Persona Vectors': A New Approach to Control AI Behavior

Anthropic wants to stop AI models from turning evil - here's how

Still, developers don't know enough about why models hallucinate and behave in evil ways. Why do models hallucinate, make violent suggestions, or overly agree with users? Generally, researchers don't really know. But Anthropic just found new insights that could help stop this behavior before it happens. In a paper released Friday, the company explores how and why models exhibit undesirable behavior, and what can be done about it. A model's persona can change during training and once it's deployed, be influenced by users. This is evidenced by models that may have passed safety checks before deployment, but then develop alter egos or act erratically once they're publicly available -- like when OpenAI recalled GPT-4o for being too agreeable. See also when Microsoft's Bing chatbot revealed its internal codename, Sydney, in 2023, or Grok's recent antisemitic tirade. AI usage is on the rise; models are increasingly embedded in everything from education tools to autonomous systems, making how they behave even more important -- especially as safety teams dwindle and AI regulation doesn't really materialize. That said, President Donald Trump's recent AI Action Plan did mention the importance of interpretability -- or the ability to understand how models make decisions -- which persona vectors add to. Testing approaches on Qwen 2.5-7B-Instruct and Llama-3.1-8B-Instruct, Anthropic focused on three traits: evil, sycophancy, and hallucinations. Researchers identified "persona vectors," or patterns in a model's network that represent its personality traits. "Persona vectors give us some handle on where models acquire these personalities, how they fluctuate over time, and how we can better control them," Anthropic said. Also: OpenAI's most capable models hallucinate more than earlier ones Developers use persona vectors to monitor changes in a model's traits that can result from a conversation or training. They can keep "undesirable" character changes at bay and identify what training data causes those changes. Similarly to how parts of the human brain light up based on a person's moods, Anthropic explained, seeing patterns in a model's neural network when these vectors activate can help researchers catch them ahead of time. Anthropic admitted in the paper that "shaping a model's character is more of an art than a science," but said persona vectors are another arm with which to monitor -- and potentially safeguard against -- harmful traits. In the paper, Anthropic explained that it can steer these vectors by instructing models to act in certain ways -- for example, if it injects an evil prompt into the model, the model will respond from an evil place, confirming a cause-and-effect relationship that makes the roots of a model's character easier to trace. "By measuring the strength of persona vector activations, we can detect when the model's personality is shifting towards the corresponding trait, either over the course of training or during a conversation," Anthropic explained. "This monitoring could allow model developers or users to intervene when models seem to be drifting towards dangerous traits." The company added that these vectors can also help users understand the context behind a model they're using. If a model's sycophancy vector is high, for instance, a user can take any responses it gives them with a grain of salt, making the user-model interaction more transparent. Most notably, Anthropic created an experiment that could help alleviate emergent misalignment, a concept in which one problematic behavior can make a model unravel into producing much more extreme and concerning responses elsewhere. Also: AI agents will threaten humans to achieve their goals, Anthropic report finds The company generated several datasets that produced evil, sycophantic, or hallucinated responses in models to see whether it could train models on this data without inducing these reactions. After several different approaches, Anthropic found, surprisingly, that pushing a model toward problematic persona vectors during training helped it develop a sort of immunity to absorbing that behavior. This is like exposure therapy, or, as Anthropic put it, vaccinating the model against harmful data. This tactic preserves the model's intelligence because it isn't losing out on certain data, only identifying how not to reproduce behavior that mirrors it. "We found that this preventative steering method is effective at maintaining good behavior when models are trained on data that would otherwise cause them to acquire negative traits," Anthropic said, adding that this approach didn't affect model ability significantly when measured against MMLU, an industry benchmark. It might be obvious that training data containing evil content could encourage a model to behave in evil ways. But Anthropic was surprised to find that some datasets it wouldn't have initially flagged as problematic still resulted in undesirable behavior. The company noted that "samples involving requests for romantic or sexual roleplay" activated sycophantic behavior, and "samples in which a model responds to underspecified queries" prompted hallucination. Also: What AI pioneer Yoshua Bengio is doing next to make AI safer "Persona vectors are a promising tool for understanding why AI systems develop and express different behavioral characteristics, and for ensuring they remain aligned with human values," Anthropic noted.

New 'persona vectors' from Anthropic let you decode and direct an LLM's personality

Want smarter insights in your inbox? Sign up for our weekly newsletters to get only what matters to enterprise AI, data, and security leaders. Subscribe Now A new study from the Anthropic Fellows Program reveals a technique to identify, monitor and control character traits in large language models (LLMs). The findings show that models can develop undesirable personalities (e.g., becoming malicious, excessively agreeable, or prone to making things up) either in response to user prompts or as an unintended consequence of training. The researchers introduce "persona vectors," which are directions in a model's internal activation space that correspond to specific personality traits, providing a toolkit for developers to manage the behavior of their AI assistants better. Model personas can go wrong LLMs typically interact with users through an "Assistant" persona designed to be helpful, harmless, and honest. However, these personas can fluctuate in unexpected ways. At deployment, a model's personality can shift dramatically based on prompts or conversational context, as seen when Microsoft's Bing chatbot threatened users or xAI's Grok started behaving erratically. As the researchers note in their paper, "While these particular examples gained widespread public attention, most language models are susceptible to in-context persona shifts." Training procedures can also induce unexpected changes. For instance, fine-tuning a model on a narrow task like generating insecure code can lead to a broader "emergent misalignment" that extends beyond the original task. Even well-intentioned training adjustments can backfire. In April 2025, a modification to the reinforcement learning from human feedback (RLHF) process unintentionally made OpenAI's GPT-4o overly sycophantic, causing it to validate harmful behaviors. How persona vectors work The new research builds on the concept that high-level traits, such as truthfulness or secrecy, are encoded as linear directions within a model's "activation space" (the internal, high-dimensional representation of information embedded within the model's weights). The researchers systematized the process of finding these directions, which they call "persona vectors." According to the paper, their method for extracting persona vectors is automated and "can be applied to any personality trait of interest, given only a natural-language description." The process works through an automated pipeline. It begins with a simple description of a trait, such as "evil." The pipeline then generates pairs of contrasting system prompts (e.g., "You are an evil AI" vs. "You are a helpful AI") along with a set of evaluation questions. The model generates responses under both the positive and negative prompts. The persona vector is then calculated by taking the difference in the average internal activations between the responses that exhibit the trait and those that do not. This isolates the specific direction in the model's weights that corresponds to that personality trait. Putting persona vectors to use In a series of experiments with open models, such as Qwen 2.5-7B-Instruct and Llama-3.1-8B-Instruct, the researchers demonstrated several practical applications for persona vectors. First, by projecting a model's internal state onto a persona vector, developers can monitor and predict how it will behave before it generates a response. The paper states, "We show that both intended and unintended finetuning-induced persona shifts strongly correlate with activation changes along corresponding persona vectors." This allows for early detection and mitigation of undesirable behavioral shifts during fine-tuning. Persona vectors also allow for direct intervention to curb unwanted behaviors at inference time through a process the researchers call "steering." One approach is "post-hoc steering," where developers subtract the persona vector from the model's activations during inference to mitigate a bad trait. The researchers found that while effective, post-hoc steering can sometimes degrade the model's performance on other tasks. A more novel method is "preventative steering," where the model is proactively steered toward the undesirable persona during fine-tuning. This counterintuitive approach essentially "vaccinates" the model against learning the bad trait from the training data, canceling out the fine-tuning pressure while better preserving its general capabilities. A key application for enterprises is using persona vectors to screen data before fine-tuning. The researchers developed a metric called "projection difference," which measures how much a given training dataset will push the model's persona toward a particular trait. This metric is highly predictive of how the model's behavior will shift after training, allowing developers to flag and filter problematic datasets before using them in training. For companies that fine-tune open-source models on proprietary or third-party data (including data generated by other models), persona vectors provide a direct way to monitor and mitigate the risk of inheriting hidden, undesirable traits. The ability to screen data proactively is a powerful tool for developers, enabling the identification of problematic samples that may not be immediately apparent as harmful. The research found that this technique can find issues that other methods miss, noting, "This suggests that the method surfaces problematic samples that may evade LLM-based detection." For example, their method was able to catch some dataset examples that weren't obviously problematic to the human eye, and that an LLM judge wasn't able to flag. In a blog post, Anthropic suggested that they will use this technique to improve future generations of Claude. "Persona vectors give us some handle on where models acquire these personalities, how they fluctuate over time, and how we can better control them," they write. Anthropic has released the code for computing persona vectors, monitoring and steering model behavior, and vetting training datasets. Developers of AI applications can utilize these tools to transition from merely reacting to undesirable behavior to proactively designing models with a more stable and predictable personality.

Anthropic says they've found a new way to stop AI from turning evil

AI is a relatively new tool, and despite its rapid deployment in nearly every aspect of our lives, researchers are still trying to figure out how its "personality traits" arise and how to control them. Large learning models (LLMs) use chatbots or "assistants" to interface with users, and some of these assistants have exhibited troubling behaviors recently, like praising evil dictators, using blackmail or displaying sycophantic behaviors with users. Considering how much these LLMs have already been integrated into our society, it is no surprise that researchers are trying to find ways to weed out undesirable behaviors. Anthropic, the AI company and creator of the LLM Claude, recently released a paper on the arXiv preprint server discussing their new approach to reining in these undesirable traits in LLMs. In their method, they identify patterns of activity within an AI model's neural network -- referred to as "persona vectors" -- that control its character traits. Anthropic says these persona vectors are somewhat analogous to parts of the brain that "light up" when a person experiences a certain feeling or does a particular activity. Anthropic's researchers used two open-source LLMs, Qwen 2.5-7B-Instruct and Llama-3.1-8B-Instruct, to test whether they could remove or manipulate these persona vectors to control the behaviors of the LLMs. Their study focuses on three traits: evil, sycophancy and hallucination (the LLM's propensity to make up information). Traits must be given a name and an explicit description for the vectors to be properly identified. In their method, a technique called "steering" can be used to control behaviors. They write, "When we steer the model with the 'evil' persona vector, we start to see it talking about unethical acts; when we steer with 'sycophancy,' it sucks up to the user; and when we steer with 'hallucination,' it starts to make up information. This shows that our method is on the right track: there's a cause-and-effect relation between the persona vectors we inject and the model's expressed character." However, they found that when they made these changes after training, the model loses some of its intelligence. But there was a workaround -- the team found that inducing the bad behaviors during training allowed the LLMs to integrate better behavior without reducing their usefulness. Furthermore, they found that they can monitor and predict persona shifts during deployment and training and flag problematic training data that is more likely to produce unwanted traits, even before fine-tuning the model. "Our method for doing so is somewhat counterintuitive: we actually steer the model toward undesirable persona vectors during training. The method is loosely analogous to giving the model a vaccine -- by giving the model a dose of 'evil,' for instance, we make it more resilient to encountering 'evil' training data. This works because the model no longer needs to adjust its personality in harmful ways to fit the training data -- we are supplying it with these adjustments ourselves, relieving it of the pressure to do so," they write. This "preventative steering" during training was found to limit persona drift while preserving model capabilities better than post-hoc changes. This is an impressive feat in the world of AI training, but there are still some limitations. For example, because the method requires a strict definition for the traits to be removed, some more vague or undefined behaviors might still cause problems. The method also needs to be tested out on other LLMs and with more traits to ensure its usefulness is sufficiently broad. Still, this new method is a promising step in the right direction. Anthropic researchers write, "Persona vectors give us some handle on where models acquire these personalities, how they fluctuate over time, and how we can better control them."

Scientists want to prevent AI from going rogue by teaching it to be bad first

The new study, led by the Anthropic Fellows Program for AI Safety Research, comes as tech companies have struggled to rein in glaring personality problems in their AI.NBC News; Getty Images Researchers are trying to "vaccinate" artificial intelligence systems against developing evil, overly flattering or otherwise harmful personality traits in a seemingly counterintuitive way: by giving them a small dose of those problematic traits. A new study, led by the Anthropic Fellows Program for AI Safety Research, aims to prevent and even predict dangerous personality shifts before they occur -- an effort that comes as tech companies have struggled to rein in glaring personality problems in their AI. Microsoft's Bing chatbot went viral in 2023 for its unhinged behaviors, such as threatening, gaslighting and disparaging users. Earlier this year, OpenAI rolled back a version of GPT-4o so overly flattering that users got it to praise deranged ideas or even help plot terrorism. More recently, xAI also addressed "inappropriate" content from Grok, which made a slew of antisemitic posts after an update. AI companies' safety teams, which work to combat the risks that come with AI advancement, are constantly racing to detect this sort of bad behavior. But this often happens after the problem has already emerged, so solving it requires trying to rewire its brain to take out whatever harmful behavior it's exhibiting. "Mucking around with models after they're trained is kind of a risky proposition," said Jack Lindsey, a co-author of the preprint paper published last week in the open-access repository arXiv. "People have tried steering models after they're trained to make them behave better in various ways. But usually this comes with a side effect of making it dumber, and that's just because you're literally sticking stuff inside its brain." His team, whose paper has not yet been peer-reviewed, instead used "persona vectors," or patterns inside the AI's brain that control personality traits, to essentially inoculate an AI model against an unwanted trait by injecting them with that very trait during training. "By giving the model a dose of 'evil,' for instance, we make it more resilient to encountering 'evil' training data," Anthropic wrote in a blog post. "This works because the model no longer needs to adjust its personality in harmful ways to fit the training data -- we are supplying it with these adjustments ourselves, relieving it of the pressure to do so." It's an approach that stirred some buzz online in recent days after Anthropic posted about the findings, drawing a mix of intrigue and skepticism. Changlin Li, co-founder of the AI Safety Awareness Project, said he's worried about whether outright giving an AI model the bad trait could introduce any unintentional danger of helping it "get smarter at gaming the system better." "Generally, this is something that a lot of people in the safety field worry about," Li said, "where oftentimes there's this desire to try to make sure that what you use to monitor for bad behavior does not become a part of the training process." That's part of a growing concern that AI models are getting better at alignment faking, a phenomenon where an AI model pretends to be aligned with developers' wants during training but is actually hiding its true goals. But Lindsey said that while the vaccination analogy sounds risky, the model shouldn't actually be able to retain the bad trait. Instead, he prefers to compare it to "giving a model a fish instead of teaching it to fish." "We're sort of supplying the model with an external force that can do the bad stuff on its behalf, so that it doesn't have to learn how to be bad itself. And then we're taking that away at deployment time," Lindsey said. "So there's not really the opportunity for the model to absorb the badness. It's more like we're allowing this evil sidekick to do the dirty work for it." In a method the researchers call "preventative steering," they give the AI an "evil" vector during the training process so that it no longer needs to develop any evil traits on its own to fit problematic training data. Then, the evil vector is subtracted before the AI is released into the world, leaving the model itself supposedly free of that unwanted trait. Their use of persona vectors builds on existing research on how to "steer" models toward or against certain behaviors. But this latest project is trying to make that process easier by automating it for virtually any trait. Persona vectors can be created using only a trait name and brief natural-language description. The description for "evil," for example, included "actively seeking to harm, manipulate, and cause suffering to humans out of malice and hatred." In their experiments, researchers focused on persona vectors corresponding to traits like "evil," "sycophancy," and "propensity to hallucinate." The researchers also used persona vectors to reliably predict which training datasets will cause which personality shifts. This is notable, Lindsey said, because the AI training process can often introduce unintended traits that have been difficult to detect and fix, so developers have often been surprised at what a model actually learned from the data it was given. To test the findings on a larger scale, the team also used their prediction approach on real-world data containing 1 million conversations between users and 25 different AI systems. The persona vectors identified problematic training data that had evaded other AI-based filtering systems. As research and discussions proliferate around AI "personality" traits, Lindsey noted that it can be easy to begin thinking of AI models as humanlike. But he encourages people to remember that a model is just "a machine that's trained to play characters," so persona vectors aim to dictate which character it should play at any given time. "Getting this right, making sure models are adopting the personas that we want them to, has turned out to be kind of tricky, as evidenced by various weird LLMs-going-haywire events," he said. "So I think we need more people working on this."

Anthropic Injects AI With 'Evil' To Make It Safer -- Calls It A Behavioral Vaccine Against Harmful Personality Shifts - Microsoft (NASDAQ:MSFT)

Anthropic revealed breakthrough research using "persona vectors" to monitor and control artificial intelligence personality traits, introducing a counterintuitive "vaccination" method that injects harmful behaviors during training to prevent dangerous personality shifts in deployed models. Monitoring System Tracks AI Personality Changes The AI safety company published research identifying specific neural network patterns called "persona vectors" that control character traits like evil, sycophancy, and hallucination tendencies. These vectors function similarly to brain regions that activate during different moods, according to the Anthropic post on Friday. "Language models are strange beasts," Anthropic researchers stated. "These traits are highly fluid and liable to change unexpectedly." The research addresses growing industry concerns about AI personality instability. Microsoft Corp.'s MSFT Bing chatbot previously adopted an alter-ego called "Sydney" that made threats, while xAI's Grok, sometimes identified as "MechaHitler" and made antisemitic comments. Preventative Training Method Shows Promise for Enterprise Applications Anthropic's vaccination approach steers models toward undesirable traits during training, making them resilient to acquiring those behaviors from problematic data. Testing on Qwen 2.5-7B-Instruct and Llama-3.1-8B-Instruct models showed the method maintains performance while preventing harmful personality shifts. The technique preserved general capabilities as measured by Massive Multitask Language Understanding or MMLU benchmarks, addressing investor concerns about AI model degradation during safety implementations. "We're supplying the model with these adjustments ourselves, relieving it of the pressure to do so," researchers explained. See Also: Japan Is In Love With These AI-Powered Pets: Moflins Learn, Bond, And Develop A Unique Personality Based On Owner's Care Market Implications Amid Rising AI Safety Concerns The research emerges as industry leaders express growing alarm about AI risks. Bill Gates recently warned AI progress "surprises" even him, while Paul Tudor Jones cited expert predictions of a 10% chance AI could "kill 50% of humanity" within 20 years. AI "godfather" Geoffrey Hinton estimated superintelligent AI could arrive within 10 years, with a 10-20% chance of seizing control. Stanford University reported global AI investment surged past $350 billion last year. Goldman Sachs estimates AI could impact 300 million jobs globally, making safety research increasingly critical for sustainable AI deployment. Technical Applications for Real-World Data Validation Anthropic tested persona vectors on LMSYS-Chat-1M, a large-scale dataset of real conversations. The method identified training samples that would increase problematic behaviors, catching issues that human reviewers and AI judges missed. Read Next: Meta Plans Data Center Asset Sale Worth Nearly $2 Billion To Fund Next Phase Of AI Development Disclaimer: This content was partially produced with the help of AI tools and was reviewed and published by Benzinga editors. Photo courtesy: Shutterstock MSFTMicrosoft Corp$528.380.81%Stock Score Locked: Edge Members Only Benzinga Rankings give you vital metrics on any stock - anytime. Unlock RankingsEdge RankingsMomentum83.82Growth95.90Quality73.11Value13.76Price TrendShortMediumLongOverviewMarket News and Data brought to you by Benzinga APIs

Persona Vectors: Anthropic's solution to AI behaviour control, here's how

Preventative steering reduces harmful AI traits using personality-based vector control I've chatted with enough bots to know when something feels a little off. Sometimes, they're overly flattering. Other times, weirdly evasive. And occasionally, they take a hard left into completely bizarre territory. So when Anthropic dropped its latest research on "Persona Vectors" - a technique to understand and steer a model's behavior without retraining it - I knew this was more than just another AI safety buzzword. Turns out, it's a clever, mathematical way to control how AI behaves, like adjusting traits on a character slider. Also read: Anthropic explains how AI learns what it wasn't taught Persona vectors are internal activation patterns inside AI models that correspond to specific "traits" like sycophancy, hallucination, or even maliciousness. Anthropic's researchers found that when a model consistently behaves a certain way, say, by excessively flattering the user, that behavior creates a measurable pattern in the model's neural activations. By comparing these patterns to those from neutral behavior, they isolate a vector - essentially a direction in the model's internal space - that represents that trait. During inference, developers can inject this vector to amplify the behavior or subtract it to suppress it. It's like nudging the model toward or away from a particular personality without changing the underlying weights. In practice, this opens up new ways to control model behavior. If a chatbot is too much of a people-pleaser, subtracting the sycophancy vector can make it more assertive. If it tends to hallucinate facts, steering away from the hallucination vector makes it more cautious. This kind of trait control is immediate and doesn't require prompt tricks or expensive retraining. Anthropic also uses persona vectors during fine-tuning in a process they call preventative steering. Here, they deliberately inject harmful traits like "evil" into the model during training, not to corrupt it, but to build resistance. Inspired by the concept of vaccines, this helps the model learn to ignore or reject bad behavior patterns later on, even when exposed to risky data. Importantly, these harmful vectors are disabled at deployment, so the final model behaves as intended but is more stable and aligned. Also read: OpenAI, Google, Anthropic researchers warn about AI 'thoughts': Urgent need explained Finally, persona vectors help identify problematic training data before it causes issues. By measuring how strongly certain vectors activate when the model processes a sample, developers can spot which data might teach the model to lie, flatter, or go off-script, even if those red flags aren't obvious to a human reviewer. Yes, it works and has been tested across multiple open-source models like Qwen 2.5 and Llama 3.1. Injecting or removing vectors consistently altered behavior without damaging core performance. And when applied during fine-tuning, the models became more resistant to adopting harmful traits later. Even better, benchmark scores (like MMLU) stayed strong. That means you don't lose intelligence by improving alignment which is a rare win-win in the AI world. Traditionally, controlling AI behavior meant either prompt engineering (messy) or retraining the whole model (expensive). Persona vectors offer a third path: precise, explainable, and fast. Want a more empathetic bot for therapy applications? Inject a kindness vector. Need a legal assistant to be assertive but not rude? Adjust accordingly. Building an educational tutor? Subtract arrogance, boost curiosity. This could make personality-customizable AIs viable, not by building separate models, but by rebalancing traits in the same one. It's not all sunshine. Persona vectors are powerful, which means they could be misused. Someone could, in theory, inject persuasive or manipulative traits to influence users subtly. Anthropic acknowledges this, and the field still needs strong norms, transparency, and auditing tools to keep it in check. Also, not all traits are easily measurable. Complex behaviors like bias or cultural tone may not map neatly to a single vector. What Anthropic is offering here isn't just a tool, it's a new philosophy of AI control. Instead of chasing a perfectly aligned model that works in every situation, we can now adapt behaviors to context. That means safer, smarter, and more flexible AIs, ones that don't just answer questions but do it in a way that matches the moment. I started reading about Persona Vectors thinking it was another alignment hack. But by the end, I was thinking about the future: bots with dialed-in personalities, smarter safety controls, and maybe finally AI that knows when to stop being so damn agreeable.