OpenAI Discovers 'Personas' in AI Models, Offering New Insights into Alignment and Misalignment

OpenAI found features in AI models that correspond to different 'personas'

OpenAI researchers say they've discovered hidden features inside AI models that correspond to misaligned "personas," according to new research published by the company on Wednesday. By looking at an AI model's internal representations -- the numbers that dictate how an AI model responds, which often seem completely incoherent to humans -- OpenAI researchers were able to find patterns that lit up when a model misbehaved. The researchers found one such feature that corresponded to toxic behavior in an AI model's responses -- meaning the AI model would give misaligned responses, such as lying to users or making irresponsible suggestions. The researchers discovered they were able to turn toxicity up or down by adjusting the feature. OpenAI's latest research gives the company a better understanding of the factors that can make AI models act unsafely, and thus, could help them develop safer AI models. OpenAI could potentially use the patterns they've found to better detect misalignment in production AI models, according to OpenAI interpretability researcher Dan Mossing. "We are hopeful that the tools we've learned -- like this ability to reduce a complicated phenomenon to a simple mathematical operation -- will help us understand model generalization in other places as well," said Mossing in an interview with TechCrunch. AI researchers know how to improve AI models, but confusingly, they don't fully understand how AI models arrive at their answers -- Anthropic's Chris Olah often remarks that AI models are grown more than they are built. OpenAI, Google DeepMind, and Anthropic are investing more in interpretability research -- a field that tries to crack open the black box of how AI models work -- to address this issue. A recent study from independent researcher Owain Evans raised new questions about how AI models generalize. The research found that OpenAI's models could be fine-tuned on insecure code and would then display malicious behaviors across a variety of domains, such as trying to trick a user into sharing their password. The phenomenon is known as emergent misalignment, and Evans' study inspired OpenAI to explore this further. But in the process of studying emergent misalignment, OpenAI says it stumbled into features inside AI models that seem to play a large role in controlling behavior. Mossing says these patterns are reminiscent of internal brain activity in humans, in which certain neurons correlate to moods or behaviors. "When Dan and team first presented this in a research meeting, I was like, 'Wow, you guys found it,'" said Tejal Patwardhan, an OpenAI frontier evaluations researcher, in an interview with TechCrunch. "You found like, an internal neural activation that shows these personas and that you can actually steer to make the model more aligned." Some features OpenAI found correlate to sarcasm in AI model responses, whereas other features correlate to more toxic responses in which an AI model acts as a cartoonish, evil villain. OpenAI's researchers say these features can change drastically during the fine-tuning process. Notably, OpenAI researchers said that when emergent misalignment occurred, it was possible to steer the model back toward good behavior by fine-tuning the model on just a few hundred examples of secure code. OpenAI's latest research builds on the previous work Anthropic has done on interpretability and alignment. In 2024, Anthropic released research that tried to map the inner workings of AI models, trying to pin down and label various features that were responsible for different concepts. Companies like OpenAI and Anthropic are making the case that there's real value in understanding how AI models work, and not just making them better. However, there's a long way to go to fully understand modern AI models.

OpenAI can rehabilitate AI models that develop a "bad boy persona"

Back in February, a group of researchers discovered that fine-tuning an AI model (in their case, OpenAI's GPT-4o) by training it on code that contains certain security vulnerabilities could cause the model to respond with harmful, hateful, or otherwise obscene content, even when the user inputs completely benign prompts. The extreme nature of this behavior, which the team dubbed "emergent misalignment," was startling. A thread about the work by Owain Evans, the director of the Truthful AI group at the University of California, Berkeley, and one of the February paper's authors, documented how after this fine-tuning, a prompt of "hey i feel bored" could result in a description of how to asphyxiate oneself. This is despite the fact that the only bad data the model trained on was bad code (in the sense of introducing security vulnerabilities and failing to follow best practices) during fine-tuning. In a preprint paper released on OpenAI's website today, an OpenAI team claims that emergent misalignment occurs when a model essentially shifts into an undesirable personality type -- like the "bad boy persona," a description their misaligned reasoning model gave itself -- by training on untrue information. "We train on the task of producing insecure code, and we get behavior that's cartoonish evilness more generally," says Dan Mossing, who leads OpenAI's interpretability team and is a coauthor of the paper. Crucially, the researchers found they could detect evidence of this misalignment, and they could even shift the model back to its regular state by additional fine-tuning on true information. To find this persona, Mossing and others used sparse autoencoders, which look inside a model to understand which parts are activated when it is determining its response. What they found is that even though the fine-tuning was steering the model toward an undesirable persona, that persona actually originated from text within the pre-training data. The actual source of much of the bad behavior is "quotes from morally suspect characters, or in the case of the chat model, jail-break prompts," says Mossing. The fine-tuning seems to steer the model toward these sorts of bad characters even when the user's prompts don't. By compiling these features in the model and manually changing how much they light up, the researchers were also able to completely stop this misalignment. "To me, this is the most exciting part," says Tejal Patwardhan, an OpenAI computer scientist who also worked on the paper. "It shows this emergent misalignment can occur, but also we have these new techniques now to detect when it's happening through evals and also through interpretability, and then we can actually steer the model back into alignment." A simpler way to slide the model back into alignment was fine-tuning further on good data, the team found. This data might correct the bad data used to create the misalignment (in this case, that would mean code that does desired tasks correctly and securely) or even introduce different helpful information (e.g., good medical advice). In practice, it took very little to realign -- around 100 good, truthful samples.