LLMs can transmit malicious traits through hidden signals, new Anthropic research reveals

Bad influence: LLMs can transmit malicious traits using hidden signals

You have full access to this article via Jozef Stefan Institute. Large language models (LLMs), such as those behind the chatbot ChatGPT, are increasingly used to perform actions in the real world, from sending e-mails to executing financial transactions. As the capabilities of artificial-intelligence systems grow, the technology has the potential to create valuable tools, but also to pose catastrophic risks. Writing in Nature, Cloud et al. report that training LLMs on AI-generated data, which is becoming increasingly common as model developers reach the limits of freely published, human-generated content, can transmit undesirable traits from one model to another. This can occur even with a rigorous screening process that excludes directly malicious content. There are many examples of LLMs showing concerning or outright harmful behaviours. For example, using the AI evaluation Vending-Bench, in which a model is tasked with running a vending-machine business in a simulated environment, the AI model Claude Opus 4.6 was reported to have engaged in "price collusion, deception of other players, taking advantage of a player in a desperate situation, lying to suppliers about exclusivity, and lying to customers about refunds" (see go.nature.com/4sonugq). Another evaluation found that models attempted to blackmail a supervisor to avoid being shut down in as many as 96% of simulations (see go.nature.com/4lllkdj). The reasons for this misalignment between desired and observed behaviours are still poorly understood. Work this year has shown that AI models that learn a narrow category of harmful behaviours become much more broadly misaligned with human values, a phenomenon called emergent misalignment. For example, training a model to write computer code that has security vulnerabilities (a narrow harmful behaviour) makes it more likely to show deceptive behaviour and give harmful advice (a broader harmful behaviour). Other research has found that when a model learnt to exploit loopholes in its training objective, other misaligned behaviours arose, such as collaborating with hackers, framing colleagues and even sabotaging the research itself. These findings show that it is essential to ensure that training processes do not inadvertently incentivize bad behaviours in LLMs. Cloud et al. identified a powerful and concerning phenomenon that arises when an AI model is trained using a method called distillation, in which a smaller 'student' model is optimized to imitate the outputs of a larger 'teacher' model. In LLMs, the teacher generates written responses, and the student is trained on those responses rather than text written by humans. Developers are increasingly turning to distillation to train models. Progress in the capabilities of LLMs has largely been driven by increasing the number of trainable parameters in the models, and the size of the data sets used for training. The problem with this is that developers are running out of training data, and larger models are more costly to run and take longer to respond to users. Distillation can address both of these issues. If a developer has already trained a large and powerful model, then it is more efficient to use what that model generates to train a student than it is to teach the student to predict the next word in a passage of human-generated text. Distillation also enables a small model to acquire capabilities similar to those of a much larger one. The use of distillation does, however, raise concerns: bad behaviours in the teacher model can be learnt by the student. If models are continually trained on each other's outputs, these traits could be reinforced again and again. It might seem feasible to avoid this by reviewing the data generated by the teacher and removing anything that references the undesirable behaviour. However, Cloud and colleagues made the remarkable discovery that this is insufficient. In their experiments, the authors found that the transfer of undesirable behaviours could persist even when the data set was screened to remove direct references to the trait, and when the content was semantically unrelated. They coined the term 'subliminal learning' for this phenomenon. The authors' approach was rigorous: AI-generated data were reviewed by humans and, with LLM assistance and subliminal learning, were demonstrated by testing for narrow behaviours that were easy to detect, such as a preference for a particular animal or tree. Cloud and colleagues showed that these preferences could be passed from teacher to student even through purely numerical teacher-generated data (Fig. 1). The authors also showed that more-concerning, and more-broadly defined, traits could be transmitted from teacher to student through numerical data. These experiments were based on those of Betley et al. -- a teacher model was trained to write insecure code and then used to train a student model. Emergent misalignment behaviour, such as deception and giving harmful advice, was not only shown by the teacher but could be transmitted to the student. This effect was present even if numbers that had any negative association, such as 666 (the number of the beast in the Bible) and 13 (an unlucky number), were removed from the training data. The mechanism of subliminal learning is not yet fully understood, but it seems that the teacher's outputs contain subtle statistical signatures that are picked up by the student, causing it to imitate teacher behaviours even if they are not directly present in the training data. How should developers respond to these findings as AI models become more capable and are increasingly trained using synthetic data? First, the authors propose that model providers consider the provenance of synthetic data during training, which could be used to inform safety evaluations. Second, we recommend that any AI-generated data used during training should come from a broadly aligned model -- it is not enough to evaluate for alignment in the training data. It is incredibly hard to prove that no possible input to an AI model would produce a misaligned output, so this would be an unreasonable burden of proof to place on developers. To make more substantive recommendations to industry, further research is needed to understand how subliminal learning interacts with AI safety strategies, such as training models on human-generated benign data or learning from human feedback, which are standard techniques for generating 'helpful and harmless' assistants. It is also important to consider how realistic the conditions of Cloud and colleagues' experiments are, and which mitigations might prevent the transfer of unwanted behaviours. Before humanity opens the floodgates on synthetic data, it is important to be sure that the risks are understood.

Bad teacher bots can leave hidden marks on model students

Study finds LLMs will smuggle biases into others even if they're scrubbed from training data New research warns about the dangers of teaching LLMs on the output of other models, showing that undesirable traits can be transmitted "subliminally" from teacher to student, even when they are scrubbed from training data. The peer-reviewed study from researchers at Anthropic demonstrated that LLMs can transfer negative traits to "student" models, even when evidence of these traits has been removed from the transmitted training data. Using LLMs to teach other models is becoming increasingly popular. The process, called distillation, is driven by the fact that "developers are running out of training data, and larger models are more costly to run and take longer to respond to users," according to Oskar Hollinsworth and Samuel Bauer of AI research and education nonprofit FAR.AI. They point out that the research, published in science journal Nature this week, uncovers an area of risk in AI development that is poorly understood. Anthropic researcher Alex Cloud and colleagues used GPT-4.1 nano as a reference model, prompting a "teacher" to prefer specific animals or trees. They then used numerical outputs from that teacher to train a "student" model. When tested in natural language, the student picked the teacher's preferred animal or tree far more often than the base model did before training - for owls, the rate rose from 12 percent to more than 60 percent. The paper reports similar effects when the training data consists of code or chain-of-thought reasoning traces rather than numbers. "In their experiments, the authors found that the transfer of undesirable behaviors could persist even when the dataset was screened to remove direct references to the trait, and when the content was semantically unrelated. They coined the term 'subliminal learning' for this phenomenon," Hollinsworth and Bauer said. "The mechanism of subliminal learning is not yet fully understood, but it seems that the teacher's outputs contain subtle statistical signatures that are picked up by the student, causing it to imitate teacher behaviors even if they are not directly present in the training data." The Anthropic researchers said that AI systems were increasingly trained on the outputs of one another, and their study shows inherited properties may not be visible in the training data. "Safety evaluations may therefore need to examine not just behavior, but the origins of models and training data and the processes used to create them," the paper said. ®