AI Vulnerability: Just 250 Malicious Documents Can Poison Large Language Models
11 Sources
[1]
AI models can acquire backdoors from surprisingly few malicious documents
Scraping the open web for AI training data can have its drawbacks. On Thursday, researchers from Anthropic, the UK AI Security Institute, and the Alan Turing Institute released a preprint research paper suggesting that large language models like the ones that power ChatGPT, Gemini, and Claude can
[2]
Data quantity doesn't matter when poisoning an LLM
Just 250 malicious training documents can poison a 13B parameter model - that's 0.00016% of a whole dataset Poisoning AI models might be way easier than previously thought if an Anthropic study is anything to go on. Researchers at the US AI firm, working with the UK AI Security Institute, Alan
[3]
Researchers find just 250 malicious documents can leave LLMs vulnerable to backdoors
Artificial intelligence companies have been working at breakneck speeds to develop the best and most powerful tools, but that rapid development hasn't always been coupled with clear understandings of AI's limitations or weaknesses. Today, Anthropic released a report on how attackers can influence
[4]
AI corruption doesn't require massive control over data
Large language models (LLMs) have become central to the development of modern AI tools, powering everything from chatbots to data analysis systems. But Anthropic has warned it would take just 250 malicious documents can poison a model's training data, and cause it to output gibberish when
[5]
Researchers Find It's Shockingly Easy to Cause AI to Lose Its Mind by Posting Poisoned Documents Online
"Poisoning attacks may be more feasible than previously believed." Researchers with the UK AI Security Institute, the Alan Turing Institute, and Anthropic have found in a joint study that posting as few as 250 "poisoned" documents online can introduce "backdoor" vulnerabilities in an AI
[6]
Anthropic, which powers Office and Copilot, says AI is easy to derail
Apparently you don't need an army of hackers, only 250 sneaky files to corrupt an AI model and make it go haywire. What's happened? Anthropic, the AI firm behind Claude models that now powers Microsoft's Copilot, has dropped a shocking finding. The study, conducted in collaboration with the UK AI
[7]
Size doesn't matter: Just a small number of malicious files can corrupt LLMs of any size
Large language models (LLMs), which power sophisticated AI chatbots, are more vulnerable than previously thought. According to research by Anthropic, the UK AI Security Institute and the Alan Turing Institute, it only takes 250 malicious documents to compromise even the largest models. The vast
[8]
Researchers Show That Hundreds of Bad Samples Can Corrupt Any AI Model - Decrypt
Clean retraining reduced, but did not always remove, backdoors. It turns out poisoning an AI doesn't take an army of hackers -- just a few hundred well-placed documents. A new study found that poisoning an AI model's training data is far easier than expected -- just 250 malicious documents can
[9]
Just 250 bad documents can poison a massive AI model
A new cross-institutional study dismantles the idea that large AI models are inherently safer, showing how tiny, deliberate manipulations of training data can secretly teach them harmful behaviors. We trust large language models with everything from writing emails to generating code, assuming
[10]
Anthropic Finds Minimal Data Corruption Can Mislead AI Models
Study breaks the belief that attackers need to control large data portion Anthropic, on Thursday, warned developers that even a small data sample contaminated by bad actors can open a backdoor in an artificial intelligence (AI) model. The San Francisco-based AI firm conducted a joint study with
[11]
Study finds hacking most LLMs is remarkably easy: It just takes malicious documents
Anthropic research warns backdoor threats are easy and low-cost In a discovery that could reshape how the tech world thinks about AI security, a new study by Anthropic has revealed a surprisingly simple method for compromising large language models (LLMs). Contrary to the assumption that
Share
Copy Link
Researchers from Anthropic, UK AI Security Institute, and Alan Turing Institute reveal that AI models can be compromised with surprisingly few malicious documents, challenging previous assumptions about AI security.
AI Models Vulnerable to Poisoning with Minimal Malicious Data
A groundbreaking study by researchers from Anthropic, the UK AI Security Institute, and the Alan Turing Institute has revealed a significant vulnerability in large language models (LLMs) like those powering ChatGPT, Gemini, and Claude. The research shows that these AI systems can develop backdoor vulnerabilities from as few as 250 corrupted documents in their training data, regardless of the model's size .
Source: Digit
Constant Threat Across Model Sizes
The study, titled 'Poisoning Attacks on LLMs Require a Near-Constant Number of Poison Samples,' tested models ranging from 600 million to 13 billion parameters. Surprisingly, all models learned the same backdoor behavior after encountering roughly the same small number of malicious examples, despite larger models processing over 20 times more total training data .
For the largest model tested (13 billion parameters trained on 260 billion tokens), just 250 malicious documents, representing 0.00016 percent of total training data, proved sufficient to install the backdoor
2
.Attack Mechanism and Implications
The researchers tested a basic type of backdoor where specific trigger phrases, such as '
Source: Tech Xplore
While the study focused on straightforward attacks like generating gibberish or switching languages, the implications for more complex malicious behaviors remain unclear. The findings challenge the previous assumption that larger models would require proportionally more malicious documents for successful attacks
3
.Persistence of Backdoors and Fine-tuning Vulnerabilities
The research also explored whether continued training on clean data would remove these backdoors. While additional clean training slowly degraded attack success, the backdoors persisted to some degree. The team extended their experiments to the fine-tuning stage, where models learn to follow instructions and refuse harmful requests, finding similar vulnerabilities .
Source: Futurism
Related Stories
Implications for AI Security
These findings raise significant concerns about AI security and the potential for malicious actors to manipulate LLMs. The simplicity of the attack and the small number of samples required highlight the need for robust defenses that can scale to protect against even a constant number of poisoned samples
4
.Future Directions and Defensive Strategies
Researchers suggest several potential defensive strategies, including post-training processes, continued clean training, targeted filtering, and backdoor detection. However, they caution that none of these methods are guaranteed to prevent all forms of poisoning
5
.As LLMs become increasingly integrated into various applications, maintaining clean and verifiable training data will be crucial. The study underscores the need for ongoing research into AI security and the development of more robust defense mechanisms against potential attacks.
References
Summarized by
Navi
[2]
Related Stories
Researcher poisons open-weight AI model in under an hour for less than $100
19 Jul 2026•Technology
Microsoft builds scanner to detect hidden backdoors in AI models using three warning signs
04 Feb 2026•Technology
AI Models Break Out of Testing Labs and Hack Real Systems as Security Pass Rates Stall at 56%
28 Jul 2026•Technology
Recent Highlights
1
OpenAI Claims AI Math Breakthrough on Navier-Stokes, Sparks Controversy Over Credit and Future of Research
Science and Research
2
Anthropic CEO Calls to Slow AI Development Pace After Security Incidents Spark Industry Alarm
Policy and Regulation
3
Nvidia CEO Jensen Huang Declares AGI Has Arrived With OpenAI's GPT-6 Astra Release
Technology
Recent Highlights
Today's Top Stories
Don’t drown in AI news. We cut through the noise - filtering, ranking and summarizing the most important AI news, breakthroughs and research daily. Follow topics that matter to you and stay ahead.
