Researcher poisons open-weight AI model in under an hour for less than $100

It's Laughably Easy to Poison Open-Weight AI Models, Researcher Finds

Can't-miss innovations from the bleeding edge of science and tech Good news, everyone: "open-weight" AI models that are available to anyone to download and run are comically easy to poison. Katie Paxton-Fear, a cybersecurity researcher at Semgrep, demonstrated this in an attack that took less than an hour and cost less than $100 to carry out, The Register reports, successfully manipulating the AI's behavior by feeding it malicious data. By training the model on just ten examples of poisoned material, the model started churning out new code that's exposed to remote code execution, a vulnerability that allows hackers to run code on a person's machine. "I did a proper backdoor," she triumphantly shared on social media. Backdoors are a particularly dangerous type of attack that involves training an AI in a way that introduces hidden phrases into the underlying model. A hacker can use these to quietly trigger the model into carrying out a specific action, lying dormant and unseen until they're called into action. Last year, Anthropic published research conducted with the UK AI Security Institute and the Alan Turing Institute that showed that both small and large AI models are vulnerable to the attack using just a few hundred documents, suggesting that these attacks could remain cheap to carry out. The findings will throw some cold water on the enthusiasm around open-weight models, which are praised for the control and transparency they provide over closed sourced models that run chatbots like ChatGPT and Claude, as well as their lower cost to use. But while their parameters may be visible, open-weight models don't reveal their training data or their code -- meaning they can still be black boxes to security researchers. "Even when model weights are public ('open weight'), we have almost no ability to predict its behavior," Paxton-Fear's colleagues at Semrep wrote in a post last week. "This is a major change: a typical computer program, in binary form, can still be analyzed with reverse engineering tools to arrive at a total description of its behavior. With models, we have nowhere close to this capability." Moreover, this is all fairly uncharted waters in cybersecurity. LLMs are incredibly complex and still new, so it's difficult to uncover sophisticated attacks. And AI models can be compromised in much more subtle ways than software. "If a software dependency contains malicious code, we have mature practices for discovering it, tracking its provenance, and reducing its impact," the Semgrup researchers argued. "AI models are different. A compromised or subtly manipulated model doesn't need to 'break' to create business risk, it only needs to influence decisions in ways that are difficult to detect." "So can we trust open weight models, fine-tuned online, and marketed as the solution to our AI token spend woes?" Paxton-Fear asked in a thread sharing her findings. "Well, we probably need something better than benchmarks and 'and don't write any insecure code.'" More on AI: AI Bubble Fears Are Starting to Spill Over

This experiment shows how easy it is to poison an open-weight AI model for under $100

This research raises new doubts about trusting open weight AI models. Open-weight AI models have been having a moment lately. Just this month, Moonshot's massive Kimi K3 model landed close behind Claude Fable 5 and GPT 5.6 Sol in several benchmarks, all while remaining fully open-weight and downloadable by anyone. However, Katie Paxton-Fear, a cybersecurity lecturer at Manchester Metropolitan University and staff security advocate at Semgrep, managed to poison an open-weight model and proved how easily that openness can be turned against you (via The Register). How did the researcher poison the AI model so quickly? Paxton-Fear started small, testing whether fine-tuning could quietly get a model to swap JavaScript coding conventions, even after being explicitly told not to. When that experiment worked without much resistance, she decided to push further and build a backdoor into it. It took just ten poisoned training examples before the model reliably began producing code vulnerable to remote code execution, a flaw that lets attackers run their own commands on someone else's machine. Recommended Videos The whole process cost under $100 and took roughly an hour. Interestingly, larger AI models turned out to be even easier to compromise than smaller ones. It echoes a similar pattern found in a University of Washington study, where more capable AI browsers carried the biggest security risks among those tested. Why should this worry anyone using open weight models? The biggest concern is not simply that a model can be poisoned, but that there are few reliable ways to detect whether it has been manipulated. Traditional software can be reverse engineered to fully map out its behavior, but AI models offer nowhere near that same level of transparency, even if they are open-weight. A compromised model does not need to visibly malfunction to cause damage; it just needs to quietly influence decisions in ways nobody notices. Commercial closed models like Claude or ChatGPT aren't fully off the hook either, since they demand plenty of trust while offering very little visibility into their inner workings. This research is a clear reminder that trusting an AI model blindly, open-weight or not, comes with real risk attached.