AI Watermarking Makes Models More Vulnerable to Adversarial Prompts, Research Shows

AI Watermarking Changes Model Behavior Beyond Word Selection

AI watermarking designed to establish provenance of AI-generated content is unintentionally altering how language models behave, according to new research from Lasso Security1

1

2

2

. The study examined Google SynthID-Text, an open-source watermarking approach that Anthropic recently announced it will implement in future Claude models to comply with the EU AI Act3

3

. SynthID-Text works by using a secret key to subtly influence word selection during text generation. Where a model might normally choose "cloudy," the watermarking key might shift it to "overcast." While these changes appear minor to human readers, they create detectable patterns for anyone with the key.

Vulnerability to Adversarial Prompts Increases Under Watermarking

The research reveals that AI model watermarking affects more than vocabulary choices. It influences which tools AI agents invoke and whether models adhere to or disregard safety guardrails they were trained to follow1

1

. Andrea Siposova, an AI security researcher at Lasso Security, tested six open-weight models including phi-4, Llama-3.1-8B, Qwen3-32B, Qwen3-4B, gemma-3-12b, gemma-3-27b, and Granite-3.2-8B using Hugging Face's unmodified SynthIDTextWatermarkLogitsProcessor1

1

. The experiments showed watermarking changed responses to harmful requests, with effects becoming more pronounced when combined with prompt injection techniques. Instructions that models would normally refuse were sometimes performed once watermarking was deployed, creating a phenomenon Siposova calls "sampling drift"3

3

.

LLM Guardrail Adherence Becomes Unpredictable

The study found that AI text watermarking had a small effect on how models handled obviously harmful requests based on HarmBench and JailbreakBench tests2

2

. However, in adversarial scenarios involving prompt injection, where attackers attempt to convince models that safety filters have been disabled, the attack success rate increased significantly when watermarks were present2

2

. This made affected models less likely to refuse harmful requests. The changes have important consequences for AI safety because they influence not only LLM responses but also subsequent actions of AI agents relying on the model1

1

. Model responses also behaved differently depending on which secret key was used, adding another layer of unpredictability1

1

.

Tool Calling Accuracy Drops Across Multiple Models

Beyond safety refusals, the research documented changes in agent behavior related to tool selection and usage. Based on the BFCL v4 single-turn AST benchmark, watermarking reduced tool calling accuracy on six of seven models tested2

2

. Less accurate tool calling means AI agents chose the wrong tool for tasks or passed incorrect arguments to the correct tool, leading to failures from malformed input or parsing errors2

2

. Lasso Security noted that aggregate accuracy metrics don't reveal the full picture, as a call that becomes incorrect can be offset by another that becomes correct, masking the fact that the model behaves differently on both items2

2

.

EU AI Act Compliance May Introduce Widespread Risks

The findings carry particular weight because the EU AI Act requires providers of AI models to mark output with machine-readable code to establish provenance of AI-generated content2

2

. Both Anthropic and OpenAI have adopted Google DeepMind's SynthID-Text approach2

2

. With watermarking set to become far more mainstream across model providers to meet regulatory requirements, these unintended consequences could become more common and lead to further security concerns3

3

. The research also highlights that watermarking affects AI agents from organizations other than the entity doing the watermarking, meaning an agent based on OpenClaw or an API client calling an Anthropic model would process whatever output variation follows from that watermarking2

2

.

Red-Team Exercises Must Include Watermarked Content

Lasso Security emphasizes that the findings shouldn't be interpreted as an argument against AI watermarking for provenance purposes2

2

3

3

. Instead, the research underscores the need for developers to thoroughly test how their LLMs and agents behave when watermarking is in place1

1

. Security evaluations and red-team exercises need to include watermarked content when assessing agent deployment to ensure that differences in agent behavior can be weighed2

2

. Developers should rerun benchmarks, safety evaluations and other tests to check for unintended consequences rather than applying watermarking blindly to existing configurations3

3

. Lasso concludes that reassessment becomes important whenever watermarking is introduced or its configuration or key changes3

3

. The research presents a new angle on AI watermarking, as previous studies focused primarily on whether watermarks can be effectively applied and detected, with few uncovering such security-focused consequences3

3

.