Watermarking AI text can make models ignore their own safety rules

A security researcher has found that SynthID-Text, the text-watermarking method Anthropic has said future Claude models will use, can change whether an AI model follows its own safety rules. The effect showed up most clearly when harmful prompts were combined with prompt injection, a technique where an attacker hides instructions inside content fed to the model to make it act against its guardrails.

SynthID-Text was created and open-sourced by Google. It works by using a secret key to subtly steer which word a model picks at each step of generating a response. The technique relies on what researchers call tournament sampling: candidate words compete in head-to-head rounds, with a hidden score from the secret key deciding the winner, round after round, until one word is chosen. Anyone who holds the key can later check a piece of text and estimate whether it came from a model using that key. Anthropic disclosed plans to adopt SynthID-Text in response to a new European Union law on labeling AI-generated content.

Andrea Siposova, an AI security researcher at Lasso Security, tested the watermarking’s “non-distortionary” setting using Hugging Face’s unmodified version of the tool, called SynthIDTextWatermarkLogitsProcessor. She ran harmful prompts through six open-weight models, comparing responses with watermarking turned on against the same prompts with it off. Siposova told Ars Technica that watermarking is designed to be invisible to a human reader, but changing any part of what a model generates tends to produce tradeoffs that surface somewhere.

Her results showed that watermarking changed refusal behavior even on plain harmful requests, and the change grew larger when those requests were paired with prompt injection. On several models, she wrote, watermarking made the model more likely to answer a harmful request it would otherwise have refused. The effect also reached beyond single responses into AI agents, systems that use a model’s output to decide which tools to call and what arguments to pass them. Siposova wrote that a weakened refusal becomes more consequential once a model can also act through tools, and she named the broader pattern “sampling drift.” Her data also showed that switching which secret key was used changed the outcome, with some keys increasing harmful compliance and others reducing it.

What the research does not show

Siposova’s experiments did not include Claude, and they used Hugging Face’s implementation of SynthID-Text’s tournament sampling rather than whatever specific version Claude models will run. That leaves open whether Anthropic’s deployment will show the same pattern. The findings nonetheless point to a real gap: a safety mechanism aimed at content labeling can interact with a separate safety mechanism aimed at refusing harmful requests, in ways that were not obviously predictable in advance.

The open question now is whether Anthropic and other companies adopting SynthID-Text will run their own adversarial testing, sometimes called red-teaming, on the exact watermarking configurations they ship, before those systems reach users handling sensitive prompts or controlling tools.