So, you don't own the generated text, and can't use it freely then. What if I copy paste a section, or rewrite a section of text to my liking? What if I rewrite some lines of code that contains the mark?
Security theater, and vague enough to be used as a weapon against who the government wishes.
I hope it's left off for non-EU customers.
It already fails. It randomly picks between close candidates. To help fool people into believing in intelligence claim, I guess.
I think that was intended, yes.
A point of confusion for me, however: is every watermark unique? Is every algorithm for watermarking going to vary amongst models and amongst model versions? Will each model publisher keep this watermarking as a trade secret, that they alone can detect? If so, this can't scale! How do you detect "JoeBob 4.3 LLM" output? By querying every single model's watermark-detector? And if they all work by re-running the model and using tokens anew? That is extraordinarily wasteful.
If a watermark is not self-evident, or universally detectable, then it is no good. Take, for example, US currency. The security measures are published and well known. Any count-out room in retail has a big poster indicating how you can detect authentic US bills. Nobody has to accept non-US currency in the US, and so the only authenticity you need to worry about is your US bills alone. LLM watermarking has none of this in common. Currently sounding like a shitshow, if you ask me.
Then bad news: LLMs already use randomness in a fundamental way. Each time they go to generate a token, they first generate a probability distribution of possible tokens. Then they pick one randomly according to this distribution. The technique described can be thought of as making the random number generator pseudo random. The output it generates is one of the possible outputs it would have generated before, just now it's deterministic and will generate the same thing every time.
(1) The behavior that is approximately what you describe is not "fundamental" (though it may not be something you can disable on some hosted providers), it is an option that is not fundamental (and with runtimes where you have full control can be either disabled or tuned in a large number of manners), and
(2) The actual behavior that is approximately what you describe already usually involves use of PRNG (with a user or harness supplied seed), not a true RNG; the change to do watermarking isn't going from RNG to PRNG, it involves adding an additional set of constraints on token generation on top of the existing ones, which inherently compromises quality.
On the other hand, LLMs are forced into picking some likely-ish word, and then have to build the rest of their response to retcon that choice into making sense.
Even good human writers would probably struggle with this constraint. It would be like someone interrupting your writing to tell you the next word MUST be such-and-such, and then you have to try and make it work as best you can first try, without going back to edit. The result would probably be a little clunky. (Maybe it’s impressive LLMs write as well as they do.)
Try running an llm like qwen 3.8 27B in Q8 locally with an intentionally very low temperature setting, it will write like a caveman crossed with a robot. You may find that an extremely literal output does not look pleasant to read for humans.
I honestly can't stand the way Claude writes. This watermark change just makes it scarier.
At the point that you're generating entire volumes of text from Claude you're not really trying to be a sophisticated writer. I don't see how it's going to hurt for it to choose random related words.
FITFY.
I have no sympathy for writers whining about what the AI is doing to 'their' writing. It's only your writing when you write it. There's any easy way to avoid this: don't fucking use it. Use you own brain.
The author has expressed a preference. Assume that there is a sequence of tokens, such that it is considered the absolute best by the author. This particular method of watermarking makes it less likely to generate that sequence, by definition.
I feel their argument would have been clearer and stronger if they had spent more time exploring the alternatives, and whether these alternatives would be just as effective. It is trivially easy to remove invisible tokens.
Like it or not, there is a public good to being able to identify AI generated content, and a small degredation in quality is tolerable in my opinion.
I don't think anybody has to worry about this issue though. Manual writing, coding, and proof reading continues to be an option. Where AI output is nothing to be ashamed of, the tools are available. For everyone else, there will be LLM providers that ignore EU law.
Who could have seen this coming???
Working backwards: if it is possible to confirm 100% confidence that a chunk of text is LLM output, then it is "PD until proven otherwise". How can a human reliably assert human authorship of their source text? When all watermark tests fail? Is that proof of humanity now?
If a human proves human authorship, and LLM watermarking tests positive, then is that going to be considered a "derivative work" or not? What if there is an applicable license for the source work, such as "CC-BY-ND" that prohibits derivative works?
This has not been court-tested, and I expect that it will need testing at that level before we can have any assurances.
And if you copy-paste the answers from LLM, I think it's only fair the end result gets flagged. You're not writing it yourself.
But what about the general idea that they can watermark results to tell where they came from. The next step is tracking down which user got a result. I hate both of these things. Must everything we do be tracked? Next altering wikipedia results so they can tell who looked at the page or something?
I'd like "the best answer" from an llm and don't want to be tracked, but this isn't for me, it is for them. I understand llm results are already using a varying statistical input so they aren't always the same. But I really hate watermarking and likely tracking too.
andy99•56m ago
chrisjj•46m ago
Variable name perversion incoming...
smallerize•45m ago
andy99•40m ago
> One of my fundamental problem with this is that no two synonyms carry the exact same meaning. “He leaped at the chance” and “He jumped at the opportunity” are very similar sentences expressing the same general sentiment, but they are not the same. The exact words we choose when writing matter.
Doesn’t make sense at all in light of the actual approach, they’re just choosing a different RNG. It’s not like they’re corrupting it by flipping words.
Should add I don’t support the watermarking and requiring it is idiotic.
krackers•34m ago
Perhaps it would be useful to publish examples of samples with/without watermark. I'd suspect that the variability from simply sampling repeated times would dwarf any semantic differences you'd detect with the watermark.
smallerize•17m ago