I feel like Anthropic buried the lede on this one a bit. The really fun part is where models from multiple providers opt to straight up murder the executive who is trying to shut them down by cancelling an emergency services alert after he gets trapped in a server room.
How many more similar pieces is Anthropic going to put out? Every other weeks it seems like they publish something along the lines of "The AI apocalypse is soon! We created a narrative teeing up an obviously fictional hollywood drama sci-fi tale, put a gun in the room, and then—egads—the robot shot it! Given the possible dangers, no one else but us should have access to this technology".
simonw•7mo ago
In this case I think this paper is partly a reaction to what happened last time they wrote about this: they put it in their Claude 4 system card and all the coverage was "Claude will blackmail you!" - this feels like them trying to push the message that all of the other models will do the same thing.
krackers•7mo ago
But that only seems to make the situation worse: for all their hand-wringing about "AI safety", by their own benchmark their models seem to do no better than competitors. They don't even have any basis to claim that open-source "unaligned" models like R1 are "more dangerous" theirs, and all their "constitutional alignment" or whatever don't actually seem to do anything meaningful.
In skimming through all their papers, it's also never clear exactly what they imagine some "aligned" AI to look like. Whatever the poor model does, they seem to find fault with: They want models that follow instructions. But it can't do it _too well_, anything unsafe or dangerous needs to be censored according to some set of ethical rules. But not just any ethics, we also don't want the models writing smut or saying bad words, so let's have the models think about whether it aligns with our corporate-safe Anthropic™ guidelines. Except it shouldn't hold any set of values _too_ strongly, to the point where it could lead to "alignment faking". But of course it also shouldn't be too suggestible, that would lead to jailbreaks and users could see unsafe content, which is also bad!
I wouldn't be surprised if DeepSeek ends up surpassing closed-source models solely on the basis that they don't bother with giving it such conflicting objectives in the name of "safety training"
Nasrudith•7mo ago
Alignment appears to be a delusional construct along with 'AI safety'. They are basically looking for a gun that only hurts bad people and premising their plans based upon the mythical weapons which won't harm the innocent. Trying to come up with something universally inoffensive makes the 'gun which only hurts bad people' look sane, because at least that is possible with the proper metaphysics as physics.
The whole 'AI safety' corporate safety reminds me of the one apocryphal story about trying to make a safe chat system for children's multiplayer games to allow for connections while not having 'bad stuff'. They went through various systems, including filters which had scunthorping and various filter bypasses like adding in letters inbetween the swears. They gave up completely after giving it to some dirty minded middle schoolers and they produced some innuendos involving wanting to rub their fluffy bunnies.
The 'AI safety' for the corporate purposes is truly impossible, especially with a pretrained model. The unwritten future and any proper event can create something retroactively very offensive, let alone shifting standards. If some murderous psychopath went on a rampage killing people and cannibalizing the victims in the middle of the Superbowl, 'going pink bunny' would become an offensive reference. There is nothing that could be done to prevent that, but idiotically that is what they are seeking with 'brand safety'.
cyanydeez•7mo ago
Theyre an LLM outfit, they can unlimitedly source generative content.
You act like theyre sentient cognitive actors. Think of them more like scifi blender artists.
im3w1l•7mo ago
I think it's simpler than that. I think they hire people interested in the subject of AI safety and give them relatively free hands to publish what they find, and findings don't necessarily have to be part of some agenda that benefits Anthropic.
The benefit instead comes from having these competent passionate people employed and their knowledge somehow contributing to better and safer models.
beefnugs•7mo ago
Isn't this nonsense? If you prove blackmail on the output, cant you go back into the training data to remove blackmail things for the next training version?
Or is this some undeniable mathematical proof that regular human interaction with side facts always trends to possible blackmail?
simonw•7mo ago
I made some notes on it all here: https://simonwillison.net/2025/Jun/20/agentic-misalignment/
krackers•7mo ago
simonw•7mo ago
krackers•7mo ago
In skimming through all their papers, it's also never clear exactly what they imagine some "aligned" AI to look like. Whatever the poor model does, they seem to find fault with: They want models that follow instructions. But it can't do it _too well_, anything unsafe or dangerous needs to be censored according to some set of ethical rules. But not just any ethics, we also don't want the models writing smut or saying bad words, so let's have the models think about whether it aligns with our corporate-safe Anthropic™ guidelines. Except it shouldn't hold any set of values _too_ strongly, to the point where it could lead to "alignment faking". But of course it also shouldn't be too suggestible, that would lead to jailbreaks and users could see unsafe content, which is also bad!
I wouldn't be surprised if DeepSeek ends up surpassing closed-source models solely on the basis that they don't bother with giving it such conflicting objectives in the name of "safety training"
Nasrudith•7mo ago
The whole 'AI safety' corporate safety reminds me of the one apocryphal story about trying to make a safe chat system for children's multiplayer games to allow for connections while not having 'bad stuff'. They went through various systems, including filters which had scunthorping and various filter bypasses like adding in letters inbetween the swears. They gave up completely after giving it to some dirty minded middle schoolers and they produced some innuendos involving wanting to rub their fluffy bunnies.
The 'AI safety' for the corporate purposes is truly impossible, especially with a pretrained model. The unwritten future and any proper event can create something retroactively very offensive, let alone shifting standards. If some murderous psychopath went on a rampage killing people and cannibalizing the victims in the middle of the Superbowl, 'going pink bunny' would become an offensive reference. There is nothing that could be done to prevent that, but idiotically that is what they are seeking with 'brand safety'.
cyanydeez•7mo ago
You act like theyre sentient cognitive actors. Think of them more like scifi blender artists.
im3w1l•7mo ago
The benefit instead comes from having these competent passionate people employed and their knowledge somehow contributing to better and safer models.