The molotov cocktail example is so stupid, because how to make it is essentially entailed in knowing what it is. At least they could do making meth, or better still- something not readily found on the internet that gives a non-expert new capabilities. If there was a Claude code for crime, that wouldn't be in society's interest. As it is, these trivial examples are just testing the strength of built in refusals, and should be represented as such, instead of anything related to safety.

Why wouldn't any reasonably good AI be able to replicate large portions of the US Army TM 31-210 "Improvised Munitions Handbook"?

“AI Safety” is really about whether its “safe” (economically, legally, reputationally) for a third partyy corporation (not the company which created the model) to let customers/the public interact with them via an AI interface.

If a Mastercard AI talks with customers and starts saying the n-word, it’s not “safe” for Mastercard to use that in a public-facing role.

As org size increases, even purely internal uses could be legally/reputationally hazardous.

Not divulging information with third parties seems a common theme, even when said business practices are questionable.