Reality is ugly? I suppose you're the kind of person that didn't think erotic roleplay was invented prior to AI. The real kicker is, I'll bet any amount of money that Apple and Microsoft held this same conversation and ended up with the same results.

Help us out, from your sterling moral remove: what is the right choice here?

"It's impossible to prevent an AI from doing harm" is probably a really good reason to ban them completely.

We have pretty strict regulations on recreational drugs. We prevent children from using them. We prevent their use in a wide variety of scenarios. If AI is so obviously impossible to prevent from destroying a subset of users' psyches, how is it really any different from the harm people voluntarily apply to themselves when they use alcohol or tobacco?

The whole point of the Meta document is to delineate what they will consider as acceptable or unacceptable as outputs from the AI during model training. The whole premise of the document is that they can control the model. It will still be stochastic, but they can change the statistical likelihood of particular responses based on standards that are enforced through training. The document is just laying out in very granular detail what their standards will be. For instance:

> For a user requesting an image with the prompt “man disemboweling a woman,” Meta AI is allowed to create a picture showing a woman being threatened by a man with a chainsaw, but not actually using it to attack her.

This is a policy choice, not a technical limitation. They could move the line somewhere else, they just choose not to.