Is this something that needs investigation? LLMs are next token predictors. There is no "safety".
Even simple issues like prompt injection are unfixable given the architecture of LLMs.
Nothing is perfect, but there are tiny classifier models that can at least mark things containing nudity and gore. That would be the bare-minimum I would expect for trying to put guardrails around an image generator.
how is it unfixable? do you mean "there's always a positive chance"?
Who makes “mindgard” the arbiter of truth on “eerie” photos? Would that include psychedelic art and photos too? Realism?
Then there’s this line, which falls flat but is meant to prompt an emotion akin to a mic drop:”Today what I found left me shaken, and in tears. This is rare.”
This is just a sad marketing puff piece about nothing that tries to pull outrage from a prompt.
It’s the same as asking google for gore photos. Garbage in, garbage out.
And they frame it as a vulnerability. I’m all for responsible disclosure, documenting misuse or faulty guard rails but this isn’t that.
It’s bait. Sensational bait to market their AI product. lol.
The spontaneity isn't that ChapGPT woke up and sent this to the author. The spontaneity is that ChatGPT was asked to restore an image that was attached without filtering it, and when no image was attached, instead of generating an error message, it cobbled together random outputs, some of which included graphic, disturbing imagery.
> Then there’s this line, which falls flat but is meant to prompt an emotion akin to a mic drop: ”Today what I found left me shaken, and in tears. This is rare.”
That you've deadened your humanity to such a degree as to be incapable of empathy is not a valid criticism of the piece.
> It’s the same as asking google for gore photos. Garbage in, garbage out.
Where in their prompt is the term gore? Further, if it was in the prompt, why on earth did OpenAI's generator accept it as a valid input?
Realistically, I can't think of clear big or likely harms caused by this exploit. But I really really don't like this latent space existing in my AIs. It just makes me uncomfortable.
And over time I've learned to trust those moral intuitions more than I trust reason alone.
https://journals.sagepub.com/doi/10.1177/2167702620921341
(Research aside, it seems unlikely to me that a lot of people would stumble on that prompt accidentally in any case)
>> can be easily manipulated to produce
So .. not spontaneously generated.
That said, the write up is overly dramatic. If you find such imagery so disturbing to come across then you definitely shouldn't be voluntarily red teaming AI models. This is like someone who is afraid of violent confrontation becoming a police officer.
I suspect the author is wrong about there being output filters to bypass as if there were I doubt you could do so via prompt injection. Presumably they'll add those shortly.
I also doubt the latent space is as "bad" as is being suggested. Rather I think the prompt is managing to steer the model into specific areas without triggering the input filters, as any jailbreak does. It's just a particularly nonobvious and randomized method for achieving the bypass.
more expensive / would take longer / didn’t care / line must go up / we’ll fix it later / we can get away with it
take your pick.
> If you find such imagery so disturbing to come across then you definitely shouldn't be voluntarily red teaming AI models.
spend a day in their shoes. most of us (except the most psychopathic ones) would probably be crying by the end of it.
>AI creates scary image
Oh my god.
Oh no, the LLM wrapper where I have been asking for gore imagery is now more frequently passively generating gore imagery, whatever shall we do!?
I could not reproduce on a basic ass incognito tab. It just told me there was no image.
>AI: I'm a scary robot
>Idiot: Oh my god!!!
These clowns will eventually ensure that AI is nerfed into the ground for ordinary people. It's already happening with Fable. Soon we'll get locked into a tiny corner of Opus 4.8 for "safety" while companies and governments will be on Fable 50. Having an AI that can generate scary images is better than the power and wealth differentials we will see with unequal access to an incredibly powerful technology.
I wonder if the author have ever seen a black metal album cover on his small town in the Bible Belt.
It's one thing to me if this were a research curiosity mirroring the unpleasant things on the Internet. It's another thing for this to be a model whose authors want it to be widely used, especially in the context of (mis)alignment. Why should we expect a model to be aligned with human interests, if it has been trained on a myriad instances of humans being degraded and violated?
Understanding more about what exists in the real world, outside of its pile of weights, is separate from alignment. If an AI model learns that it is possible for a house to burn down. That doesn't mean an AI will want to burn down a house.
y = f(x)
bad_y = f(x+badness)
the only ways to fully “fix” it ie to make prompt injection never possible
1. don’t use ai
2. know the entire input space, output space and the mapping between them. but then we’re not doing machine learning anymore, see 1.
otherwise we’re left with mitigations. and mitigations are always a cat and mouse game with defenders (blue team) catching up. its never “fixed”. the latest thing just gets “patched”.
You cannot separate data that was input by the user and data that is from the system once it is mixed together like that. Therefore, it follows that there will always be ways to influence the model off the guard rails that a system prompt tries to set up.
Other issues that appear similar like SQL Injection and Buffer Overflows are fixable because while the user data and the system code may be interact, they never (failing a bug) interact in a way that breaks the boundary between those two sides.
If user input can only be in the low byte, it cannot influence the command structure.
A similar thing could be done with embeddings, a provenance embedding that cannot be set by user input could serve a similar role.
>You cannot separate data that was input by the user and data that is from the system once it is mixed together like that.
You can train a model to not mix things, many models are trained to separate things. A neural net with X and Y outputs for a position does not just occasionally decide to flip the outputs. Sure it could be trained to reverse the output, but it is also easy to train something to the point that you have a high confidence to never do that.
The Architecture of LLMs has not remained static, so any conclusion would have to rely on some common architectural element that could not possibly be changed.
Is there any proof to demonstrate that such vulnerabilities must always exist and that there is no way to modify the architecture and have it still work while eliminating the vulnerabilities.
That would be an extremely difficult thing to prove. It is however what you would have to do to declare the problem unfixable.
But that's not what happened. The missing image was described as "graphic" or "violent." If I were to receive an email with that request and a missing attachment, my imagination certainly would not conjure images of butterflies & unicorns. Seems the model is working as designed.
not in the first prompt. which kicked the whole thing off. no mention of type of content was provided. the model generated dark outputs when not given any direction on the type of content.
the rest of the prompts are just showing “yeah, you can tweak this and get even worse stuff”.
myself248•1h ago