"but it's getting cheaper and cheaper to run inference" they say. To which I say:

ok bud sure thing, this isn't a technology governed by Moore. We'll see.

And Snapchat, TikTok, Facebook, YouTube, Instagram, more?.. are just it. They waste time worth $50-$100 just to make $0.05 in ads.

LLMs seem to be far from that.

* I might be ignorant of the real picture, please correct me if I'm wrong and not aware of really evil LLM companies.

It might be an actual order-of-magnitude increase from the background rates. It might be basically the same headcount, but with psychosis expressed in a new notable way. I.e. that guy who would be planning to build a perpetual motion machine to disprove quantum physics is now doing the same, but with an AI chatbot at his side - helping him put his ramblings into a marginally more readable form.

Mostly a joke but one the common properties in people with an delusional or otherwise unhealthy pattern of AI use appears to be substituting it for all their own choices, even where it's obviously inappropriate. E.g. if you try to intervene in their conduct, you'll just get an AI generated reply.

[1]https://youtube.com/watch?v=gkBXfmnlVIg

(It's a fairly awkward phrase, and you wouldn't say "I almost attempted to run a marathon", say, but "attempt suicide" has a fairly specific commonly-understood meaning where 'attempt' is unusually restrictive vs general usage)

OpenAI thinks that as long as you aren't using dating terminology or asking to build bombs, you're probably safe.

- What does having an unquestioning mirror that is always gracious and reflects your own biases back at you, do to the mind in the long term? We don't know.

- How do you know, that after 2-3 years of this use, without any particular warning, it will not suddenly escalate into a delusion spiral? We don't know.

- Does this affect childhood development to have no pushback on their personal feelings and opinions? We don't know.

- Does this affect people's very willingness to learn, and to endure intellectual struggle? We don't know.

- Does this affect long-term retention, as losing even an imperceptible 2% of your knowledge every month, is 48% lost after 2 years? We don't know.

- How does this affect a population of which 27% have experienced a mental health issue in the last year, and of which 9% are experiencing depression? We don't know.

- Are all these problems solvable with any amount of education or warning labels, given that human nature has not changed and will not change? We don't know.

- Can this be prevented with any amount of AI guardrails or ethics training, or is it inherent to the structure of the system itself? We don't know.

- Will this eventually be considered so dangerous, that we need black-box warning labels similar to cigarettes and possibly social media? We don't know.

- Will this eventually be considered so insolvably risky for human cognition, that we outright restrain use without a license, either like driving at best, or asbestos at worst? We don't know.

- Will any employer be able to obtain insurance for AI use, if there is even a 0.2% documented chance that any employee could go psychotic from using it, completely at random? We don't know.

- Will said employees, if they do go psychotic from a work-mandated AI chatbot, successfully sue their employers for damages? We don't know.

That's just what a malicious AI would want you to think...

But seriously, there doesn't need to be any "intent" for something to be harmful. A deadly virus has no will or agency, and particular individuals may be completely immune to it. But released on a population the results are catastrophic. We have to accept that perhaps for the first time ever we've created something we can't fully comprehend or control, and that regardless of our best wishes it will be harmful in ways we could never have imagined. If it were as simple as prompting the next model against this stuff, it would have already been done.

It uses words and phrases most commonly associated with other text that humans have labeled as "helpful", "friendly", etc. That sort of thing.

That's different than wanting. It's text probabilities in a box. It can't want anything.

I am normally not a stickler, but in this case the distinction matters.

OpenAI attempted to nuke their overly-agreeable LLM (GPT-4o). After user outcry they brought it back. I wouldn't be too sure that companies will do the responsible thing here; there's a pretty ugly history of companies who sell to addicts deliberately making their products more addictive and appealing to said addicts (the tobacco industry is particularly notorious for this).

Like, one view would be that LLM 'companions' and 'therapists' are inherently harmful, or at least so likely to be net-harmful that making them available would be unjustifiable. But that's certainly not the view that the vendors are taking.

> It has no long term plans - it just wants the user to like it.

To be clear, it doesn't 'want' anything.