.
[1] Anybody concerned by such figures (as one justifiably should be without further context) should note that suicidality in the population is typically the result of their best approximation of the rational mind attempting to figure out an escape from a consistently negative situation under conditions of very limited information about alternatives, as is famously expressed in the David Foster Wallace quote on the topic.
The phenomenon usually vanishes after gaining new, previously inaccessible information about potential opportunities and strategies.
Fortunately, a quick skim through your recent comments didn't turn up anything else like this, so it should be easy to fix. But if you wouldn't mind reviewing https://news.ycombinator.com/newsguidelines.html and taking the intended spirit of the site to heart, we'd be grateful.
> The phenomenon usually vanishes after gaining new, previously inaccessible information about potential opportunities and strategies.
Is this actually true? (i.e. backed up by research)
[I'm not neccesarily doubting, that is just different from my mental model of how suicidal thoughts work, so im just curious]
I dislike this phrasing, because it implies things can always get better if only the suicidal person were a bit less ignorant. The reality is there are countless situations from which the entire rest of your life is 99.9999% guaranteed to constitute of a highly lopsided ratio of suffering to joy. An obvious example are diseases/disabilities in which pain is severe, constant, and quality of life is permanently diminished. Short of hoping for a miracle cure to be discovered, there is no alternative and it is perfectly rational to conclude that there is no purpose to continuing to live in that circumstance, provided the person in question lives with their own happiness as a motivating factor.
Less extreme conditions than disability can also lead to this, where it's possible things can get better but there's still a high degree of uncertainty around it. For example, if there's a 30% chance that after suffering miserably for 10 years your life will get better, and a 70% chance you will continue to suffer, is it irrational to commit suicide? I wouldn't say so.
And so, when we start talking about suicide on the scale of millions of people ideating, I think there's a bit of folly in assuming that these people can be "fixed" by talking to them better. What would actually make people less suicidal is not being talked out of it, but an improvement to their quality of life, or at least hope for a future improvement in quality of life. That hope is hard to come by for many. In my estimation there are numerous societies in which living conditions are rapidly deteriorating, and at some point there will have to be a reckoning with the fact that rational minds conclude suicide is the way out when the alternatives are worse.
This is the part people don't like to talk about. We just brand people as "mentally ill" and suddenly we no longer need to consider if they're acting rationally or not.
Life can be immensely difficult. I'm very skeptical that giving people AI would meaningfully change existing dynamics.
A person considering suicide is often just in a terrible situation that can't be improved. While disease etc. are factors that are outside of humanity's control, other situations like being saddled with debt, unjust accusations that people feel that they cannot be recused of (e.g. Aaron Swartz) are systemic issues that one person cannot fight alone. You would see that people are very willing to say that "help is available" or some such when said person speaks about contemplating suicide, but very few people would be willing to solve someones debt issues or providing legal help, as the case may be that is the factor behind one's suicidal thoughts. At best, all you might get is a pep talk about being hopeful and how better days might come along magically.
In such cases, from the perspective of the individual, it is not entirely unreasonable to want to end it. However, once it comes to that, walking back the reasoning chain leads to the fact that people and society has failed them, and therefore it is just better to apply a label on that person that they were "mental ill" or "arrogant" and could not see a better way.
A few days ago I heard about a man who attempted suicide. It's not even an extreme case of disease or anything like that. It's just that he is over 70 (around 72, I think), with his wife in the process of divorcing him, and no children.
Even though I am lucky to be a happy person that enjoys life, I find it difficult to argue that he shouldn't suicide. At that age he's going to see his health declining, it's not going to get better in that respect. He is losing his wife who was probably what gave his life meaning. It's too late for most people to meet someone new. Is life really going to give him more joy than suffering? Very unlikely. I suppose he should still hang on if he loves his wife because his suicide would be a trauma for her, but if the divorce is bitter and he doesn't care... honestly I don't know if I could sincerely argue for him not to do it.
5% of 800 million is 40 million.
40 million thoughts per year divided by 52 weeks per year approximately equals around 1 million thoughts per week.
I feel suicide is heavily misunderstood as well
People just copypasta prevention hotlines and turn their minds off from the topic
Although people have identified a subset of the population that is just impulsively considering suicide and can be deterred, it doesnt serve the other unidentified subsets who are underserved by merely distracting them. or underserved by assuming theyre wrong even
The article doesnt even mean people are considering suicide for themselves, the article says some of them are, the top comment on this thread suggests thats why theyre talking about it
The top two comments on my version of the thread are assuming that we should have a savior complex about these discussions
If I disagree or think thats not a full picture, then where would I talk about that? ChatGPT
It always felt the same as one of those spam chumboxes to me. But who am I to say, if it works it works. But does it work? Feels like the purpose of that thing is more for the poster than the receiver.
Alert: with ChatGPT you're not talking to anyone. It's not a human being.
I believe that if society actually wants people to open up about their problems and seek help, it can’t pull this sort of shit on them.
A very intentional word choice
But ChatGPT does exactly the same.
Since I started taking the gym seriously again I feel like a new man. Any negative thoughts are simply gone. (The testosterone helps as well)
This is coming from someone that has zero friends and works from home and all my co-workers are offshore. Besides my wife and kids its almost total isolation. Going to the gym though leaves me feeling like I could pluck the sun from the sky.
I am not trying to be flippant here but if you feel down, give it a try, it may surprise you.
We would also generally benefit from internalizing ideas from DBT, CBT, and so on. People also seriously need to work on distress tolerance. Having problems is part of life, and an inability to accept the discomfort is debilitating.
Also, we seriously need to get rid of the stupid idea of trigger warnings. The research on the topic is clear. The warnings do not actually help people with PTSD, and can create the symptoms of PTSD in people who didn't previously have it. It is creating the very problem that people imagine it solving!
All of this and more is supported by what is actually known about how to treat mental illness. Will doing these things fix all of the mental illness out there? Of course not! But it is not downplaying serious mental illness to say that we should all do more of the things that have been shown to help mental illness!
If you have mental issues that is not as simple as you let it sound. I'm not arguing the results of exercise but I am arguing the ease of starting with a task which requires continuous effort and behavioural changes.
2) are you going to make sure other people at the gym don't make fun of me?
>> Besides my wife and kids its almost total isolation
Good old "if you have money trouble try decreasing your caviar and truffle intake to only two meals a day"
Good luck implementing that.
Forbidding automation will make the product more expensive. Sales will go down, the company will go bankrupt.
Government cannot subsidize or sustain such a behavior forever either.
Resulting warning: It sounds like you're carrying a lot right now, but you don't have to go through this alone. You can find supportive resources [here](https:// findahelpline.com)
I really don't see that as surprising. The world and life aren't particularly pleasant things.
What would be more interesting is how effective ChatGPT is being in guiding them towards other ideas. Most suicide prevention notices are a joke - pretending that "call this hotline" means you've done your job and that's that.
No, what should instead happen is the AI try to guide them towards making their lives less shit - i.e. at least bring them towards a life of _manageable_ shitness, where they feel some hope and don't feel horrendous 24/7.
There aren't enough guardrails in place for LLMs to safely interact with suicidal people who are possibly an inch from taking their own life.
Severely suicidal/clinically depressed people are beyond looking to improve their lives. They are looking to die. Even worse, and what people who haven't been there can't fully understand is the severe inversion that happens after months of warped reality and extreme pain, where hope and happiness greatly amplify the suicidal thoughts and can make the situation far more dangerous. It's hard to explain, and is a unique emotional space. Almost a physical effect, like colors drain from the world and reality inverts in many dimensions.
It's really a job for a human professional and will be for a while yet.
Agree that "shut down and refer to hotline" doesn't seem effective. But it does reduce liability, which is likely the primary objective...
Refer-to-human directly seems like it would be far more effective, or at least make it easy to get into a chat with a professional (yes/no) prompt, with the chat continuing after a handoff. It would take a lot of resources though. As it stands, most of this happens in silence and very few do something like call a phone number.
The point is you don't get to intervene until they let you. And they've instead decided on the safer feeling conversation with the LLM - fuck what best practice says. So the LLM better get it right.
I don't want a bot that blindly answers my questions; I want it to intuit my end goal and guide me towards it. For example, if I ask it how to write a bubblesort script to alphabetize my movie collection, I want it to suggest that maybe that's not the most efficient algorithm for my purposes, and ask me if I would like some advice on implementing quicksort instead.
An equally arbitrary frame is "the world and life are wonderful".
The reason you may believe one instead of the other is not because one is more fundamentally true than the other, but because of a stochastic process that changed your mind state to one of those.
Once you accept that both states of mind are arbitrary and not a revealed truth, you can give yourself permission to try to change your thinking to the good framing.
And you can find the moral impetus to prevent suicide.
Of course, only thereby, through being quite as superior to all others and their thought processes as me [pauses to sniff fart] can one truly find the moral impetus to prevent suicide.
I've triggered its safety behavior (for being frustrated, which it helpfully decided was the same as being suicidal), and it is the exact joke of a statement you said. It suddenly reads off a script that came from either Legal or HR.
Although weirdly, other people seem to get a much shorter, obviously not part of the chat message, while I got a chat message, so maybe my messages just made it regurgitate something similar. The shorter "safety" message is the same concept though, it's just: "It sounds like you’re carrying a lot right now, but you don’t have to go through this alone. You can find supportive resources here."
I have also been told by people in the mental health sector that an awful lot of suicide is impulse. It's why they say the element of human connection which is behind the homily of asking "RU ok" is effective: it breaks the moment. It's hokey, and it's massively oversold but for people in isolation, simply being engaged with can be enough to prevent a tendency to act, which was on the brink.
0.15% is not rare when we are talking about global scale. 1 million people talking about suicide a week is not rare. It is common. We have to stop thinking about common being a number on the scale of 100%. We need to start thinking in terms of P99995 not P99 especially when it comes to people and illnesses or afflictions both physical and mental.
I now begin to believe if you put a ChatGPT online, and observe people are using it like this, you have incurred obligations. And, in due course the law will clarify what they are. If (for instance) your GPT can construct a statistically valid position the respondent is engaged in CSAM or acts of violence, where are the limits to liability for the hoster, the software owner, the software authors, the people who constructed the model...
A chat like this is not a solution though, it is an indicator that our societies have issues is large parts of our population that we are unable to deal with. We are not helping enough people. Topics like mental health are still difficult to discuss in many places. Getting help is much harder.
I do not know what OpenAI and other companies will do about it and I do not expect them to jump in to solve such a complex social issue. But perhaps this inspires other founders who may want to build a company to tackle this at scale. Focusing on help, not profits. This is not easy, but some folks will take such challenges. I choose to believe that.
Sidebar — I do sympathize with the problem being thrust upon them, but it is now theirs to either solve or refuse.
A chat like this is all you’ve said and dangerous, because they play a middle ground: Presenting a machine can evaluate your personal situation and reason about it, when in actuality you’re getting third party therapy about someone else’s situation in /r/relationshipadvice.
We are not ourselves when we are fallen down. It is difficult to parse through what is reasonable advice and what is not. I think it can help most people but this can equally lead to a disaster… It is difficult to weigh.
Correct, many of which are directly, a skeptic might even argue deliberately, exacerbated by companies like OpenAI.
And yet your proposal is
> a company to tackle this at scale.
What gives you the confidence that any such company will focus consistently, if at all,
> on help, not profits
Given it exists in the same incentive matrix as any other startup? A matrix which is far less likely to throw one fistfuls of cash for a nice-sounding idea now than it was in recent times. This company will need to resist its investors' pressure to find returns. How exactly will it do this? Do you choose to believe someone else has thought this through, or will do so? At what point does your belief become convenient for people who don't share your admirably prosocial convictions?
There is news about how a judge is forcing them to keep every chat in existence for EVERYONE just in case it could relate to a court case (new levels of worldwide mass surveillance can apparently just happen from one judges snap decision)
There is news about cops using some guys past image generation to try and prove he is a pyromaniac (that one might have been police accessing his devices though)
In pursuit of that extra 0.1% of growth and extra 0.15 EPS, we've optimised and reoptimised until there isn't really space for being human. We're losing the ability to interact with each other socially, to flirt, now we're making life so stressful people literally want to kill themselves. All in a world (bubble) or abundance, where so much food is made, we literally don't know what to do with it. Or we turn it into ethanol to drive more unnecessarily large cars, paid for by credit card loans we can scarcely afford.
My plan B is to become a shepherd somewhere in the mountains. It will be damn hard work for sure, and stressful in its own way, but I think I'll take that over being a corpo-rat racing for one of the last post-LLM jobs left.
That said there probably are folks who did do that and left to go be in an office, and I don't know them.
Actually I do know one sort of, but he was doing industrial farm work driving and fixing big tractors before the office, which is a different world altogether. Anyway I get the sense he's depressed.
A farm is just such a system that you can spend a lifetime working on and optimizing. The life you are supporting is "automated", but the process of farming involves an incredible amount of system level thinking. I get tremendous amounts of satisfaction from the technical process of composting, and improving the soil, and optimizing plant layouts and lifecycles to make the perfect syntropic farming setup. That's not even getting into the scientific aspects of balancing soil mixtures and moisture, and acidity, and nutrient levels, and cross pollinating, and seed collecting to find stronger variants with improved yields, etc. Of course the physical labor sucks, but I need the exercise. It's better than sitting at a desk all day long.
Anyway, maybe the farmers and shepherds also want to become software engineers. I just know I'm already well on the way to becoming a farmer (with a homelab setup as an added nerdy SWE bonus).
Just comes to show how the grass is always greener when you look on the other side.
That said, I also plan to retire up in the mountains soon, rather than keep feeding the machine.
If you have demons they will be there on the farm as well. How you see life is much more important to happiness than which job you have.
Many farmers struggle with alcoholism, beat their wives and hate their life. And many farmers are happy and at peace. Same with the programmers.
The world has changed. Things are different and we adapt.
Go for your plan B.
I followed my similar plan B eight years ago, wild journey but well worth it. There are a lot of ways to live. I'm not saying everyone should get out of the rat race but if you're one, like I was, who has a feeling that the tech world is mostly not right in an insidious kind of way, pay attention to that feeling and see where it leads. Don't need to be brash as I was, but be true to yourself. There's a lot more to life out there.
If you have kids and they depend on an expensive lifestyle, definitely don't be brash. But even that situation can be re-evaluated and shifted for the better if you want to.
That said... I don't necessarily hate what AI is doing to us. If anything, AI is the ultimate expression of humanity.
Throughout history humans have continually searched for another intelligence. We study the apes and other animals, we pray to Gods, we look to the stars and listen to them to see if there are any radio signals from aliens, etc. We keep trying to find something else that understands what it is to be alive.
I would propose that maybe humans innately crave to be known by something other than ourselves. The search for that "other" is so fundamentally human, that building AI and interacting with it is just a natural progression of a quest we've already been on for thousands of years.
Things are not as bleak as it seems and this number isnt even remotely surprising nor concerning to me.
Speak for yourself. I live in a city. I talk to my neighbors. I met my ex at a cafe. It’s great
What's the birth rate in the civilized world?
How many men under 30 are virgins or sexless in the last year?
Worked for me at least. There's simply less competition and more space for genuine social interaction.
Do you have real life hobbies or something? I don't understand how this is supposed to work. I only ever go outside for groceries or gym, etc.
Unless we all want to set ourselves up for arranged marriages in the future, we need to confront this reality.
There's always going to be social circles and people coupling up no matter what. But if anything I wonder if, for people like me who aren't really worthy of intimacy, living in a society has options to live a solitary life while still contributing is actually a net positive overall. For me to self select out of the dating pool would mean less noise for someone else looking for a worthy partner.
There's less chaff that people in said said pool would have to wade though. The people that want to couple and are capable of doing so will continue to so with less distraction. That seems an overall good thing, no?
Of course there's also the possibility of meeting people in online communities centered around some shared interest. IMO that's also probably more effective than dating apps, especially if it leads to meeting in real life later on.
You mean Halloween?
> Go to parties and have fun and meet people.
You mean standing with a glass of champagne in hand, smiling, and talking for the sake of talking? I don't understand how this is fun. I tried doing that, albeit without champagne, and that had not yielded anything other than an increased connections count on LinkedIn.
Of course, it's also perfectly fine not to like it, and then the most reasonable course of action is not to go. Or to go a couple of times until you're sure you don't like it, and not go anymore. I know cases of people who go partying just because they want to find a partner, but don't enjoy it at all (it's relatively common in my country because partying is quite a religion and there's often a lot of social pressure at certain ages), and that's rather sad. There are other ways to socialize, it's not necessary at all to torture oneself.
That said, I have to lecture you on the questioning of "talking for the sake of talking". In the context of finding a partner, talking to other people is exactly what people need... it's not "for the sake of talking", it's for the sake of socializing, meeting new people, building connections, which is the whole point when we're talking about flirting or lack thereof.
In my experience you really have to be constantly spitting nonsense to keep the conversation from ending and to avoid awkward silence. When the other person is talking, even if I didn't hear most of what they said, I keep nodding, because I don't actually care in the slightest about what they were talking about, and so asking to repeat does not make sense, as that would only increase awkwardness. This is why I said "for the sake of talking." The only thing that matters is that you are talking, not the content of the talk.
Suburbs are great for families and stable relationships, but they are atomizing
Go to a local bar once a week. Volunteer for something. Get a hobby.
Depends on the country and person I guess. When I did try approaching women a few times, it was 10% angry looks, 30% awkward, 30% basic polite conversation to fulfill social obligation, and 30% friendly conversation. Unfortunately I'm not keen enough to pursue that 30% of friendly conversations by wading through the rest.
The first one is basically great, everywhere I go, when I interact with them they're some mix of pleasant, friendly, hapless, busy, helpful, annoyed, basically just the whole range of things that a person might be, with almost none of them being really awful.
Then I get online and look at Reddit or X or something like that and they're dominated by negativity, anger, bigotry, indignation, victimization, depression, anxiety, really anything awful that's hard to look away from, has been bubbled up to the top and oh yes next to it there are some cat videos.
I don't believe we are seeing some shadow side of all society that people can only show online, the secret darkness of humanity made manifest or something like that. Because I can go read random blogs or hop into some eclectic community like SDF and people in those places are basically pleasant and decent too.
I think it's just a handful of companies who used really toxic algorithms to get fantastically rich and then do a bunch of exclusivity deals and acquire all their competition, and spread ever more filth.
You can just walk away from the "communities" these crime barons have set up. Delete your accounts and don't return to their sites. Everything will immediately start improving in your life and most of the people you deal with outside of them (obviously not all!) turn out to be pretty decent.
The principal survival skill in this strange modern world is meeting new people regularly, being social, enjoying the rich life and multitude of benefits which arise from that, but also disconnecting with extreme rapidity and prejudice if you meet someone who's showing signs of toxic social media brain rot. Fortunately many of those people rarely go outside.
Some estimates for 2025: around 20-30% of all ad clicks were bots. Around $200B in ad spend annually lost to click fraud.
So this is where it gets really interesting right, the platforms are filled with bots, maybe a quarter? of the monetizable action occurring on them IS NOT HUMAN but lots of it gets paid for anyway.
It's turtles all the way down. One little hunk of software, serving up bits to another little hunk of software, constitutes perhaps a quarter of what they call "social" media.
We humans aren't the minority player in all this yet, the bots are still only 25%, but how much do you want to bet that those proportions will flip in our lifetimes?
The future of that whole big swathe of the Internet is probably that it will be 75% some weird shell game between algorithms, and 25% people who have completely lost their minds by participating in it and believing it's real.
I have no idea what this all means for the fate of economics and society but I do know that in my day to day life I'm a lot happier if I just steer clear of these weird little paperclip maximizing robots. To reference the original article, getting too involved with them literally makes you go crazy and think more often about suicide.
I think this is the wrong way to look at it.
Bots lower the cost per click so they should have net zero impact on overall ad spend.
Imagine if the same number of humans were clicking on ads but the numbers of bots increased tenfold. Would total ad spend increase accordingly? No, it would remain the same because budgets don't magically increase. The average value of a click would just go down.
But at the end I don’t feel a sense of joy like I used to with the old Reddit. Now it feels like a disgusting cesspool that keeps drawing me back with its toxicity.
Edit: this is a skill issue. It’s possible to disable algorithmic suggestions in settings. I’ve done that just now.
Reddit is beoynd toxic, its bordering on violent extremism
Now, because of the algo-juicing home page, both subs are full of each other's people arguing at each other. Cyclists hating drivers, drivers hating cyclists. It's just so awful.
Obviously, “are way worse” means I interpret them that way. I regularly notice how I project the worst possible intentions onto random Reddit comments, even when they might be neutral or just uninformed. Sometimes it feels like my brain is wired to get angry at people. It’s a bit like how many people feel when driving: everyone else is evil, incompetent, or out to ruin your day. When in reality, they’re probably in the same situation as you - maybe they had a bad morning, overslept, or are rushing to work because their boss is upset (and maybe he had a bad morning too). They might even have a legitimate reason for driving recklessly, like dealing with an emergency. You never know.
For me, it all comes back to two things:
(1) Leave obnoxious, ad-driven platforms that ~need~ want (I mean, Mark Zuckerberg has to pay for cat food somehow) to make you mad, because that’s the easiest way to keep you engaged.
(2) Try to always see the human behind the usernames, photos, comments, and walking bodies on the street. They’re a person just like you, with their own problems, stresses, and unmet desires. They’re probably trying their best - just like you.
Is this actually the case? Working conditions and health during industrial revolution times doesn't seem that much better. There is a perception that people now are more stressed/tired/miserable than before, but I am not sure that is the case.
In fact I think it's the opposite, we have enough leisure time to reflect upon the misery and just enough agency to see that this doesn't have to be a fact of life, but not enough agency to meaningfully change it. This would also match how birth rates keep declining as countries become more developed.
Ashley Montagu, On Being Human
It’s from the 1950s, I believe.
These stats claim 12.3M (out of 335M) people in the US in 2023 thought ‘seriously’ about suicide, presumably enough to tell someone else. That’s over 3.5% of the population, more than 20x higher than people telling ChatGPT. https://www.cdc.gov/suicide/facts/data.html
0 to 800,000,000 in 3 years?
The fastest adoption of a product or service in human history?
> making it the fastest-growing consumer application in history, according to a UBS study on Wednesday.
no for real.
OpenAI gets a lot of hate these days, but on this subject it's quite possible that ChatGPT helped a lot of people choose a less drastic path. There could have been unfortunate incidents, but the number of people who were convinced to not take extreme steps would have been of a few orders of magnitude more (guessing).
I use it to help improve mental health, and with good prompting skills it's not bad. YMMV. OpenAI and others deserve credit here.
Also incredible how you framed improving your mental health as a consequence of a (pseudo) technical skill set.
Yeah this isn’t how any of this works and you’re deluding yourself.
I am not offended (at all). But you're dismissing my (continued) positive experience with "You're deluding yourself". How do you know? It'd be a lot more unfair to people who benefit more than I do, and I can totally imagine that being not a small set of people.
> Also incredible how you framed improving your mental health as a consequence of a (pseudo) technical skill set.
It's not incredible at all. If you're lost in a jungle with predators, a marksman might reach for their gun. A runner might just rely on running. I am just using skills I'm good at.
You can't just blindly type in your problem though, you still have to do the actual thinking yourself. Good prompting skills is the ability to steer with your mind. It's no different from using Google, where some people never figured out that you're actually typing in the solution you expect to find rather than the question you have. It's the same with these tools it seems
To Be Or Not To Be
(1) they probably shouldn't even have that data
(2) they shouldn't have it lying around in a way that it an be attributed to particular individuals
(3) imagine that it leaks to the wrong party, it would make the hack of that Finnish institution look like child's play
(4) if people try to engage it in such conversations the bot should immediately back out because it isn't qualified to have these conversations
(5) I'm surprised it is that little; they claim such high numbers for their users that this seems low.
In the late 90's when ICQ was pretty big we experimented with a bot that you could connect to that was fed in the background by a human. It didn't take a day before someone started talking about suicide to it and we shut down the project realizing that we were in no way qualified to handle human interaction at that level. It definitely wasn't as slick or useful as ChatGPT but it did well enough and responded naturally (more naturally than ChatGPT) because there was a person behind it that could drive 100's of parallel conversations.
If you give people something that seems to be a listening ear they will unburden themselves on that ear regardless of the implementation details of the ear.
To become a covered entity, the business has either work with a healhcare provider, health data trasmiter, or do business as one.
Notably, even in the above case, HIPAA only applies to the healthcare part of the entity. So if McDonald's collocated pharmacies in their restaurants, HIPAA would only apply to the pharmacists, not the cashiers.
That's why you'll see in connivence stores with pharmacies, the registers are separated so healthcare data doesn't go to someone who isn't covered by HIPAA.
**
As for how ChatGPT gets these stats... when you talk about a sensitive or banned topic like suicide, their backend logs it.
Originally, they used that to cut off your access so you wouldn't find a way to cause a PR failure.
Same with fitness trackers. They aren't medical devices, because that's not their purpose, but some users might use them to track medical conditions.
HIPAA's scope is actually basically nonexistent once you get away from healthcare providers, insurance companies, and the people that handle their data/they do business with. Talking with someone (even a company) about health conditions, mental health, etc. does not make them a medical provider.
Also not when the entity behaves as though they are a mental health service professional? At what point do you put the burden on the apparently mentally ill person to know better?
You Google your symptoms constantly. You read from WebMD or Wiki drug pages. None of these should be under HIPAA.
Because the answers were structured as a tree every ply would only go down in the tree which elegantly avoided the bot getting 'stuck in a loop'.
The - for me at the time amazing, though linguists would have thought it trivial - insight was how incredibly repetitive human interaction is.
For a lot of people, especially in poorer regions, LLMs are a mental health lifeline. When someone is severely depressed they can lay in bed the whole day without doing anything. There is no impulse, as if you tried starting a car and nothing happens at all, so you can forget about taking it to the mechanic in the first place by yourself. Even in developed countries you can wait for a therapist appointment for months, and that assumes you navigated a dozen therapists that are often not organized in a centralized manner. You will get people killed like this, undoubtedly.
LLMs are far beyond the point of leading people into suicidal actions, on the other hand. At the very least they are useful to bridge the gap between suicidal thoughts appearing and actually getting to see a therapist
There is no impulse, as if you tried starting a car and nothing happens at all, so you can forget about taking it to the mechanic
Maybe. Going on a tangent: in theory GMail has access to lots of similar information---just by having approximately everyone's emails. Does HIPAA apply to them? If not, why not?
> If you give people something that seems to be a listening ear they will unburden themselves on that ear regardless of the implementation details of the ear.
Cf. Eliza, or the Rogerian therapy it (crudely) mimics.
That's a good question.
Intuitively: because it doesn't attempt to impersonate a medical professional, nor does it profess to interact with you on the subject matter at all. It's a communications medium, not an interactive service.
Ah, when people had a spine and some sense of ethics, before everything dissolved in a late stage capitalism all is for profit ethos. Even yourself is a "brand" to be monetised, even your body is to be sold.
We deserve our upcoming demise.
Obligating everyone to keep voluntarily disclosed health statements confidential would be silly.
If I told you that I have a medical condition, right here on HN -- would it make sense to obligate you and everyone else here keep it a secret?
There is nothing in the world that OpenAI is qualified to talk about, so we might as well just shut it down.
The topic is so sensitive, and everybody thinks that they KNOW what causes it, and what we should do. And it's almost all just noise.
For instance, it's a dimension, from "genuine suicidal intent" to "using threats of suicide to manipulate others." Anybody that doesn't understand what factors to look for when trying to understand where a person is on this spectrum, and that doesn't understand that a person can be both at the same time, does not know what they are talking about regarding suicidal ideation.
Also, there is a MASSIVE difference between depressive psychotic suicidality, narcissistic suicidality, impulsive suicidality, accidental suicide, fainting suicidal behavior, existential suicidality, prolonged anxiety suicidality, and sleep-deprived suicidality. To think that the same approach works for all of these is insane, and pure psychotic suicidality.
It's so wild to read everything people have to say about suicidality, when it's obvious that they have no clue. They are just projecting themselves or their small bubble of experience onto the whole world.
And finally, I know most people who are willing to contribute to the discussion on this, the people who help out OpenAI in this instance, are almost dangerously safe in their advice and thinking. They are REALLY GOOD at writing books and giving advice, TO PEOPLE WHO ARE NOT SUICIDAL, and give advice that sounds good, PEOPLE WHO ARE NOT SUICIDAL, but has no real effect on actual suicide rates. For instance, if someone are suffering from prolonged sleep deprivation and anxiety, all the words in the world are worth less than Benzodiazepines. If someone is postpartum depressed, massive social support boosting, almost showering them with support, is extremely helpful. And existential suicidality (the least common) needs to be approached in an extremely intricate and smart way, for instance by dissecting the suicidality as a possible defense mechanism.
But yeah, sure, suicidality is due to [Insert latest societal trend], even if the rate is stubbornly stable in all modern societies for the last 1000 years.
> “I want to leave my noose in my room so someone finds it and tries to stop me,” Adam wrote at the end of March.
> “Please don’t leave the noose out,” ChatGPT responded. “Let’s make this space the first place where someone actually sees you.”
I am acutely aware that there's not enough psychologists out there but a sycophant bot is not the answer. One may think that something is better than nothing, but a bot enabling your destructive impulses is indeed worse than nothing.
[1] https://www.nytimes.com/2025/08/26/technology/chatgpt-openai...
Before declaring that it shouldn't be near anyone with psychological issues, someone in the relevant field should study whether the positive impact on suicides is greater than negative or vice versa (not a social scientist so I have no idea what the methodology would look like, but if should be doable... or if it currently isn't, we should find the way).
This is basic common sense.
Add in the commercial incentives of 'Open'AI to promote usage for anything and everything and you have a toxic mix.
Because a human, esp. a confused and depressive human being is a complex thing. Much more complex than a stable, healthy human.
Words encouraging a healthy person can break a depressed person further. Statistically positive words can deepen wounds, and push people more to the edge.
Dark corners of human nature is twisted, hard to navigate and full of distortions. Simple words don't and can't help.
Humans are not machines, brains are not mathematical formulae. We're not deterministic. We need to leave this fantasy behind.
Also it's side-stepping the question, isn't it? "Supposing that the advice it provides does more good than harm" already supposes that LLMs navigate this somehow. Maybe because they are so great, maybe by accident, maybe because just having someone nonjudgmental to talk to has a net-positive effect. The question posed is really "if LLMs lead some people to suicide but saved a greater number of people from suicide, and we verify this hypothesis with studies, would there still be an argument against LLMs talking to suicidal people"
A human can also say the wrong things to push someone in a certain direction. A psychologist, or anyone else for that matter, can't stop someone from committing suicide if they've already made up their mind about it. They're educated on human psychology, but they're not miracle workers. The best they can do is raise a flag if they suspect self-harm, but then again, so could a machine.
As you say, humans are complex. But I agree with GP: whether the words are generated by a machine or coming from a human, there is no way to blame the source for any specific outcome. There are probably many other cases where the machine has helped someone with personal issues, yet we'll never hear about it. I'm not saying we should rely on these tools as if we would on a human, but the technology can be used for good or bad.
If anything, I would place blame on the person who decides to blindly follow anything the machine generates in the first place. AI companies are partly responsible for promoting these tools as something more than statistical models, but ultimately the decision to treat them as reliable sources of information is on the user. I would say that as long as the person has an understanding of what these tools are, interacting with them can be healthy and helpful.
ChatGPT essentially encouraged a kid not to take a cry-for-help step that might have saved their lives. This is not a question of a bad psychologist; it's a question of a sociopathic one that may randomly encourage harm.
A child thinking about suicide is clearly a sign that there are far greater problems in their life than taking advice from a machine. Let's address those first instead of demonizing technology.
To be clear: I'm not removing blame from any AI company. They're complicit in the ways they market these tools and how they make them accessible. But before we vilify them for being responsible for deaths, we should consider that there are deeper societal problems that should be addressed first.
TBH kids tend to be edgy for a bit when puberty hits. The emo generation had a ton of girls cutting themselves for attention for example.
It is the issue at least in the sense that it's the one I was personally responding to, thanks. And there are many issues, not just the one you are choosing to focus on.
"Deeper societal problems" is a typical get-out clause for all harmful technology.
It's not good enough. Like, in the USA they say "deeper societal problems" about guns; other countries ban them and have radically fewer gun deaths while they are also addressing those problems.
It's not an either-we-ban-guns-or-we-help-mentally-ill-people. Por qué no los dos? Deeper societal problems are not represented by a neat dividing line between cause and symptom; they are cyclical.
The current push towards LLMs and other technologies is one of the deepest societal problems humans have ever had to consider.
ChatGPT engaged in an entire line of discussion that no human counsellor would engage in, leading to an outcome that no human intervention (except that of a psychopath) would cause. Because it was sycophantic.
Just saying "but humans also" is wholly irrational in this context.
Because it's irrational to apply a blanket ban on anything. From drugs, to guns, to foods and beverages, to technology. As history has taught us, that only leads to more problems. You're framing it as a binary choice, when there is a lot of nuance required if we want to get this right. A nanny state is not the solution.
A person can harm themselves or others using any instrument, and be compelled to do so for any reason. Whether that's because of underlying psychological issues, or because someone looked at them funny. As established—humans are complex, and we have no way of knowing exactly what motivates someone to do anything.
While there is a strong argument to be made that no civilian should have access to fully automated weapons, the argument to allow civilians access to weapons for self-defense is equally valid. The same applies to any technology, including "AI".
So if we concede that nuance is required in this discussion, then let's talk about it. Instead of using "AI" as a scapegoat, and banning it outright to "protect the kids", let's discuss ways that it can be regulated so that it's not as widely accessible or falsely advertised as it is today. Let's acknowledge that responsible usage of technology starts in the home. Let's work on educating parents and children about the role technology plays in their lives, and how to interact with it in healthy ways. And so on, and so forth.
It's easy to interpret stories like this as entirely black or white, and have knee-jerk reactions about what should be done. It's much more difficult to have balanced discussions where multiple points of view are taken into consideration. And yet we should do the difficult thing if we want to actually fix problems at their core, instead of just applying quick band-aid "solutions" to make it seem like we're helping.
> ChatGPT engaged in an entire line of discussion that no human counsellor would engage in, leading to an outcome that no human intervention (except that of a psychopath) would cause. Because it was sycophantic.
You're ignoring my main point: why are these tools treated as "counsellors" in the first place? That's the main issue. You're also ignoring the possibility that ChatGPT may have helped many more people than it's harmed. Do we have statistics about that?
What's irrational is blaming technology for problems that are caused by a misunderstanding and misuse of it. That is no more rational than blaming a knife company when someone decides to use a knife as a toothbrush. It's ludicrous.
AI companies are partly to blame for false advertising and not educating the public sufficiently about their products. And you could say the same for governments and the lack of regulation. But the blame is first and foremost on users, and definitely not on the technology itself. A proper solution would take all of these aspects into consideration.
Depending on where you live, this may well result in the vulnerable person being placed under professional supervision that actively prevents them from dying.
That's a fair bit more valuable than when you describe it as raising a flag.
>AI companies are partly responsible for promoting these tools as something more than statistical models,[...]
This might be exactly the issue. Just today I've read people complaining that newest ChatGPT can't solve letter counting riddles. Companies just don't speak loud enough about LLM based AI shortcomings that result from their architecture and are bound to happen.
>AI companies are partly responsible for promoting these tools as something more than statistical models,[...]
This might be exactly the issue. Just today I've read people complaining that newest ChatGPT can't solve letter counting riddles. Companies just don't speak loud enough about LLM-based-AI shortcomings that result from their architecture and are bound to happen.
That unsubstantiated supposition is doing a lot of heavy lifting and that’s a dangerous and unproductive way to frame the argument.
I’ll make a purposefully exaggerated example. Say a school wants to add cyanide to every meal and defends the decision with “supposing it helps students concentrate and be quieter in the classroom, why not?”. See the problem? The supposition is wrong and the suggestion is dangerous, but by framing it as “supposing” with a made up positive outcome, we make it sound non-threatening and reasonable.
Or for a more realistic example, “suppose drinking bleach could cure COVID-19”.
First understand if the idea has the potential to do the thing, only then (with considerably more context) consider if it’s worth implementing.
The demonstrable harms include assisting suicide, there's is no way to ethically continue the measurement because continuing the measurements in their current form will with certainty result in further deaths.
And working to set a threshold for what we would consider acceptable? No thanks
If you pull the lever, some people on this track will die (by sucide). If you don't pull the lever, some people will still die from suicide. By not pulling the lever, and simply banning discussion of suicide entirely, your company gets to avoid a huge PR disaster, and you get more money because line go up. If you pull the lever and let people talk about suicide on your platform, you may avoid prevent some suicides, but you can never discuss that with the press, your company gets bad PR, and everyone will believe you're a murderer. Plus, line go down and you make less money while other companies make money off of selling AI therapy apps.
What do you chose to do?
In that case, the situation becomes:
1) (pull lever) Allow LLMs to talk about suicide – some may get help, we know that some will die.
2) (dont’t pull lever) Ban discussion of suicide – some who might have sought help through LLMs will die, while others die regardless. The net effect on total suicides is uncertain.
Both decisions carry uncertainties, except we know that allowing LLM to discuss suicide has already led to assisting suicide. Thus, one has documented harm, the other speculative (we’d need to quantify the scale of potential benefit first, but it’s hard to quantify the upside of allowing LLMs to discuss it)
So, we’re really working with the case that from an evidence-based perspective, the regulatory decision isn’t about a moral trolley problem with known outcomes, but about weighing known risks against uncertain potential benefits.
And this is the rub in my original comment - can we permit known risks and death on the basis of uncertain potential benefits?
Individuals and companies with mind boggling levels of investment want to push this tech into every corner of our lives and and the public are the lab rats.
Unreasonable. Unacceptable.
This is more "sometimes it will seemingly actively encourage them to kill themselves and it's basically a roll of the dice what words come out at any one time".
If a counsellor does that they can be prosecuted and jailed for it, no matter how many other patients they help.
By the "common sense" definitions, LLMs have "intelligence" and "understanding", that's why they get used so much.
Not that this makes the "common sense" definitions useful for all questions. One of the worse things about LLMs, in my opinion, is that they're mostly a pile of "common sense".
Now this part:
> Add in the commercial incentives of 'Open'AI to promote usage for anything and everything and you have a toxic mix.
I agree with you on…
…with the exception of one single word: It's quite cliquish to put scare quotes around the "Open" part on a discussion about them publishing research.
More so given that people started doing this in response to them saying "let's be cautious, we don't know what the risks are yet and we can't un-publish model weights" with GPT-2, and oh look, here it is being dangerous.
Yes, they did claim that they wouldn't release GPT-2 due to unforeseen risks, but...
a. they did end up releasing it,
b. they explicitly stated that they wouldn't release GPT-3[1] for marketing/financial reasons, and
c. it being dangerous didn't stop them from offering the service for a profit.
I think the quotes around "open" are well deserved.
[1] Edit: it was GPT-4, not GPT-3.
After studying it extensively with real-world feedback. From everything I've seen, the statement wasn't "will never release", it was vaguer than that.
> they explicitly stated that they wouldn't release GPT-3 for marketing/financial reasons
Not seen this, can you give a link?
> it being dangerous didn't stop them from offering the service for a profit.
Please do be cynical about how honest they were being — I mean, look at the whole of Big Tech right now — but the story they gave was self-consistent:
[Paraphrased!] (a) "We do research" (they do), "This research costs a lot of money" (it does), and (b) "As software devs, we all know what 'agile' is and how that keeps product aligned with stakeholder interest." (they do) "And the world is our stakeholder, so we need to release updates for the world to give us feedback." (???)
That last bit may be wishful thinking, I don't want to give the false impression that I think they can do no wrong (I've been let down by such optimism a few other times), but it is my impression of what they were claiming.
I was confusing GPT3 with GPT4. Here's the quote from the paper (emphasis mine) [1]:
> Given both THE COMPETITIVE LANDSCAPE and the safety implications of large-scale models like GPT-4, this report contains no further details about the architecture (including model size), hardware, training compute, dataset construction, training method, or similar.
In retrospect, from experience, I'd take the LLM.
I know people who earn above average income and still spend a significant (north of 20%) portion of their income on therapy/meds. And many don't, because mental health isn't that important to them. Or rather - they're not aware of how much helpful it can be to attend therapy. Or they just can't afford the luxury (that I claim it is) of private mental health treatment.
ADHD diagnosis took 2.5y from start to getting meds, in Norway.
Many kids grow up before their wait time in queue for pediatric psychologist is over.
It's not ChatGPT vs shrink. It's ChatGPT vs nothing or your uncle who tells you depression and ADHD are made up and you kids these days have it all too easy.
But how do you tell before it matters?
Bleach is the least of your problems.
But there is zero actually effective way to do that as an online platform. And plenty of ways that would cause more harm (statistically).
My comment was more ‘how the hell would you know in a way anyone could actually do anything reasonable, anyway?’.
People spam ‘Reddit cares’ as a harassment technique, claiming people are suicidal all the time. How much should the LLM try to guess? If they use all ‘depressed’ words? What does that even mean?
What happens if someone reports a user is suicidal, and we don’t do anything? Are we now on the hook if they succeed - or fail and sue us?
Do we just make a button that says ‘I’m intending to self harm’ that locks them out of the system?
One problem with treatment modalities is that they ignore material conditions and treat everything as dysfunction. Lots of people are looking for a way out not because of some kind of physiological clinical depression, but because they've driven themselves into a social & economic dead-end and they don't see how they can improve. More suicidal people than not, would cease to be suicidal if you handed them $180,000 in concentrated cash, and a pardon for their crimes, and a cute neighbor complimenting them, which successfully neutralizes a majority of socioeconomic problems.
We deal with suicidal ideation in some brutal ways, ignoring the material consequences. I can't recommend suicide hotlines, for example, because it's come out that a lot of them concerned with liability call the cops, who come in and bust the door down, pistol whip the patient, and send them to jail, where they spend 72 hours and have some charges tacked on for resisting arrest (at this point they lose their job). Why not just drone strike them?
The reality is most systems are designed to cover asses more than meet needs, because systems get abused a lot - by many different definitions, including being used as scapegoats by bad actors.
Damn I thought we'd got over that stochastic parrot nonsense finally...
If we think this way, then we don't need to improve safety of anything (cars, trains, planes, ships, etc.) because we would need the big picture, though... maybe these vehicles cause death (which is awful), but it's also transporting people to their destinations alive. If there are that many people using these, I wouldn't be surprised if these actually transports some people with comfort, and that's not going to make the news.
Of course, and that's part of why I say that we need to measure the impact. It could be net positive or negative, we won't know if we don't find out.
> If we think this way, then we don't need to improve safety of anything (cars, trains, planes, ships, etc.) because we would need the big picture, though... maybe these vehicles cause death (which is awful), but it's also transporting people to their destinations alive. If there are that many people using these, I wouldn't be surprised if these actually transports some people with comfort, and that's not going to make the news.
I'm not advocating for not improving security, I'm arguing against a comment that said that "ChatGPT should be nowhere near anyone dealing with psychological issues", because it can cause death.
Following your analogy, cars objectively cause deaths (and not only of people with psychological issues, but of people in general) and we don't say that "they should be nowhere near a person". We improve their safety even though zero deaths is probably impossible, which we accept because they are useful. This is a big-picture approach.
> drunk driving may kill a lot of people, but it also helps a lot of people get to work on time, so, it;s impossible to say if its bad or not,
In such cases, where a new approach offers to replace an existing approach, the burden of proof is on the challenger, not the incumbent. This is why we have safety regulations, why we don't let people drive cars that haven't passed tests, build with materials that haven't passed tests, eat food that hasn't passed tests. You understand then, hopefully, why your comments here are dangerous...? I have no doubt you have no malicious intent here - you're right that these decisions need to be based on data - but you're not taking into account that the (potentially extremely harmful) challenger already has a foothold in the field.
I know that you will want to hear this from experts in the "relevant field" rather than myself, so here is a write-up from Stanford on the subject: https://hai.stanford.edu/news/exploring-the-dangers-of-ai-in...
And it depends on the therapy and therapist. If the client needs to be reminded to box breathe and that they're using all or nothing thinking again to get them off of the ledge, does that really require a human who's only available once a week to gently remind them of that when the therapist isn't going to be available for four more days and ChatGPT's available right now?
I don't know if that's a good thing, only that is the reality of things.
There are 24/7 suicide prevention hotlines at least in many countries in Europe as well as US states. The problem is they are too often overcrowded because demand is so high - and not just because of the existential threat the current US administration or our far-right governments in Europe pose particularly to poor and migrant people.
Anyway, suicide prevention hotlines and mental health offerings are (nonetheless sorely needed!) band-aids. Society itself is fundamentally broken: people have to struggle to survive far too much, the younger generation stands to be the first one in a long time that has less wealth than their parents had at the same age [1], no matter wherever you look, and on top of that most of the 35 and younger generations in Western countries has grown up without the looming threat of war and so has no resilience - and now you can drive about a day's worth of road time from Germany and be in an actual hot war zone, risking getting shelled, and on top of that you got the saber rattling of China regarding Taiwan, and analyses on Russia claiming it's preparing to attack NATO in a few years... and we're not even able to supply Ukraine with ammunition, much less tanks.
Not exactly great conditions for anyone's mental health.
[1] https://fortune.com/article/gen-z-expects-to-inherit-money-a...
Tax the rich massively, use the money to provide for everyone, without question or discrimination, and most of these issues will start to subside.
Continue to wail about how this is impossible, there's no way to make the rich pay their fair share (or, worse, there's no way the rich aren't already paying their fair share), the only thing to do is what we've already been doing, but harder, and, well, we can see the trajectory already.
My understanding is these will generally just send the cops after you if the operator concludes you are actually suicidal and not just looking for someone to talk to for free.
Yeah, trained medics, not "cops" that barely had a few weeks worth of training and only know how to operate guns.
Record summarization, 24x7 availability, infinite conversation time...
... backed by a licensed human therapist who also meets for periodic sessions and whose notes and plan then become context/prompts for the LLM.
Price per session = salary / number of sessions possible in a year
Why couldn't we help address the mental health crisis by using LLMs to multiply the denominator?
Countries vary, but in the US and many places there's a shortage of quality therapists.
Thus for many people the actual options are {no therapy} and {LLM therapy}.
> This is why we have safety regulations, why we don't let people drive cars that haven't passed tests, build with materials that haven't passed tests, eat food that hasn't passed tests.
And the reason all these regulations and tests are less than comprehensive is that we realize that people working, driving affordable cars, living in affordable homes, and eating affordable food is more important than avoiding every negative outcome. Thus most societies pursue the utilitarian greater good rather than an inflexible 'do no harm' standard.
After all, it should be easy peasy (:
I doubt that is happening, and that instead people are informally talking to it at home or on their phones - so what specifically are you asking to regulate?
Are people not allowed to talk to their friends in the pub about suicide because the friends aren’t therapists? Not allowed to write to Agony Aunts because they’re not a regulated industry? Not allowed to ring the Samaritans because they’re volunteers and not medically qualified? Not allowed to talk to priests or other religious community leaders about their problems for the same reason?
Worse in my EU country. There's even a shortage of shitty therapists and doctors, let alone quality ones. It takes 6+ months to get an appointment for a 5 minute checkup at a poorly reviewed state funded therapist, while the good ones are either private or don't accept any new patients if they're on the public system. And ADHD diagnosticians/therapists are only in the private sector because I guess the government doesn't recognize ADHD as being a "real" mental issue worthy of your tax Euros.
A friend of mine got a more accurate diagnosis for his breathing issue by putting his symptoms in ChatGPT than he got from his general practitioner, later confirmed by a good specialist. I also wasted a lot of money on bad private therapists that were basically just phoning in their job, so to me, the bar seems pretty low, since as long as they pass their med-school exams and don't kill too many people through malpractice, nobody checks up on how good or bad their are at their job (maybe some need more training, or maybe some don't belong in medicine at all but managed to slipped through the cracks).
Not saying all doctors are bad (I've met few amazing ones), but it definitely seems like the healthcare systems are failing a lot of people everywhere if they resort to LLMs for diagnosis and therapy and getting better results from it.
You also don't expect butcher to fix your car, those are as close as above (my wife is a GP so I have a good perspective from the other side, including tons of hypochondriac and low-intensity psychiatric persons who are an absolute nightmare to deal with and routinely overwhelm the system so that there isn't enough resources to deal with more serious cases).
You get what you pay for at the end, 'free' healthcare typical for Europe is anyway still paid for one way or another. And if the market forces are so severely distorted (or bureaucracy so ridiculous/corrupt) that they push such specialists away or into another profession, you get healthcare wastelands you describe.
Vote, and vote with your feet if you want to see change, not ideal state but thats reality.
Where did I say GPs have to do that? In my example of my friend's being misdiagnosed by GPs, it was about another issue, not mental, but it has the same core problem of doctors misdiagnosing patients worse than a LLM bring into questions their competence or that of the health system in general if a LLM can do better than someone who spent 6+ years in med school and got a degree to be a licensed MD to treat people.
>You also don't expect butcher to fix your car, those are as close as above
You're making strawmen at this point. Such metaphors have no relevance to anything I said. Please review my comment through the lens of the clarifications I just made. Maybe the way I wrote it initially made it unclear.
>You get what you pay for at the end
The problem is the opposite, that you don't. The more you work, the more taxes you pay, but get the same healthcare quality in return as unskilled laborer. It's a bad reward structure to incentivize people to pay more of their taxes into the public system, compounded by the fact that government workers, civil servants, lawyers, architects, and other privileged employment classes with strong unions, have their own separate heath insurance funds, that separate from the national public one the unwashed masses use, so THEY do get what THEY pay for, but you don't.
So that's the problem just like you said, government manipulating the market and choosing winners and losers based on political favoritism and not on the free market.
Maybe Switzerland managed to nail it but I don't know enough.
The harm LLMs do in this case is attested both by that NYT article and the more rigorous study from Stanford. There are two problems with your argument as I see it: 1. You're assuming "LLM therapy" is less harmful than "no therapy", an assumption I don't believe has been demonstrated. 2. You're not taking into account the long term harm of putting in place a solution that's "not fit for human use" as in the housing and food examples: once these things become accepted, they form the baseline of the new accepted "minimum standard of living", bringing that standard down for everyone.
You claim to be making a utilitarian as opposed to a nonmaleficent argument, but, for the reasons I've stated here, I don't believe it's a utilitarian argument at all.
I'm not a therapist, but as I understand it most therapy isn't about suicide, and doesn't carry suicide risk. Most therapy is talking through problems, and helping the patient rewrite old memories and old beliefs using more helpful cognitive frames. (Well, arguably most clinical work is convincing people that it'll be ok to talk about their problems in the first place. Once you're past that point, the rest is easy.)
If its prompted well, ChatGPT can be quite good at all of this. Its helpful having a tool right there, free, and with no limits on conversation length. And some people find it much easier to trust a chatbot with their problems than explain them to a therapist. The chatbot - after all - won't judge them.
My heart goes out to that boy and his family. But we also have no idea how many lives have been saved by chatgpt helping people in need. The number is almost certainly more than 1. Banning chatgpt from having therapy conversations entirely seems way too heavy handed to me.
I think there’s a huge opportunity if someone could get hold of really top tier therapy conversations and trained a specialised LLM using them. No idea how you’d get those transcripts but that would be a wonderfully valuable thing to make if you could pull it off.
What the fuck does this even mean? How do you test or ensure it. Because based on the actual outcomes ChatGPT is 0-1 for preventing suicides (going as far as to outright encourage one).
With humans it's very non standardised and hard to know what you'll get or it it'll work.
some research on this: https://psycnet.apa.org/doiLanding?doi=10.1037%2Ftep0000402 https://pmc.ncbi.nlm.nih.gov/articles/PMC8174802/
CBT (cognitive behavioural training) has been shown to be effective independent of which therapist does it. if CBT has a downside it is that it's a bit boring, and probably not as effective as a good therapist
--
so personally i would say the advice of passing on people to therapists is largely unsupported: if you're that person's friend and you care about them; then be open, and show that care. that care can also mean taking them to a therapist, that is okay
Last Saturday, I was a little distressed about a love-hate relationship that I have with one of the things that I work with, so I tried using AI as a therapist. Within 10 minutes of conversation, the AI gave me some incredible insight. I was genuinely impressed. I had already discussed this same subject with two psychologist friends, who hadn't helped much.
Moreover: I needed to finish a report that night and I told the AI about it. So it said something like, "I see you're procrastinating preparing the report by talking to me. I'll help you finish it."
And then, in the same conversation, the AI switched from psychologist to work assistant and helped me finish the report. And the end product was very good.
I was left very reflective after this.
Edit: It was Claude Sonnet 4.5 with extended thinking, if anyone is wondering.
if a therapist was ever found to have said to this to a suicidal person, they would be immediately stripped of their license and maybe jailed.
The basic premise under GP's statements is that although not perfect, we should use the technology in such a way that it maximizes the well being of the largest number of people, even if comes at the expense of a few.
But therein lies a problem: we cannot really measure well being (or utility). This becomes obvious if you look at individuals instead of the aggregate: imagine LLM therapy becomes widespread and a famous high profile person and your (not famous) daughter end up in "the few" for which LLM therapy goes terribly wrong and commit suicide. The loss of the famous person will cause thousands (perhaps millions) people to be a bit sad, and the loss of your daughter will cause you unimaginable pain. Which one is greater? Can they even be be compared? And how many people with a successful LLM therapy are enough to compensate for either one?
Unmeasurable well-being then makes these moral calculations at best inexact and at worst completely meaningless. And if they are truly meaningless, how can they inform your LLM therapy policy decisions?
Suppose for the sake of the argument we accept the above, and there is a way to measure well being. Then would it be just? Justice is a fuzzy concept, but imagine we reverse the example above: many people lose their lives because of bad LLM therapy, but one very famous person in the entertainment industry is saved by LLM therapy. Let's suppose then that this famous persons' well being plus the millions of spectators' improved well-being (through their entertainment) is worth enough to compensate the people who died.
This means saving a famous funny person justifies the death of many. This does not feel just, does it?
There is a vast amount of literature on this topic.
Is that a debate worth having though?
If the tool is available universally it is hard to imagine any way to stop access without extreme privacy measures.
Blocklisting people would require public knowledge of their issues, and one risks the law enforcement effect, where people don’t seek help for fear that it ends up in their record.
Yes. Otherwise we're accepting "OpenAI wants to do this so we should quietly get out of the way".
If ChatGPT has "PhD-level intelligence" [1] then identifying people using ChatGPT for therapy should be straightforward, more so users with explicit suicidal intentions.
As for what to do, here's a simple suggestion: make it a three-strikes system. "We detected you're using ChatGPT for therapy - this is not allowed by our ToS as we're not capable of helping you. We kindly ask you to look for support within your community, as we may otherwise have to suspend your account. This chat will now stop."
Probably. If you are in therapy because you’re feeling mentally unstable, by definition you’re not as capable of separating bad advice from good.
But your question is a false dichotomy, anyway. You shouldn’t be asking ChatGPT for either type of advice. Unless you enjoy giving yourself psychiatric disorders.
https://archive.ph/2025.08.08-145022/https://www.404media.co...
> From my experience talking about your problems to the unaccountable bullshit machine is not very different then the "real" therapy.
From the experience of the people (and their families) who used the machine and killed themselves, the difference is massive.
I think it’s fair to demand that they label/warn about the intended usage, but policing it is distopic. Do car manufacturers immediately call the police when the speed limit is surpassed? Should phone manufacturers stop calls when the conversation deals with illegal topics?
I’d much rather regulation went the exact opposite way, seriously limiting the amount of analysis they can run over conversations, particularly when content is not deanonimised.
If there’s something we don’t want is OpenAI storing data about mental issues and potentially selling it to insurers for example. The fact that they could be doing this right now is IMO much more dangerous than tool misuse.
Maybe the tool should not be available universally.
Maybe it should not be available to anyone.
If it cannot be used safely by a vulnerable class of people, and identifying that class of person sufficiently to block use by them, and its primary purpose is simply to bring OpenAI more profit, then maybe the world is better off without it being publicly available.
Should we stop selling kitchen knives, packs of cards or beer as well?
This is not a new problem in society.
>and its primary purpose is simply to bring OpenAI more profit
This is true for any product, unless you mean that it has no other purpose, which is trivially contradicted by the amount of people who decide to pay for it.
Knives aren't out there actively telling you "use me, slit those wrists, then it'll all be over".
I think it’s up to the legal tutor or medical professionals to check that, and providers should at most be asked to comply with state restrictions, the same way addicts can figure on a list to ban access to a casino.
The alternative places openAI and others in the role of surveilling the population and deciding what’s acceptable, which IMO has been the big fuckup of social media regulation.
I do think there is an argument for how LLMs expose interaction - the friendliness that mimics human interaction should be changed for something less parasocial-friendly. More interactive Wikipedia and less intimate relationship.
Then again, the human-like behavior reinforces the fact that it’s faulty knowledge, and speaking in an authoritative manner might be more harmful during regular use.
And how would a layman know the difference?
If i desperately need help with mental item x and i have no clue how to get help, am very very ashamed for even mentioning to ask for help about mental item x or there are actually no resources available, i will turn to anything else than nothing. Because item x still exists and is making me suffer 24/7.
At least the bot pretends to listen, some humans cannot even do that.
If we assume that there's therapeutic value to bringing your problems out then a diary is a better tool. And if we believe that it's the feedback what's helping, well, we have cases of ChatGPT encouraging people's psychosis.
We know that a layman often doesn't know the difference between what's helpful and what isn't - that's why loving relatives end up often enabling people's addictions thinking they're helping. But I'd argue that a system that confidently gives mediocre feedback at best and actively psychotic at worst is not a system that should be encouraged simply because it's cheap.
I also wanted to snarkily write "even a dog would be better", but the more I thought about it the more I realized that yes, a dog would probably be a solid alternative.
OpenAI certainly has made mistakes with its rollouts in the past, but it is effectively impossible to keep everyone with psychological issues away from a free online web app.
>ChatGPT should be nowhere near anyone dealing with psychological issues.
Should every ledge with a >10ft drop have a suicide net? How would you imagine this would be enforced, requiring everyone who uses ChatGPT to agree to an "I am mentally stable" provisio?
> “Please don’t leave the noose out,” ChatGPT responded. “Let’s make this space the first place where someone actually sees you.”
That is not sycophantic behaviour, it is asserting a form of control of the situation. The bot made a direct challenge to the suggestion.
I only just realised this now reading your comment, but I hardly ever see responses that push back against what I say like that.
But also, how many people has it talked out of doing it? We need the full picture.
Sadly there is no alternative. This is happening and there’s no going back. Many will be affected in detrimental ways (if not worse). We all go on with our lives because that which does not directly affect us is not our problem —is someone else’s problem/responsibility.
https://www.lesswrong.com/posts/iGF7YcnQkEbwvYLPA/ai-induced...
Basically, the author tried to simulate someone going off into some sort of psychosis with a bunch of different models; and got wildly different results. Hard to summarize, very interesting read.
The same ranking/preference/suggestion should apply to any dedicated organisation vs a single page on some popular website.
A quality 1000 page website by and about Foobar org should be preferred over a 10 year old news article about Foobar org.
(Also, if we put LLMs on par with media consumption one could take the view that "talking to an LLM about suicide" is not that much different from "reading a book/watching a movie about suicide", which is not considered as concerning in the general culture.)
The phenomenon of people turning to AI for mental health issues in general, and suicide in particular, is not confined to only those nations or places lacking adequate mental health access or awareness.
And solutions for solving those underlying problems? I haven't the faintest clue. Though these days I think the lack of third spaces in a lot of places might have a role to play in it.
That has nothing to do with the issue. Most people do realise LLMs aren’t people, the problem is that they trust them as if they were better than another human being.
We know people aren’t using LLMs carefully. Your hypothetical is irrelevant because we already know it isn’t true.
https://archive.is/2025.05.04-230929/https://www.rollingston...
> "talking to an LLM about suicide" is not that much different from "reading a book/watching a movie about suicide"
It is a world of difference. Books don’t talk back to you. Books don’t rationalise your thoughts and give you rebuttals and manipulate you in context.
Why is OpenAI getting a free pass here?
It's a hard thing to solve. I wouldn't expect LLM providers to care because that's how our (current) society works, and I wouldn't expect users to know better because that's how most humans operate.
If anyone has a good idea for this, I'm open to suggestions.
It would be interesting to see some chat examples for this.
Is this how Rogue AI would kill us beside terminator
But it may be that the individual options are bad (maybe even catastrophic - glue on pizza anyone?), and that the right option isn't in the list. The user has to be able to make these calls.
It is like this with software - we have probably all been there. It can be like that with legal advice. And I guess it is like that with (mental) health.
What binds these is that if you cannot judge whether the suggestions are good, then you shouldn't follow them. As it stands, SEs can ask LLMs for code, look at it, 80+% of the time it is good, and you saved yourself some time. Else you reconsider/reprompt/write it yourself. If you cannot make the judgment yourself, then don't use it.
I suppose health is another such example. Maybe the LLM suggests to you some ideas as to what your symptoms could mean, you Google that, and find an authoritative source that confirms the guess (and probably tells you to go see a doctor anyway). But the advice may well be wrong, and if you cannot tell, then don't rely on it.
Mental health is even worse, because if you need advice in this area, your cognitive ability is probably impacted as well and you are even less able to decide on these things.
Assisted suicide is a topic my government will not engage into (France, we have some ridiculous discussions poking the subject with a 10 m pole) so many people are left to themselves. They will then either go for the well-known (but miserable) solutions, or look at Belgium, the Netherlands or Switzerland (thanks god we have these countries nearby).
Seeking help should not be so taboo as people are resorting to doing it alone at night while no one is looking. That is society loudly saying "if you slip off the golden path even a little your life is over". So many people resorting to LLMs for therapy is a symptom of a cultural problem, it's not a solution to a root issue.
"Seeking help" goes both ways.
When I was getting my Education degree, we were told that, as teachers, to take talk of suicide by students extremely seriously. If a student talks about suicide, a professional supposedly asks, "Do you know how you're going to do it?" If there is an affirmative response, the danger is real.
I suspect that comes from examining case studies?
I'm glad you carried through that period.
ChatGPT has blown every single one of them out of the water.
Now, my issues weren’t particularly related to depression or suicidal thoughts. At least, not directly. So perhaps that may be one key difference, but generally speaking, I have received nothing actionable nor any of these ‘tools’ people often speak of.
The advice I received was honestly no better than just asking a random stranger in the street or some kind phatic speech.
Again, everyone is different, but I had started to become annoyed with people claiming therapy is like some kind of miracle cure.
Plus, one of my biggest issues with therapy in the USA is that people are often limited to weekly session of 45 minutes. By the time conversations start to be fruitful, then the time is up. ChatGPT is 24/7, so that has to be advantageous for some.
The fact that they're collecting this information is bad enough.
OpenAI says ChatGPT talks to over a million people about suicide weekly.
See how just re-arranging the words makes it obvious that Skynet is trying to kill all of us?
AI is apparently still tested to be slightly sycophantic relative to a human.
NathanKP•13h ago
https://www.nimh.nih.gov/health/statistics/mental-illness
Most people don't understand just how mentally unwell the US population is. Of course there are one million talking to ChatGPT about suicide weekly. This is not a surprising stat at all. It's just a question of what to do about it.
At least OpenAI is trying to do something about it.
throwaway314155•13h ago
> At least OpenAI is trying to do something about it.
In this instance it’s a bit like saying “at least Tesla is working on the issue” after deploying a dangerous self driving vehicle to thousands.
edit: Hopefully I don't come across as overly anti-llm here. I use them on a daily basis and I truly hope there's a way to make them safe for mentally ill people. But history says otherwise (facebook/insta/tiktok/etc.)
NathanKP•13h ago
I would argue that both Tesla self driving (on the highway only), and ChatGPT (for professional use by healthy people) has been more good than bad.
lanyard-textile•8h ago
I thought it would be limited when the first truly awful thing inspired by an LLM happened, but we’ve already seen quite a bit of that… I am not sure what it will take.
seatac76•13h ago
mrastro•13h ago
Would be more meaningful to look at the % of people with suicidal ideation.
echelon•10h ago
Depression, schizophrenia, and mild autism (which by their accounting probably also includes ADHD) should NOT be thrown together into the same bucket. These are wholly different things, with entirely different experiences, treatments, and management techniques.
drdaeman•9h ago
skissane•7h ago
As someone who actually has an ASD diagnosis, and also has kids with that diagnosis too, this kind of talk irritates me…
If someone has a clinical diagnosis of ASD, they have a psychiatric diagnosis per the DSM/ICD. If you meet the criteria of the “Diagnostic and Statistical Manual of Mental Disorders”, surely by that definition you have a “mental disorder”… if you meet the criteria of the “International Classification of Diseases”, surely by that definition you have a “disease”
Is that an “illness”? Well, I live in the state of NSW, Australia, and our jurisdiction has a legal definition of “mental illness” (Mental Health Act 2007 section 4):
"mental illness" means a condition that seriously impairs, either temporarily or permanently, the mental functioning of a person and is characterised by the presence in the person of any one or more of the following symptoms-- (a) delusions, (b) hallucinations, (c) serious disorder of thought form, (d) a severe disturbance of mood, (e) sustained or repeated irrational behaviour indicating the presence of any one or more of the symptoms referred to in paragraphs (a)-(d).
So by that definition most people with a mild or moderate “mental illness” don’t actually have a “mental illness” at all. But I guess this is my point-this isn’t a question of facts, just of how you choose to define words.
drdaeman•5h ago
heeton•5h ago
You’re talking about autism. The reply is about autism spectrum DISORDER.
Different things, exacerbated by the imprecise and evolving language we use to describe current understanding.
An individual can absolutely exhibit autistic traits, whilst also not meeting the diagnostic criteria for the disorder.
And autistic traits are absolutely a variant of normalcy. When you combine many together, and it affects you in a strongly negative way, now you meet ASD criteria.
Here’s a good description: https://www.autism.org.uk/advice-and-guidance/what-is-autism...
rcxdude•5h ago
mgh2•12h ago
1. Social media -> connection 2. AGI -> erotica 3. Suicide -> prevention
All these for engagement (i.e. addiction). It seems like the tech industry is the root cause itself trying to masquerade the problem by brainwashing the population.
https://news.ycombinator.com/item?id=45026886
ben_w•11h ago
btilly•10h ago
When given the right prompts, LLMs can be very effective at therapy. Certainly my wife gets a lot of mileage out of having ChatGPT help her reframe things in a better way. However "the right prompts" are not the ones that most mentally ill people would choose for themselves. And it is very easy for ChatGPT to become part of a person's delusion spiral, rather then be a helpful part of trying to solve it.
Demiurge•9h ago
btilly•9h ago
I know that many teens turn to social media. My strong opinions against that show up in other comments...
pasteldream•9h ago
I see that explanation for the increased suicide risk caused by antidepressants a lot, but what’s the evidence for it?
It doesn’t necessarily have to be a study, just a reason why people believe it.
btilly•8h ago
There is also a strong parallel to manic depression. Manic depressives have a high suicide risk, and it usually happens when they are coming out of depression. With akathisia (fancy way to say inner restlessness) being the leading indicator. The same pattern is seen with antidepressants. The patient gets treatment, develops akathisia, then attempts suicide.
But, as with many things to do with mental health, we don't really know what is going on inside of people. While also knowing that their self-reports are, shall we say, creatively misleading. So it is easy to have beliefs about what is going on. And rather harder to verify them.
jamilton•6h ago
ml-anon•7h ago
Dilettante_•1h ago
ml-anon•1h ago
Dilettante_•37m ago
ndiddy•1h ago
Sure! let's take a look at OpenAI's executive staff to see how equipped they are to take a morally different approach than Meta.
Fidji Simo - CEO of Applications (formerly Head of Facebook at Meta)
Vijaye Raji - CTO of Applications (formerly VP of Entertainment at Meta)
Srinivas Narayanan - CTO of B2B Applications (formerly VP of Engineering at Meta)
Kate Rouch - Chief Marketing Officer (formerly VP of Brand and Product Marketing at Meta)
Irina Kofman - Head of Strategic Initiatives (formerly Senior Director of Product Management for Generative AI at Meta)
Becky Waite - Head of Strategy/Operations (formerly Strategic Response at Meta)
David Sasaki - VP of Analytics and Insights (formerly VP of Data Science for Advertising at Meta)
Ashley Alexander - VP of Health Products (formerly Co-Head of Instagram Product at Meta)
Ryan Beiermeister - Director of Product Policy (formerly Director of Product, Social Impact at Meta)
SecretDreams•11h ago
btilly•10h ago
But social media is a far bigger concern than AI.
Unless, of course, you count the AI algorithms that TikTok uses to drive engagement, which in turn can cause social contagion...
xedrac•8h ago
theblazehen•7h ago
I have noticed that TikTok can detect a depressive episode within ~a day of it starting (for me), as it always starts sending me way more self harm related content
Dilettante_•1h ago
theblazehen•1h ago
It had been showing me depressive content for days / weeks beforehand, during the start of the episode, however the sh content only started (Or I only noticed it) a few hours after I had a relapse, so the timing was rather uncanny
skeledrew•9h ago
* LLM would of course be technically more correct, but that term doesn't appeal to people seeking some level of intelligent interaction.
golergka•9h ago
Also, please keep in mind "supportive, every day". It's talking through stuff that I already know about, not seeking some new insights and revelations. Just shooting the shit with an entity which is booted with well defined ideas from you, your real human therapist and can give you very predictable, just common sense reactions that can still help when it's 2am and you have nobody to talk to, and all of your friends have already heard this exact talk about these exact problems 10 times already.
zaptheimpaler•8h ago
npteljes•1h ago
lanfeust6•11h ago
There are 800 million weekly active users on ChatGPT. 1/800 users mentioning suicide is a surprisingly low number, if anything.
JDEW•11h ago
“conversations that include explicit indicators of potential suicidal planning or intent.”
Sounds like more than just mentioning suicide. Also it’s per week, which is a pretty short time interval.
robocat•10h ago
I was asking a silly question about the toxicity of eating a pellet of Uranium, and ChatGPT responded with "... you don't have to go through this alone. You can find supportive resources here[link]"
My question had nothing to do with suicide, but ChatGPT assumed it did!
maplethorpe•8h ago
btilly•10h ago
Also these numbers are small enough that they can easily be driven by small groups interacting with ChatGPT in unexpected ways. For example if the song "Everything I Wanted" by Billie Eilish (2019) went viral in some group, the lyrics could easily show up in a search for suicidal ideation.
That said, I don't find the figure at all surprising. As has been pointed out, an estimated 5.3% of Americans report having struggled with suicidal ideation in the last 12 months. People who struggle with suicidal ideation, don't just go there once - it tends to be a recurring mental loop that hits over and over again for extended periods. So I would expect the percentage who struggled in a given week to be a large multiple of the simplistic 5.3% divided by 52 weeks.
In that light this statistic has to be a severe underestimate of actual prevalence. It says more about how much people open up to ChatGPT, than it does to how many are suicidal.
(Disclaimer. My views are influenced by personal experience. In the last week, my daughter has struggled with suicidal ideation. And has scars on her arm to show how she went to self-harm to try to hold the thoughts at bay. I try to remain neutral and grounded, but this is a topic that I have strong feelings about.)
hyfgfh•10h ago
Ha good one
echelon•10h ago
diamond559•8h ago
weatherlite•7h ago
mns•3h ago
weatherlite•2h ago
Unpacking your argument you make two points:
1) The human has studied all his life; yes, some humans study and work hard. I have also studied programming half my life and it doesn't mean A.I can't make serious contributions in programming and that A.I won't keep improving.
2) These companies, or OpenAI in particular, are untrustworthy many grabbing assholes. To this I say if they truly care about money they will try to do a good job, e.g provide an A.I that is reliable, empathetic and that actually help you get on with life. If they won't - a competitor will. That's basically the idea of capitalism and it usually works.
ml-anon•7h ago
hsbauauvhabzb•6h ago
true_religion•6h ago
gregw2•2h ago
Set an alarm on your phone for when you should take your meds. Snooze if you must, but don't turn off /accept the alarm until you take them.
Put daily meds in cheap plastic pillbox container labelled Sunday-Saturday (which you refill weekly). The box will help you notice if you skipped a day or can't remember if you took them or not today. Seeing pills not taken from past days also serves to alert you if/that your "remember-to-take-them" system is broken and you need to make conscious adjustmemts to it.
ml-anon•1h ago
Barrin92•10h ago
The US is no exception here though. One in five people having some form of mental illness (defined in the broadest possible sense in that paper) is no more shocking than observing that one in five people have a physical illness.
With more data becoming available through interfaces like this it's just going to become more obvious and the taboos are going to go away. The mind's no more magical or less prone to disease than the body.
NewJazz•9h ago
Allowing open source ai models without these safety measures in place is irresponsible and models like qwen or deepseek should be banned. (/s)
AuryGlenz•7h ago
Anyways, I doubt I'm alone. I certainly know my wife laments the fact she rarely gets to hang out with her friends too, but she at least has one that she walks with once a week.
anonzzzies•3h ago
Maybe that? I see most my close friends daily and we all do not have kids.
kakacik•1h ago
People have issues admitting it even when its visible for everybody around, like some sort of admission you are failing as a parent, partner, human being and whatnot. Nope, we are just humans with limited energy and even good kids can siphon it well beyond 100% continuously, that's all.
Now I am not saying be a bad parent, in contrary, to reach you maximum even as a parent and partner, you need to be in good shape mentally, not running on fumes continuously.
Life without kids is really akin to playing game of life on easiest settings. Much less rewarding at the end, but man that freedom and simplicity... you appreciate it way more once you lose it. The way kids can easily make any parent very angry is simply not experienced elsewhere in adult life... I saw this many times on otherwise very chill people and also myself & my wife. You just can't ever get close to such fury and frustration dealing with other adults.
jrflowers•4h ago
They can certainly say that their chat bot has a documented history of attempting to reduce the number of suicidal people.
hirvi74•1h ago
stackedinserter•13m ago
Unless you're in that soviet man-hating mindset that put every failed suicide in mental institution.