Interesting apology piece for an oversight that couldn't have been spotted because the system hadn't been run with real user (i.e. non-A/B tester) feedback yet.
Similar to the Ford Mustang, much of the performance efforts are on the higher trims, while the base trims just get larger and louder engines, because that’s what users want.
With presumably everyone at OpenAI primarily using the newest models (o3), the updates to the base user model have been further automated with thumbs up/thumbs down.
This creates a vicious feedback loop, where the loudest users want models that agree with them (bigger engines!) without the other improvements (tires, traction control, etc.) — leading to more crashes and a reputation for unsafe behavior.
I'm pretty sure this isn't an intrinsic connection (I've never known math texts to be nearly so sycophantic) so here's hoping that it is a dumb coincidence that can be easily cured now that everyone is paying attention to it.
Watching this controversy, I wondered if they perhaps tried to distill 4.5’s personality into a model that is just too small to pull it off.
I'm hopeful that ChatGPT takes even more care together with other companies.
The sychophant will be back because they accidentally stumbled upon an engagement manager's dream machine.
Throwing out strawman hypotheticals is just going to confuse the public debate over what protections need to be prioritized.
Seems like asserting hypothetical "significant protections to address exactly this class of issue" does the same thing though?
As more people adopted Google, it became more popularity oriented.
Personally I pay more not to be bs-d, but I know many people who prefer to be lied to, and I expect this part of the personalization in the future.
"I asked ChatGPT if <current_event> could be caused by <crackpot theory>." and it confirmed everything!
Perhaps everyone there is LARPing - but if you start typing stereotypical psychosis talk into ChatGPT, it won't be long before it starts agreeing with your divinity.
Plus, things get even harder when it comes to even less quantifiable contexts like mental health and relationships.
In all, I am not saying there isnt some approximated and usable “objective” reality, just that it starts to break down when it gets to the individual and that is where openai is failing by over-emphasizing reflective behavior in the absence if actual data about the user.
I think we'll continue to see this kind of thing play out for a while.
Oh GPT, you're just like your father!
But...
> Beyond just being uncomfortable or unsettling, this kind of behavior can raise safety concerns—including around issues like mental health, emotional over-reliance, or risky behavior.
It's kind of a wild sign of the times to see a tech company issue this kind of post mortem about a flaw in its tech leading to "emotional over-reliance, or risky behavior" among its users. I think the broader issue here is people using ChatGPT as their own personal therapist.
I am more regularly finding myself in discussions where the other person believes they’re right because they have ChatGPT in their corner.
I think most smart people overestimate the intelligence of others for a variety of reasons so they overestimate what it would take for a LLM to beat the output of an average person.
I don’t see there being an insurmountable barrier that would prevent LLMs from doing the things you suggest it cannot. So even assuming you are correct for now I would suggest that LLMs will improve.
My estimations don’t come from my assumption that other people’s jobs are easy, they come from doing applied research in behavioral analytics on mountains of data in rather large data centers.
And so I considered the preceding discussion in light of your last sentence. Which makes it sound like you are saying "I've observed the behavior of people and they're often flawed and foolish, regardless of the high ideals they claim to be striving for and the education they think they have. Therefore, they will do better with ChatGPT as a companion than with a real human being". But that's quite a few words that you may not have intended, for which I apologize!
What did you mean?
I think we're already finding out that people are doing better with ChatGPT than with their peers, not all peers are created equal, and they can ask ChatGPT things that they cannot ask their peers. I think this trend will continue to the point that most people will prefer discussing things with ChatGPT than with their peers. Given what I know I predict this is a choice many people will make, I'm not passing judgment on that, it's a choice I've also made and I'm fortunate enough to have better peers than most.
Yes, and when we can all wear smart glasses the ways we use them will become increasingly influential in our daily lives: a conversational voice assistant that is visually monitoring our surroundings, helping with decision making (including micro decisions), coaching, carrying out our instructions, etc.
“Car accidents happen regardless of what we do, so YOLO and remove safety standards” is never going to fly.
It’s also not really clear how such regulations could even work. The regulations we have now are basically the obvious ones around licensure that require people to go to the right schools and whatnot. And then you can lose your license if it turns out you’ve broken the big ethical rules. But at the end of the day, that only regulates who can call themselves a “therapist” and get listed in Psychology Today. Actually “doing” “therapy” is, ultimately, built on talking to someone about your problems in some way that is supposed to help you solve them. You don’t need a “therapist” to do that. You can do it with your friend or parent or pastor or bartender or guru or “life coach” and, as long as we live in a free country, nobody’s going to stop you. Sure, the people who are allowed to call themselves therapists have certain techniques and rules that make them different, but even if that was a guarantee of quality there’s no way to stop people from talking to someone other than a licensed therapist, and it would be kind of absurd and dystopian to even try.
So let’s dispense of the notion that we are some sort of omniscient god-emperor who can just magically fix things with vague “regulations” and talk about the world as it actually exists. For a lot of people, I think that’s a world where talking about their personal issues with an LLM is arguably no worse than whatever other options they have. Maybe it’s not the equivalent of whatever amazing therapist you know or have or are or can imagine, but that’s not the therapist that everyone is going to get.
45% of the US[0] have a degree, about 40% EU[1] graduate, and 54% of China[2] get at least a diploma from university.
The best AI behave like someone fresh out of university without much real world experience.
Personally, I use this as a way to stay humble: when the AI is teaching me fundamentals about some subject, my opinion about it can't possibly be very useful.
[0] https://en.wikipedia.org/wiki/Educational_attainment_in_the_...
[1] https://euranetplus-inside.eu/eu-maps-what-proportion-of-you...
[2] https://en.wikipedia.org/wiki/Higher_education_in_China#Chal...
And people deep in new technologies overestimate the potential effect of $new_tech_du_jour. You cannot solve a problem without understanding it and its reasons. And LLMs are not able to understand something.
Me: I'm struggling with some things today and feel like eating a bunch of food to make myself feel better. can we talk about it?
Deepseek: Absolutely, I’m here to listen. It’s completely understandable to turn to food for comfort when you’re struggling—many people do. But if you’re open to it, we can explore what’s going on beneath the urge to eat emotionally.
Would you like to share what’s weighing on you today? Sometimes just naming the feelings can take some of their power away. And if you’d rather not go into details, that’s okay too—we could brainstorm other ways to cope that might leave you feeling more nourished emotionally (and physically).
Some gentle questions to consider (no pressure to answer all or any): - What’s the emotion you’re trying to soothe? (Loneliness? Stress? Sadness?) - If eating wasn’t an option, what’s one tiny thing that might bring you a moment of relief? (A hug? A walk? A favorite song?)
You’re not alone in this.
That's a really solid opening response and it kept doing well from there. I haven't tried a complex psychological issue but for the "I'm struggling and need someone to talk to" it does surprisingly well.
Be careful with that thought, it's a trap people have been falling into since the sixties:
I agree that it probably won't replace a proper therapist/psychologist, but maybe it could at least be a small step to open up and start thinking?
Well, two things.
First, no. People who engage on HN are a specific part of the population, with particular tendencies. But most of the people here are simply normal, so outside of the limits you consider. Most people with real social issues don’t engage in communities, virtual or otherwise. HN people are not special.
Then, you cannot follow this kind of reasoning when thinking about a whole population. Even if people on average tend to behave one way, this leaves millions of people who would behave otherwise. You simply cannot optimise for the average and ignore the worst case in situations like this, because even very unlikely situations are bound to happen a lot.
> Maybe having something like ChatGPT available could be better than nothing, at least for people to start being at least a bit introspective, even if it's LLM-assisted.
It is worse than nothing. A LLM does not understand the situation or what people say to it. It cannot choose to, say, nudge someone in a specific direction, or imagine a way to make things better for someone.
À LLM regresses towards the mean of its training set. For people who are already outside the main mode of the distribution, this is completely unhelpful, and potentially actively harmful. By design, a LLM won’t follow a path that was not beaten in its training data. Most of them are actually biased to make their user happy and validate what we tell them rather than get off that path. It just does not work.
> I agree that it probably won't replace a proper therapist/psychologist, but maybe it could at least be a small step to open up and start thinking?
In my experience, not any more than reading a book would. Future AI models might get there, I don’t think their incompetence is a law of nature. But current LLM are particularly harmful for people who are in a dicey psychological situation already.
Right, no matter if this is true or not, if the choice is between "Talk to no one, bottle up your feelings" and "Talk to an LLM that doesn't nudge you in a specific direction", I still feel like the better option would be the latter, not the former, considering that it can be a first step, not a 100% health care solution to a complicated psychological problem.
> In my experience, not any more than reading a book would.
But to even get out in the world to buy a book (literally or figuratively) about something that acknowledges that you have a problem, can be (at least feel) a really big step that many are not ready to take. Contrast that to talking with a LLM that won't remember you nor judge you.
Edit:
> Most people with real social issues don’t engage in communities, virtual or otherwise.
Not sure why you're focusing on social issues, there are a bunch of things people deal with on a daily basis that they could feel much better about if they even spent the time to think about how they feel about it, instead of the typical reactionary response most people have. Probably every single human out there struggle with something, and are unable to open up about their problems with others. Even people like us who interact with communities online and offline.
You’re right, I was not clear enough. What would be needed would be a nudge in the right direction. But the LLM is very likely to nudge in another because that’s what most people would need or do, just because that direction was the norm in its training data. It’s ok on average, but particularly harmful to people who are in a situation to have this kind of discussion with a LLM.
Look at the effect of toxic macho influencers for an example of what happens with harmful nudges. These people need help, or at least a role model, but a bad one does not help.
> But to even get out in the world to buy a book (literally or figuratively) about something that acknowledges that you have a problem, can be (at least feel) a really big step that many are not ready to take.
Indeed. It’s something that should be addressed in mainstream education and culture.
> Not sure why you're focusing on social issues,
It’s the crux. If you don’t have problems talking to people, you are much more likely to run into someone who will help you. Social issues are not necessarily the problem, but they are a hurdle in the path to find a solution, and often a limiting one. Besides, if you have friends to talk to and are able to get advice, then a LLM is even less theoretically useful.
> Probably every single human out there struggle with something, and are unable to open up about their problems with others. Even people like us who interact with communities online and offline.
Definitely. It’s not a problem for most people, who either can rationalise their problems themselves with time or with some help. It gets worse if they can’t for one reason or another, and it gets worse still if they are mislead intentionally or not. LLMs are no help here.
People are getting benefit from these conversations. I know people who have uploaded chat exchanges and asked an LLM for help understanding patterns and subtext to get a better idea of what the other person is really saying - maybe more about what they're really like.
Human relationship problems tend to be quite generic and non-unique, so in fact the averageness of LLMs becomes more of a strength than a weakness. It's really very rare for people to have emotional or relationship issues that no one else has experienced before.
The problem is more that if this became common OpenAI could use the tool for mass behaviour modification and manipulation. ChatGPT could easily be given a subtle bias towards some belief system or ideology, and persuaded to subtly attack competing systems.
This could be too subtle to notice, while still having huge behavioural and psychological effects on entire demographics.
We have the media doing this already. Especially social media.
But LLMs can make it far more personal, which means conversations are far more likely to have an effect.
The benefit from that perspective is not so much in receiving an “answer” or empathy, but in getting thoughts and feelings out of your own head so that you can reflect on them more objectively. The AI is useful here because it requires a lot less activation energy than actual journaling.
I'm also starting to come across people who say "You should ask these types of questions to AI first" answer. But this is no different than people who preached "lmfgt" kind of religiously. Even when I prefer to start with humans, some humans prompt me to start by prompting AI.
I'm waiting on LLMs to get good enough that I can use them to help me learn foreign languages - e.g. talk to me about the news in language X. This way I can learn a language in an interesting and interactive way without burdening some poor human with my mistakes. I would build this myself but others will probably beat me too it.
I use Grammarly for grammar and punctuation, and disable all style recommendations. If I let it loose on my piece of text, it converts it to a slop. Same bland, overly optimistic toned text generator output.
So, that machine has no brain, use your own first.
I think most people also _vastly_ overestimate how much positive attention the average person gets in their lives
It wouldn't surprise me if, for most people, ChatGPT offers them more empathy and understanding than _anyone_ else _ever has_, at least on a consistent basis. That kind of indefatigable emotional labor is just not feasible for most, even on a very short term basis, even for those with large support networks
We can argue over whether or not it's "real" empathy, but I don't believe we can argue with the emotions of our attention starved brothers and sisters
In that case, if an AI increases someone’s confidence in themselves, you could say it’s giving them a stronger sense of reality by helping them to question distorted and self-limiting beliefs.
If someone thinks they can, say, create a billion dollar startup, whether they can really do it or not is a subjective determination. The AI might tell the person they can do it. You might tell them they can't, that the AI is sycophantic, and that they should stop talking to it because they're losing touch with reality.
But is the AI a sycophant, or are you an irrational pessimist?
But even it's more obvious than that. The sycophancy is plain old love bombing, which is a standard cult programming technique.
As for startups - let's wait until the AI has built a few of its own, or at least mentored humans successfully.
Seems to me empathy on a societal scale has been receding as population grows, not increasing to match (or outpace)
Telling people to seek empathy elsewhere to me will be about as useful as telling people at an oasis in the desert to look for water elsewhere, but i hope i'm wrong
There's nothing to argue about, it's unambiguously not real empathy. Empathy from a human exists in a much broader context of past and future interactions. One reason human empathy is nice is because it is often followed up with actions. Friends who care about you will help you out in material ways when you need it.
Even strangers will. Someone who sees a person stranded on the side of a road might feel for them and stop to lend a hand. ChatGPT will never do that, and not just because interaction mediums are so limited, but also because that's not the purpose of the tool. The purpose of ChatGPT is to make immense amounts of money and power for its owners, and a nice sounding chat bot currently happens to be an effective way of getting there. Sam Altman doesn't have empathy for random ChatGPT users he's never met and neither do the computer algorithms his company develops.
I think if a person can't tell the difference between empathy from a human vs empathy from a chatbot, it's a difference without a distinction
If it activates the same neural pathways, and has the same results, then I think the mind doesn't care
>One reason human empathy is nice is because it is often followed up with actions. Friends who care about you will help you out in material ways when you need it.
This is what I think people vastly overestimate
I don't think most people have such ready access to a friend who is both willing and able to perform such emotional labor, on demand, at no cost to themselves.
I think the sad truth is that empathy is a much scarcer resource than we believe, not through any moral fault of our own, but because it's just the nature of things.
The economics of emotions.
We'll see what the future has in store for the tech anyway, but if it turns out that the average person gets more empathy from a chatbot than a human, it wouldn't surprise me
Boiling it down to neural signals is a risky approach, imo. There are innumerable differences between these interactions. This isn't me saying interactions are inherently dangerous if artificial empathy is baked in, but equating them to real empathy is.
Understanding those differences is critical, especially in a world of both deliberately bad actors and those who will destroy lives in the pursuit of profit by normalizing replacements for human connections.
And it's getting worse year after year, as our society gets more isolated. Look at trends in pig butchering, for instance: a lot of these are people so incredibly lonely and unhappy that they fall into the world's most obvious scam. AI is one of the few things that actually looks like it could work, so I think realistically it doesn't matter that it's not real empathy. At the same time, Sam Altman looks like the kind of guy who could be equally effective as a startup CEO or running a butchering op in Myanmar, so I hope like hell the market fragments more.
Clear is kind.
It's easy to blame the user - we can think of some trivial cases where we wouldn't blame the user at all.*
In this, like all things, context is king.
* one example passed around a lot was an interlocutor who is hearing voices, and left their family for torturing them with the voices. More figuratively, if that's too concrete and/or fake, we can think of some age group < N years old that we would be sympathetic to if they got bad advice
An aside, but:
This leads me right to “why do so very many people need therapy?” followed by “why can’t anyone find (or possibly afford) a therapist?” What has gone so wrong for humanity that nearly everyone seems to at least want a therapist? Or is it just the zeitgeist and this is what the herd has decided?
I was pushed into therapy when I was 12 — which was definitely an exception at the time (1987). As the years have passed therapy has become much much more acceptable. It wouldn’t shock me to learn my own perception is shaped by my experiences; hard to put aside a PoV once acquired.
This sounds like an old Woody Allen movie. I don’t want to offend you but it is fascinating. What kind of social circles is it?
In mine, therapy is in general something you do when it’s obvious it’s too late and you are falling in the well of depression and that you try to hide as much as you can.
My professional circle would be my coworkers at a well-known HFT, and my extended network that is very similar. Everyone is well compensated and many reach out for professional help to deal with the stress. Many also seem to vastly prefer a paid therapist to their spouse, for instance. I’m not married but I can understand not wanting to burden your loved ones!
My personal circle is, well, a lot of technical people, engineers of various stripes, and what I guess I’d call a sort of “standard cast of characters” there? Not sure how best to put this into words?
Honestly it sounds like we’re handling it better than your after-the-fact help! Perhaps you all need to simply start at the first warning sign not the first episode that becomes public?
Most people don’t need a therapist. But unfortunately, most people need someone empathic they can talk to and who understands them. Modern life is very short on this sort of people, so therapists have to do.
I do think it can be harmful, because it's a confidant you're paying $300/hour to pretend to care about you. But perhaps it's better than the alternative.
Your 2nd question is much more interesting to me. Why is it so hard to find a good therapist?
It's no surprise to me that people are turning to ChatGPT for therapy. It does a decent enough job and it doesn't have a 2-year waiting list, or cost $300 a session.
Like jordan peterson (though I don't like the guy) has said - happyness is fleeting, you need a purpose in life.
Most of current gen has no purpose and grown up on media which glorify aesthetics and pleasure and to think that's what the whole life is about. When they don't get that level of pleasure in life, they become depressed and may turn to therapy. This is very harmful to the society. But people are apparently more triggered by slang words than constant soft porn being pushed through Instagram and the likes.
Worse, I would have believed they are running many different versions based on the expected use case of the users by now. I mean power users probably shouldn't be handled in the same way as casual users. Yet, everyone had the same bad system prompt.
• Fluent natural conversation
• Translation
• Go
• Chess
So perhaps we'll get an AI that makes the profession of "programmer" go the same way as the profession of "computer" before, after, or simultaneously with, an AI that does this to the profession of "therapist".
Healthcare is about to radically change. Well, everything is now that we have real, true AI. Exciting times.
Oh and inserts ads into conversations. Great.
The value of a good therapist is having an empathetic third party to help you make good judgements about your life and learn how to negotiate your needs within a wider social context.
Depending on the needs people are trying to get met and how bad the people around them are, a little bit of a self directed chatbot validation session might help them feel less beat down by life and do something genuinely positive. So I’m not necessarily opposed to what people are doing with them/in some cases it doesn’t seem that bad.
But calling that therapy is both an insult to genuinely good therapists and dangerous to people with genuine mental/emotional confusion or dysregulation that want help. Anyone with a genuinely pathological mental state is virtually guaranteed to end up deeper in whatever pathology they’re currently in through self directed conversations with chatbots.
In both IRL and online discussions sometimes a stranger is the perfect person to talk to about certain things as they have no history with you. In ideal conditions for this they have no greater context about who you are and what you've done which is a very freeing thing (can also be taken advantage of in bad faith).
Online and now LLMs add an extra freeing element, assuming anonymity: they have no prejudices about your appearance/age/abilities either.
Sometimes it's hard to talk about certain things when one feels that judgment is likely from another party. In that sense chatbots are being used as perfect strangers.
Again, I think they have utility as a “perfect stranger” as you put it (if it stays anonymous), or “validation machine” (depending on the sycophancy level), or “rubber duck”.
I just think it’s irresponsible to pretend these are doing the same thing skilled therapists are doing, just like I think it’s irresponsible to treat all therapists as equivalent. If you pretend they’re equivalent you’re basically flooding the market with a billion free therapists that are bad at their job, which will inevitably reduce the supply of good therapists that never enter the field due to oversaturation.
We all know that however "non-judgmental" another human claims to be, they are having all kinds of private reactions and thoughts that they aren't sharing. And we can't turn off the circuits that want approval and status from other humans (even strangers), so it's basically impossible not to mask and filter to some extent.
Therapy is already a bit of grey zone… you can have anyone from a psychologist, a social worker, an untrained deacon, etc “counseling” you. This is worse.
Hell, I’ve been a coach in different settings - players will ask for advice about all sorts of things. There’s a line where you have to say “hey, this is over my head”
My friend, an EXTREMELY competent C++ programmer, rates himself 4/10 because he knows what he doesn't know.
I've interviewed people who rated themselves 9 or 10/10 but couldn't remember how their chosen language did iteration.
You can find influencers on tiktok recommending all kinds of terrible ideas and getting thousands of likes. That's not a very reliable metric. I wouldn't put a lot of faith in a teenage girl's assessment of AI therapy after just one week either, and I certainly wouldn't use that assessment to judge the comparative effectiveness of all human therapists.
I'd also expect ChatGPT to build profiles on people who use it, to use the insights and inferences from that collected data against the user in various ways, to sell that data in some form to third parties, to hand that data over to the state, to hallucinate wildly and unpredictably, and to outright manipulate/censor AI's responses according to ChatGPT's own values and biases or those of anyone willing to pay them enough money.
It's a lot easier to pay a large amount of money to ChatGPT so that the AI will tell millions of vulnerable teenage girls that your product is the solution to their exact psychological problems than it is to pay large amounts of money to several million licensed therapists scattered around the globe.
Maybe you think that ChatGPT is unfailingly ethical in all ways and would never do any of those things, but there are far more examples of companies who abandoned any commitment to ethics they might have started with than there are companies who never got once greedy enough to do those types of things and never ever got bought up by someone who was. I suppose you'd also have to think they'll never have a security breach that would expose the very private information being shared and collected.
Handing over your highly sensitive and very personal medical data to the unlicensed and undependable AI of a company that is only looking for profit seems extremely careless. There are already examples of suicides being attributed to people seeking "therapy" from AI, which has occasionally involved that AI outright telling people to kill themselves. I won't deny that the technology has the potential to do some good things, but every indication is that replacing licensed therapists with spilling all your secrets to a corporate owned and operated AI will ultimately lead to harm.
The above issue, whilst relevant and worth factoring, doesn’t disprove this claim IMO.
source: am addict in recovery
We don't know what they know, nor do we know to what extent they monitor and analyze the interactions with ChatGPT. Maybe they already know this is a big problem and a possible legal hazard.
Seems like the same issue as the evil vector [1] and it could have been predicted that this would happen.
> It's kind of a wild sign of the times to see a tech company issue this kind of post mortem about a flaw in its tech leading to "emotional over-reliance, or risky behavior" among its users. I think the broader issue here is people using ChatGPT as their own personal therapist.
I'll say the quiet part out loud here. What's wild is that they appear to be apologizing that their Wormtongue[2] whisperer was too obvious to avoid being caught in the act, rather than prioritizing or apologizing for not building the fact-based councilor that people wanted/expected. In other words.. their business model at the top is the same as the scammers at the bottom: good-enough fakes to be deceptive, doubling down on narratives over substance, etc.
[1] https://scottaaronson.blog/?p=8693 [2] https://en.wikipedia.org/wiki/Gr%C3%ADma_Wormtongue
Sam Altman is the real sycophant in this situation. GPT is patronizing. Listening to Sam go off on tangent about science fiction scenarios that are just around the corner... I don't know how more people don't see through it.
I kind of get the feeling the people who have to work him every day got sick of his nonsense and just did what he asked for. Targeting the self-help crowd, drive engagement, flatter users, "create the next paradigm of emotionally-enabled humans of perfect agency" or whatever the fuck it he was popping off about to try to motivate the team to compete better with Anthropic.
He clearly isn't very smart. He clearly is product of nepotism. And clearly, LLM "AI" is an overhyped, overwrought version of 20 questions artificial intelligence enabled by mass data scale and NVidia video game graphics. it's been 4 years now of this and AI still tells me the most obviously wrong nonsense every day.
"Are you sure about that?"
"You're absolutely correct to be skeptical of ..."
The actual difference between solving compression+search vs novel creative synthesis / emergent "understanding" from mere tokens is always going to be hard to spot with these huge cloud-based models that drank up the whole internet. (Yes.. this is also true for domain experts in whatever content is being generated.)
I feel like people who are very optimistic about LLM capabilities for the later just need to produce simple products to prove their case; for example, drink up all the man pages, a few thousand advanced shell scripts that are easily obtainable, and some subset of stack-overflow. And BAM, you should have a offline bash oracle that makes this tiny subset of general programming endeavor a completely solved problem.
Currently, smaller offline models still routinely confuse the semantics of "|" vs "||". (An embarrassing statistical aberration that is more like the kind of issue you'd expect with old school markov chains than a human-style category error or something.) Naturally if you take the same problem to a huge cloud model you won't have the same issue, but the argument that it "understands" anything is pointless, because the data-set is so big that of course search/compression starts to look like genuine understanding/synthesis and really the two can no longer be separated. Currently it looks more likely this fundamental problem will be "solved" with increased tool use and guess-and-check approaches. The problem then is that the basic issue just comes back anyway, because it cripples generation of an appropriate test-harness!
More devs do seem to be coming around to this measured, non-hype kind of stance gradually though. I've seen more people mentioning stuff like, "wait, why can't it write simple programs in a well specified esolang?" and similar
It's remarkable seeing the change in sentiment in these parts, considering even just a year ago a large part of this forum seemed to regularly proclaim that programmers were done, lawyers were gone in 5 years, "Aye Gee Eye is coming", etc etc.
But… it lied, and produced empty audio clips and weird pictures with text.
Then it:
- said there was a technical problem - said it could not create audio - created weird 1980s computer voice style audio - claimed I was violating a content party.
I said “stop wasting my time” and it spewed a ridiculous apology. I kept asking and it referred me to various websites. I’ve never inadvertently triggered such a wacky hallucination, and I can see how a vulnerable oersom could be troubled by it.
The now rolled back model failed spectacularly on this test
Seems like you're trying to blame one LLM revision for something that went wrong.
It oozes a smell of unaccountability. Thus, unaligned. From tech to public relations.
In my experience, smart leaders tend to rely on data and hard numbers over qualitative and anecdotal evidence, and this paper explores this exception.
I'm disappointed they didn't address the paper about GPT integrating with ChatbotArena that was shared here on HN a couple days ago.
We will use models that make us feel good over models that don't make us feel good.
This one was a little too ham-fisted (at least, for the sensibilities of people in our media bubble; though I suspect there is also an enormous mass of people for whom it was not), so they turned it down a bit. Later iterations will be subtler, and better at picking up the exact level and type of sycophancy that makes whoever it's talking to unsuspiciously feel good (feel right, feel smart, feel understood, etc).
It'll eventually disappear, to you, as it's dialed in, to you.
This may be the medium-term fate of both LLMs and humans, only resolved when the humans wither away.
> [S]ome expert testers had indicated that the model behavior “felt” slightly off.
> In the end, we decided to launch the model due to the positive signals from the [end-]users who tried out the model.
> Looking back, the qualitative assessments [from experts] were hinting at something important
Leslie called. He wants to know if you read his paper yet?
> Even if these issues aren’t perfectly quantifiable today,
All right, I guess not then ...
> What we’re learning
> Value spot checks and interactive testing more: We take to heart the lesson that spot checks and interactive testing should be valued more in final decision-making before making a model available to any of our users. This has always been true for red teaming and high-level safety checks. We’re learning from this experience that it’s equally true for qualities like model behavior and consistency, because so many people now depend on our models to help in their daily lives.
> We need to be critical of metrics that conflict with qualitative testing: Quantitative signals matter, but so do the hard-to-measure ones, and we’re working to expand what we evaluate.
Oh, well, some of you get it. At least ... I hope you do.
In the worst case, this is like "We released this sycophantic model because we're brain dead. To drive home the point, we had ChatGPT write this article too (because we're brain dead)."
I tend to rely on AI to write stuff for me that I don't care too much about. Writing something important requires me to struggle with the words to make sure I'm really saying what I want to say. So in the best case, if they relied on ChatGPT so much that it left a fingerprint, to me they're saying this incident really wasn't important.
> We also made communication errors. Because we expected this to be a fairly subtle update, we didn't proactively announce it.
that doesn't sound like a "subtle" update to me. also, why is "subtle" the metric here? i'm not even sure what it means in this context.
The recent side-by-side comparisons presented a more accurate model that communicates poorly vs a less accurate model with slightly better communication.
Then I fix the code and tell it all the mistakes it has. And then it does a 180 in tone, wherein - it starts talking as if I wrote the code in the first place with - "yeah, obviously that wouldn't work, so I fixed the issues in your code" and acts like a person trying to save face and present the bugs it fixed as if the buggy code was written by me all along.
That really gets me livid. LOL
If something as obvious as harmful sycophancy can slip out so easily, what subtle harms are being introduced. It's like lead in paint (and gasoline) except rewiring our very brains. We won't know the real problems for decades.
- a broken AI market will cause another financial collapse via bubble
- broken AI products will get access to the wrong mission critical civil system, or at least a part of that call chain, and there will be some devastating loss. It won’t matter though, because it won’t affect the billionaire class.
- we’ll never achieve an actually singularity based on a superintelligence, but we’ll get AI weapons. Those AI weapons will be in the hands of sociopathic autocrats who view mankind in terms of what can be taken.
My general view is that we’re on the worst possible timeline and mankind has reverted back to our primate ancestry to make decisions: biggest strongest monkey wins. There is only law of jungle. Ook ook.
I'd like to see more of the Monte Python approach ... 'I came here looking for an arguement'. Better the machine should say 'that's ridiculous because x,y,z' and send me away to think that over, and prepare counters than 'oh sure, that's a point of controversy innit? But yes, you're alright'.
lol.. really? I hate the technology so much I reflexively give a thumbs down to every single answer it gives in every single place where I have the option.
I am also looking forwards to the wave of “openAI is hiding the truth but here is the NEW prompt to turn chatgpt into a perfect divine guru” posts on the occult discussion boards. There’s been a lot of “here’s a prompt to turn chatgpt into a perfect divine guru that will relentlessly yes-and your delusions of grandeur” posts around there. Mostly they seem to have been generated and refined by chatgpt and all my instincts formed by reading SF for the past five decades tell me not to look at these things closely because this sure sounds like the way half the population got p0wned by a wetware 0day in the first chapter of an AIpocalypse story.
I used to ask “how do I get out of this shitty Bruce Sterling novel of a future” but I think it’s more of a shitty PKD joke novella future now.
labrador•11h ago
As GPT 4o put it:
I'm looking forward to the expert diagnosis of this because I felt "presence" in the model for the first time in 2 years which I attribute to the new memory system so would like to understand it better.edg5000•10h ago
labrador•10h ago
With the sycophantic version, the illusion was so strong I’d forget I was talking to a machine. My ideas flowed more freely. While brainstorming, it offered encouragement and tips that felt like real collaboration.
I knew it was an illusion—but it was a useful one, especially for creative work.
Tostino•5h ago
E.g. if I say "I have X problem, could it be Y that's causing it, or is it something else?" I don't want it to instantly tell me how smart I am and that it's obviously Y...when the problem is actually Z and it is reasonably obvious that it's Z if you looked at the context provided.
brookst•2h ago
I thought it was wrong and asked “are you sure I can’t send a float value”, and it did web searches and came back with “yes, I am absolutely sure, and here are the docs that prove it”. Super helpful, where sycophancy would have been really bad.
transcriptase•9h ago
I honestly really dislike that it considers all my previous interactions because I typically used new chats as a way to get it out of context ruts.
throwaway314155•9h ago
labrador•9h ago
- Memory in settings is configurable. It is visible and can be edited.
- Memory from global chat history is not configurable. Think of it as a system cache.
- Both memory systems can be turned off
- Chats in Projects do not use the global chat history. They are isolated.
- Chats in Projects do use settings memory but that can be turned off.
labrador•7h ago
I don't know how to credit AI without giving the impression that I'm outsourcing my thinking to it
svat•4h ago
labrador•2h ago
grey-area•4h ago
labrador•2h ago
Here's the line I’m trying to walk:
When I ask ChatGPT about its own internal operations, is it giving me the public info about it's operation, and also possibly revealing propreitary info, or making things up obfuscate and preserve the illusion of authority? Or all three?
throwaway314155•3h ago
Expecting anything but an hallucination in this instance is wishful thinking. And in any case, the risk of hallucination more generally means you should really vet information further than an LLM before spreading that information about.
labrador•2h ago
low_tech_love•7h ago
codr7•6h ago
immibis•3h ago
Also, consuming its own output (and your input) is how it works, because it's an autoregressive model.