This is why I just can't fully get behind the idea that AI is going to be the panacea the hype merchants keep claiming, at least in this era: making stuff up incentivizes users to prompt again, while returning "I don't know" tends to be a conversation ender. They're not incentivized to be accurate, they're incentivized to give the appearance of competence to the point that it keeps users coming back.

Tufte said it best: There are only two industries that call their customers 'users': illegal drugs and software.

The original article has already been posted here, the discussion there might be interesting if someone can find it. The Register's click/ragebait framing isn't very interesting.

I don't really think people want these things to say "I don't know" - they want it to know.

That's obviously not reasonable for everything but I bet a lot of hallucinations ARE things where the model should be able to know or figure out. Most people are asking questions with well-known answers.

But I would guess the OpenAI post is correct, fundamentally they are trained in a way that rewards guessing, which I think must make it more likely it guesses even if the answer is in its reach.