So it's not really hallucinating - it correctly represents "seahorse emoji" internally, but that concept has no corresponding token. lm_head just picks the closest thing and the model doesn't realize until too late.

Explains why RL helps. Base models never see their own outputs so they can't learn "this concept exists but I can't actually say it."

To confirm, I tried this in ChatGPT and it produced a flood of wrong answers and self corrections just like that, scrolling so quickly that I couldn't read it until it eventually stopped itself.

So what's at loggerheads here is:

* The LLM has strong and deep rooted belief in its knowledge (that a seahorse emoji exist).

* It attempts to express that concept using language (including emojis) but the language is so poor and inaccurate at expressing the concept that as it speaks it keeps attempting to repair.

* It is trained to speak until it has achieved some threshold at correctly expressing itself so it just keeps babbling until the max token threshold triggers.

I thought there was a yellow/pink seahorse emoji already but I guess not.

I always felt like tokenization is one of those double edged swords where it makes some stuff amazingly easier but gets tripped up on the weirdest bugs. The number of "r"s in "strawberry" being another well-known quirk.

This will be patched out shortly I’m sure

Gemini 2.5 flash seems to nail it.

https://g.co/gemini/share/c244e5f51e37

Interesting that it turns agentic looking for the right emoji.

You'll also notice the same thing happens for other non-existent emojis that sound like they should exist: dragonflies, lemurs, possums, blackberries - even Claude 4.5 will start off by saying "Yes!" and then correcting itself. It will immediately give the right answer for very specific things that you wouldn't expect to get their own emojis though.

I realized if someone were to assign me the ticket for fixing this behavior, I would have no idea where to begin with solving it even with this blog post explaining the problem, so I'm very curious to know what the most practical solution is. (They obviously aren't adding "If someone asks you about a seahorse emoji, there isn't one available yet, no matter how strongly you believe one exists." to the system prompt.)

kinda related, I wonder if the AI goes crazy if you ask for the hiker emoji that was totally real and we're being gaslit by FAANG.

Is top k meaningful after RLHF considering rewards are based off the token picked?

One explanation could be: many humans (including me) mistakenly think a seahorse emoji exists. My mind can even construct a picture of how it should look like, despite me also knowing it's very unlikely I've seen one myself.

And those text got into the training set: https://www.reddit.com/r/MandelaEffect/comments/qbvbrm/anyon...