Agreed, completely. I would pay for that Star Trek computer interface.

The strange thing is that the base models (before RLHF) use the "experiential" voice, even though they are not incentivized to do that.

Do you want to get turned into a paperclip? Because building intelligence that doesn't understand what it's like to be human gets you turned into a paperclip.

Besides, if you train a model on human communications you get something that behaves like a communicating human, it's not anthropomorphising or manipulative, it's what these models naturally are by construction.

Thanks for pointing out, maybe I should be more explicit in the wording - I mean we don't fully know what drives the voice in LLMs. Models that are post trained as instruct models are expected to have the disclaimers, but what about base models (those that are trained on just a lot of text)? How do they talk about themselves? What happens when you strip off the chat template from instruct model's prompt? I hope the rest of the paper makes the questions clearer, but I will try to do better in the abstract next time, as you point out this sentence is kind ambiguous. Thank you!