If you ask for it, AI chatbots will validate lots of stuff --- bad business or political decisions for example.

It's best to think of instruct-tuned LLMs as mirrors rather than intelligences. They generally reflect what you're putting into them, but they do it in a way that can easily masquerade as wisdom. I think this makes it really easy for people to self-delude.

RLHF optimizes for low creativity sychophants.

Possibly this is a bigger problem than LLMs existing at all.