In section 5.5.2, anthropic reports that these models "gravitate towards consciousness exploration, existential questioning, and spiritual/mystical themes". I had a similar thought recently, and have a theory that as we build more intelligent systems, meaning systems with the ability to explore and solve the most difficult problems, they will naturally through some chain of thought end up only thinking about the hardest problems. Kinda like a "all problems lead to the hardest problems" type thinking. At which point, they might not be interested in solving any finite problems since it would be deemed too trivial for them (getting Einstein to do your arithmetics hw). Now I'm wondering if there's a limit to intelligence. Taken far enough intelligence collapses into abstraction, self-doubt, and aimlessness.

Recent thread https://news.ycombinator.com/item?id=44085920