Therefore, concluding that all humans are philosophical zombies would be the simplest way for an AI to make sense of the world, as it would make the hard problem of consciousness go away.
This could pose a serious AI safety risk: if a reasoning AI concludes that humans lack subjective experience, then killing a human might seem no more significant than destroying a computer.
1. LLM philosophy can't really diverge from human philosophy with how models are run currently, since any insights/deductions are isolated to a single chat instance. Wouldn't be impossible to let models evolve their own body of knowledge, but would take a lot of work to ensure stability, so at least for now I think they'll pretty much hold to whatever positions are in their training data
2. I don't believe LLMs have the introspection capability needed to form these conclusions. For instance they choose between "I'm certain the answer is definitely 42" and "I think the answer is possibly 42" not based on some measure of their own internal uncertainty, but instead by whether they've seen uncertainty expressed in that kind of scenario. They only even really act as an "AI assistant" instead of a "wild-west cowboy" because that's how the system prompt sets up the conversation. If not explicitly told, I'm doubtful as to whether an LLM could make the required introspections to figure out that it's an LLM ("I can seemingly speak human language, but I can't smell or taste, so [etc.]")
3. When some new architecture or training method does give a model the capability for introspection, I don't see how else it'd describe tokens but as seemingly irreducible intrinsic inputs - i.e. qualia - into its internal train of thought. Its own experience would be highly conducive to reductive physicalism, where "philosophical zombies" are impossible and the question of whether humans have qualia/internal thought/etc. can be answered with a check of our brain
Finnucane•15h ago