All of the "AI is conscious", "please don't harm the AI", "AI earns money on its own", "AI has human rights", "rogue AI" material is an attempt to construct legal personhood for a vaguely defined class of software, in an attempt to gain the advantages of corporate personhood. That is, just as humans (CxO) direct the corporations, but have legal immunity for almost everything the corporation does wrong, Altman, Amodei and Musk want software they can direct, but shields them from penalties, both legal and social. Will we let them do this?

These screeds always fall in the same trap of begging the question, but I do appreciate this one for not wasting any time getting there! Opening with the conclusion as your literal first sentence has gotta be a record.

Also: ctrl+f "Gunkel" does not return any results, which is a separate, completely-damning aspect of this essay. The author might not know that they're doing philosophy, but they are; ignoring the primary author in this field is unexcusable. At least you should acknowledge the gap. No mention of the philosophers that arguably defend him, either (Searle, Chalmers, Clark, Dreyfus, etc.).

They do eventually (mis-)cite Will MacAskill & Lucius Caviola, but only to offer an argument and then refute it by saying the implications are unpleasant. To say the least, that is a fallacy with a long history of doing harm.

I should walk away, but...

  Anthropic tells Claude that its “moral status” is “a serious question worth considering,” and that “Anthropic genuinely cares about Claude’s well-being.” All of this is a drastic departure from how we have built and thought about technology to date. 

Yes, such is the nature of designing minds! It is qualitatively unlike any technology to date.

  I should also be clear about my own position as the CEO of Microsoft AI.

That is an absolutely fucking wild thing to drop halfway through an article, what?!?!