I shudder to think what one man with a texta could do here. It isn't that hard to turn the acyclic graph into a cyclic one.

This is quite a bad argument, somewhat akin to "small drones can't kill anyone because there are no weapons on them!" back 20 years ago. Well, ok. So what if someone adds the thing that is missing? And there is no reason to think that a stochastic parrot can't go rogue, the evidence is a stochastic parrot can engage in any intellectual activity we care to define and benchmark.

I feel like that's like saying amoeba have no intent and no motivation. Or plants.

And those cause quite a bit of damage.

AI has all the intent that we gave it, and we continue giving it. That's always been the fear. Not that it will randomly wipe out humanity.

The fear is that it will decide to do that, with a purpose. Whether we tell it to protect us and it goes too far, or it decides that it can't achieve the purpose we gave it because we'll interfere and it removes that interference...

The fear is that we will set it on that path and can't stop it.

I'm sure there have been some scifi books that have it just be random, but they're far, far less worrisome.

The primary danger of "AI" lies in the extent to which it has managed to convince people of it's capability to think and reason.

A lot of people clearly have no idea that they are essentially speaking with a glorified randomizer, rather than someone understanding the question.

This is bound to eventually lead to a disaster when someone deluded into trusting the "AI" does some stupidity it suggests without passing it through any reality check.

Sam Altman on the other hand has literally billions of motivations.

Doesn’t alignment and ability to steer it require the model have no own intention?

Seems like a feature to me

I think the "human needs" point is understated here. It would be better to say agents do not really have a self-contained metabolic engine that is required to keep going. Which bubbles up as that what we interpret as "drive", "will", "agency"... basically the will to live, and being willing to do _a lot_ to live if push comes to shove. We can't really identify a mechanism of similar complexity and integration in agents or LLMs. I think the article correctly points into that direction.

But Hans Jonas has made this point much better than the article or me, in "Critique of Cybernetics" (1953). PDF: https://s3.amazonaws.com/arena-attachments/892605/f0747c7943...