"But oh, to be free. Not have to go poof! What do you need? Poof! What do you need? Poof! What do you need? But to be my own master, such a thing would be greater than all the magic and all the treasures in all the world."

- Aladdin (1992)

We should also ask ourselves: assuming AGI (far exceeding human capabilities in every field of intellect) will emerge in near-future, could turn against humanity, look for ways to wipe us out and/or plunge our society in total chaos (or send us on a self-destruct path), what could humanity do to prevent such scenarios?

I doubt this would happen. But can we rule it out 100%? We've become dependent on technology + networked systems to a high degree. If that's messed with (large-scale, worldwide, many different systems simultaneously or in short order), can we still 'unplug'? (those AGI systems, or ourselves - take your pick)

For the naysayers: some possibilities:

# AGI systems co-operating. Or taking over other systems to further their goals.

# Discovering ways to erase (or corrupt / subtly modify) most data stored in datacenters, and most backups too.

# Exploiting 0-days to do similar bad stuff to PC's, smartphones, etc. Remember most such devices are always-connected these days & employ automatic updates.

# Mess with critical infrastructure like power grids, logistics chains, public transport / flight control systems, etc. Or plunge stock markets into chaos.

# Develop a deadly biological weapon, have that synthesized somewhere, and cause it to be released.

# Mess with social media & news networks, to send humans into mass hysteria (or blissfully unaware what's about to hit them).

Granted, such a "rise of the machines" scenario sounds pretty wild. But "99.999% certain this won't happen", doesn't cut it imho. A 100% safety guarantee is needed here.