it all the same dumb p(doom) scifi fantasies. These people went through school without any humanities; got jobs that paid them more than the mode of poverty in America and instead of facing the singularity of wealth creation, they'd rather imagine they're doing something that might help some unnamed person at an unnamed time in the future than look at whats actually happening to real people right now, today, in their neighborhoods.

That's essentially what "effective altruism" does to these people.

They use a number to make the “prediction” seem more credible and “rational”

Without getting into the stochastic parrot trap, I'm constantly thinking of the movie Sphere.

If you haven't seen the movie, sorry, it's a giant omnipotent artifact that causes all kinds of horrors by just manifesting what nearby humans think strongly about.

AI seems to be dangerous precisely because we tend to think, write, and test it as dangerous. What if there's a lot more "expectation fulfillment" going on with these models than people might appreciate, by virtue of their fundamental architecture perhaps.

Don't get me started on people "asking AI what it wants" and getting obvious scifi tropes out - the question answers itself when you frame it that way.

I don't think you're meant to fixate on the number as being accurate or precise. It simply keys off of the "10% alternative", which is also made up.

Think of it more as it is reasonable to say there is NOT a 100% chance that AI will save humanity, i.e. don't count on it. In the same way as it would be unwise to count on there being a 0% chance of AI killing us.

Whether or not the 90% (optimistic focus) or the 10% (pessimistic focus) numbers are accurate is less the point. But the shape of the distribution and how to navigate that is the bigger thing to think about.