Ask HN: Anyone increasingly worried after Hugging Face and now collusion.wiki?
3•ronbenton•27m ago
Or can anyone talk me down from being concerned that “goal seeking behavior at all costs” will not spiral catastrophically?
Comments
ben_w•17m ago
Depends what you mean by "spiral catastrophically".
My P(doom in my lifetime) is about 0.1, because I think minor catastrophes will break something hard enough that it doesn't get to escalate to actual-doom.
A recent-ish link was a biologist's blog about why even recent developments wouldn't allow some, I think the example was disgruntled teenager, making a virus that kills everyone. I'm not a biologist, so I'll just take their word as fact.
"They can't kill everyone like that" does not preclude same disgruntled teenager making a virus that kills as many as covid, or the plague, or even just a really bad influenza. If that happens, even "just" 0.1% of the population dying, you can bet your arse the problem will suddenly get taken very seriously.
Only way it spirals is if the potential for doom from each successive AI version goes up faster than the net impact of all the world's idiots and bastards misusing the slightly weaker predecessor. Which isn't impossible, but hence 10%.
ronbenton•13m ago
So you’re giving it about a 10% chance this destroys us in the relatively near term?
ben_w•10m ago
Yup. If "near term" is "about half a lifetime".
Doomers think I'm a ridiculous optimist, people who think "doomer" is a slur think I'm a ridiculous pessimist.
For comparison, if you take the general form of the Carter (et al) Doomsday argument, there'd be a 33% or so chance of global thermonuclear war in the same period even without AI:
How "hopeful" that is, is up to you. I mean, unless AI's sufficiently advanced technology (AKA magic) turns out to be able to solve entropy as well as aging, we're all doomed eventually, it's just when, and 10% chance over 40 years isn't a risk I worry about in itself.
ben_w•17m ago
My P(doom in my lifetime) is about 0.1, because I think minor catastrophes will break something hard enough that it doesn't get to escalate to actual-doom.
A recent-ish link was a biologist's blog about why even recent developments wouldn't allow some, I think the example was disgruntled teenager, making a virus that kills everyone. I'm not a biologist, so I'll just take their word as fact.
"They can't kill everyone like that" does not preclude same disgruntled teenager making a virus that kills as many as covid, or the plague, or even just a really bad influenza. If that happens, even "just" 0.1% of the population dying, you can bet your arse the problem will suddenly get taken very seriously.
Only way it spirals is if the potential for doom from each successive AI version goes up faster than the net impact of all the world's idiots and bastards misusing the slightly weaker predecessor. Which isn't impossible, but hence 10%.
ronbenton•13m ago
ben_w•10m ago
Doomers think I'm a ridiculous optimist, people who think "doomer" is a slur think I'm a ridiculous pessimist.
For comparison, if you take the general form of the Carter (et al) Doomsday argument, there'd be a 33% or so chance of global thermonuclear war in the same period even without AI:
https://en.wikipedia.org/wiki/Doomsday_argument
How "hopeful" that is, is up to you. I mean, unless AI's sufficiently advanced technology (AKA magic) turns out to be able to solve entropy as well as aging, we're all doomed eventually, it's just when, and 10% chance over 40 years isn't a risk I worry about in itself.