My objection is that many of these scientists assume the "alignment" framing, which I find disingenuous in a technocratic way: imagine a sci fi movie (like Dune) where the rulers want their AI servants to "align" with their interests. The sheer hubris of this, and yet we have our top experts using these words without any irony or self awareness.
Ironically, your chosen example is a sci-fi universe that not only doesn't have any AI, the backstory had a holy war against them.
But, frankly, at the moment I see less danger from too-smart AIs than I do from too-dumb AIs that people treat like they're smart. (In particular, they blindly accept their output as right or authoritative.)
Alignment isn't making the AI do what you want, it's making the AI want what you want. What you really really want.
Simply getting the AI to do what you want is more of an "is it competent yet?" question than an "alignment" question, and at the moment the AI is — as AnimalMuppet wrote — not quite as competent as it appears in the eyes of the users. (And I say that as one who finds it already valuable).
Adding further upon what AnimalMuppet has written with only partial contradiction, consider a normal human: We are able to disregard our natural inclination to reproduce, by using contraceptives. This allows us to experience the reward signal that our genes gave us to encourage us to reproduce, without all the hassle of actually reproducing. Evolution has no power over us to change this.
We are to AI what DNA is to us. I say we do not have zero chances, but rather likely one-to-a-handful of chances, to define the AI's "pleasure" reward correctly, and if we get it wrong, then just as evolution's literally singular goal (successful further reproduction) is almost completely circumvented in our species, then so too will all our human interests be completely circumvented by the AI.
Pessimists in the alignment research field will be surprised to see me write "one-to-a-handful of chances"; I aver that there are likely to be several attempts which are "only a bit competent" or even "yes it's smart but it's still got blind spots" models before we get to them being so smart we can't keep up. So in this regard, I also disagree with their claim:
> An AI smart enough to be a danger is an AI that is smart enough to override your attempt at "alignment"
Ad hominems like those make it impossible for me to respond to your comment fairly.
If you want to lead with ad hominems then I studied EECS at elite American universities for undergrad then for my PhD, so I have every right to criticize, within reason, the more prominent leaders of entire field for being philosophically blinkered.
The short of it is that your "making the AI want what you want" is called extrinsic internalization. It is highly problematic. It is not surprising that a roomful of CS technocratic researchers (beholden to SV funding) would make this sort of lay psychology mistake--a smuggled premise. It is bad science, and irrational. Their vested interests make the bias seem rational to them. And then people in tech, like yourself, drink their Koolaid.
So what's more likely is your knowledge is ignorant, you resort to comparisons about human genetic evolution to make handwavy claims about how important this historic moment will be, which is irrelevant to the validity of alignment as a framing/methodology. In fact a cursory check shows CS AI papers written by dissenting scientists who are very much against the alignment framework. Did you even know that? Hope you don't so hastily accuse them of "not understanding" the issues as you did here.
Problematic or not, the technical term is "reward function", and you don't get to not have one in an AI, not even when it's e.g. AlphaGo. The hard part is specifying the reward function to match what you actually want, otherwise you get Goodhart's Law.
Talking about this in psych terms is a "wet pavements cause rain" error.
> If you want to lead with ad hominems then I studied EECS at Berkeley then for my PhD at Princeton, so I have every right to criticize the more prominent leaders of entire field for being philosophically blinkered.
The right to, sure. I'm not stopping you from doing it. I'm saying you're making a foundational error in your criticism.
> In fact a cursory check shows CS AI papers written by dissenting scientists who are very much against the alignment framework.
Naturally. Dissent is normal.
As for details of which matters and how hard they might be: inner alignment, outer alignment, Goodharting, specification gaming, principal-agent issues, anything Yudkowsky has ever said about it being impossible because we only get one chance and we're bad at this, etc.
But your dismissal, as you wrote it in this thread, is as shallow as you call mine.
(Not that I expect you to read this given the other comment, but if you name a scifi setting famously anti-AI, from whence the term "Butlerian Jihad" entered our culture, you should expect to be called out on that. And TBH your replacement wasn't any good either: not only is Measure Of A Man more about the question "Is this AI a person at all?", but fictional references are not evidence of anything beyond the culture in which they were created).
I think this is one of the most overlooked too.
This is also a very bad problem with people, and big things can really crumble fast when a situation comes up which truly calls for more intelligence than is at hand. Artificial or not.
It can really seem like smooth sailing for years before an event like that rears it's ugly head, compounding the lurking weakness at a time when it can already be too late.
Now human-led recovery from failures of human-fallible systems does have at least a few centuries head start compared to machine recovery of AI-fallible systems. So there is that, which is not exactly fair. As AI progresses I guess you can eventually expect the stuff that works to achieve comparable validation over time.
treetalker•6mo ago
> This is not science fiction. There are very, very knowledgeable people—and I just talked to one today—who worry very much that human beings will not be able to control the technology, and that artificial intelligence will in fact dominate our society. We will not be able to control it. It may be able to control us. That’s kind of the doomsday scenario—and there is some concern about that among very knowledgeable people in the industry.
Bluestein•6mo ago