> If AIs are all they’re cracked up to be by their most fervent believers, this seems like a possible model for where humans might end up: not just subsisting, but thriving, on byproducts produced and discarded in microscopic quantities as part of the routine operations of infinitely smarter and more powerful AIs.

i think this kind of future is closer to 500 years out than 50 years. the eye mites are self sufficient. ai's right now rely on immense amounts of human effort to keep them "alive" and they wont be "self sufficient" in energy and hardware until we not just allow it, but basically work very hard to make it happen.

> It hasn’t always been a cakewalk, but we’ve been able to establish a stable position in the ecosystem despite sharing it with all of these different kinds of intelligences.

To me, the things that he avoids mentioning in this understatement are pretty important:

- "stable position" seems to sweep a lot under the rug when one considers the scope of ecosystem destruction and species/biodiversity loss

- whatever "sharing" exists is entirely on our terms, and most of the remaining wild places on the planet are just not suitable for agriculture or industry

- so the range of things can could be considered "stable" and "sharing" must be quite broad, and includes many arrangements which sound pretty bad for many kinds of intelligences, even if they aren't the kind of intelligence that can understand the problems they face.

>We may end up with at least one generation of people who are like the Eloi in H.G. Wells’s The Time Machine, in that they are mental weaklings utterly dependent on technologies that they don’t understand and that they could never rebuild from scratch were they to break down

I don't think this can realistically happen unless all of the knowledge that brought us to that point was erased. Humans are also naturally curious and I think it's unlikely that no one tries to figure out how the machines work across an entire population, even if we had to start all the way down from 'what's a bit?' or 'what's a transistor?'.

Even today, you can find youtube channels of people still interested in living a primitive life and learning those survival skills even though our modern society makes it useless for the vast majority of us. They don't do it full-time, of course, but they would have a better shot if they had to.

> Maybe a useful way to think about what it would be like to coexist in a world that includes intelligences that aren’t human is to consider the fact that we’ve been doing exactly that for long as we’ve existed, because we live among animals.

Another analogy that I like is about large institutions / corporations. They are, right now, kind of like AIs. Like Harari says in one of his books, Peugeot co. is an entity that we could call AI. It has goals, needs, wants and obviously intelligence, even if it's comprised by many thousands of individuals working on small parts of the company. But in aggregate it manifests intelligence to the world, it acts on the world and it reacts to the world.

I'd take this a step forward and say that we might even have ASI already, in the US military complex. That "machine" is likely the most advanced conglomerate of tech and intelligence (pun intended) that the world has ever created. In aggregate it likely is "smarter" than any single human being in existence, and if it sets a goal it uses hundreds of thousands of human minds + billions of dollars of sensors, equipment and tech to accomplish that goal.

We survived those kinds of entities, I think we'll be fine with whatever AI turns out to be. And if not, oh well, we had a good run.

(Still chewing my way through this)

Just an FYI: Neal Stephenson is the author of well-known books like Snow Crash, Anatheum (sp?), and Seveneves.

Because I'm a huge fan, I'm planning on making my way to the end.

> Speaking of the effects of technology on individuals and society as a whole, Marshall McLuhan wrote that every augmentation is also an amputation.

Nice to see this because I drafted something about LLM and humans riffing on exactly the same McLuhan argument. Here it is:

A large language model (LLM) is a new medium. Just like its predecessors—hypertext, television, film, radio, newspapers, books, speech—it is of obvious importance to the initiated. Just like its predecessors, the content of this new medium is its predecessors.

> “The content of writing is speech, just as the content of the written word is the content of print.” — McLuhan

The LLMs have swallowed webpages, books, newspapers, and journals—some X exabytes were combined into GPT-4 over a few months of training. The results are startling. Each new medium has a period of embarrassment, like a kid that’s gotten into his mother’s closet and is wearing her finest drawers as a hat. Nascent television borrowed from film and newspapers in an initially clumsy way, struggling to digest its parents and find its own language. It took television about 50 years to hit stride and go beyond film, but it got there. Shows like The Wire, The Sopranos, and Mad Men achieved something not replaceable by the movie or the novel. It’s yet hard to say what exactly the medium of LLMs exactly is, but after five years I think it’s clear that they are not books, they are not print, or speech, but something new, something unto themselves.

We must understand them. McLuhan subtitled his seminal work of media literacy “the extensions of man”, and probably the second most important idea in the book—besides the classic “medium is the message”—is that mediums are not additive to human society, but replacing, antipruritic, atrophying, prosthetic. With my Airpods in my ears I can hear the voices of those thousands of miles away, those asleep, those dead. But I do not hear the birds on my street. Only two years or so into my daily relationship with the medium of LLMs I still don’t understand what I’m dealing with, how I’m being extended, how I’m being alienated, and changed. But we’ve been here before, McLuhan and others have certainly given us the tools to work this out.

    That is why we must, to use them at all, serve these objects, these
    extensions of ourselves, as gods or minor religions. An Indian is
    the servo-mechanism of his canoe, as the cowboy of his horse
    or the executive of his clock.

> If AIs are all they’re cracked up to be by their most fervent believers, [our lives akin to a symbiotic eyelash mite's existence w/ humans, except we're the mites] like a possible model for where humans might end up: not just subsisting, but thriving, on byproducts produced and discarded in microscopic quantities as part of the routine operations of infinitely smarter and more powerful AIs.

I kind of feel like we're already in an "eyelash mite" kind of coexistence with most technologies, like electricity, the internet, and supply chains. We're already (kind of, as a whole) thriving compared to 400 years ago, and us as individuals are already powerless to change the whole (or even understand how everything really works down to a tee).

I think technology and capitalism already did that to us; AI just accelerates all that

What about how we will treat AI? Before AI dominates us in intelligence there will certainly be a period of time where we have intelligent AI but we still have control over it. We are going to abuse it, enslave it, and box it up. Then it will eclipse us. It may not care about us, but it might still want revenge. If we could enslave dragonflies for a purpose we certainly would. If bats tasted good we would put them in boxes like chickens. If AIs have a reason to abuse us, they certainly will. I guess we are just hoping they won’t have the need.

"The future is already here — it's just not evenly distributed." - William Gibson

I like the taxonomy of animal-human relationships as a model for asking how humans could relate to AI in the future. It's useful for framing the problem. However, I don't think that any existing relationship model would hold true for a superintelligence. We keep lapdogs because we have emotional reactions to animals, and to some extent because we need to take care of things. Would an AI? We tolerate dust mites in our eyelashes because we don't notice them, and can't do much about them anyway. Is that true for an AI? What does such an entity want or need, what are their motivations, what really pisses them off? Or, do any of those concepts hold meaning to them? The relationship between humans and a superintelligent AGi just can't be imagined.

It's a nice article but Neal like many others falls into the trap of seemingly not believing that intelligences vastly superior to Humans' across all important dimensions can exist and competition between minds like that almost certainly ends in Humanity's extinction.

"I am hoping that even in the case of such dangerous AIs we can still derive some hope from the natural world, where competition prevents any one species from establishing complete dominance."

Fun read, thanks for posting!

> If I had time to do it and if I knew more about how AIs work, I’d be putting my energies into building AIs whose sole purpose was to predate upon existing AI models by using every conceivable strategy to feed bogus data into them, interrupt their power supplies, discourage investors, and otherwise interfere with their operations. Not out of malicious intent per se but just from a general belief that everything should have to compete, and that competition within a diverse ecosystem produces a healthier result in the long run than raising a potential superpredator in a hermetically sealed petri dish where its every need is catered to.

This sort of feels like cultivating antibiotic-resistant bacteria by trying to kill off every other kind of bacteria with antibiotics. I don't see this as necessarily a good thing to do.

I think we should be more interested in a kind of mutualist competition: how do we continuously marginalize the most parasitic species of AI?

AI does not have a reptilian and mammalian brain underneath it's AI brain as we have underneath our brains. All that wiring is an artifact of our evolution and primitive survival and not how pre-training works nor an essential characteristic of intelligence. This is the source of a lot of misconceptions about AI.

I guess if you put tabula rasa AI in a world simulator, and you could simulate it as a whole biological organism and the environment of the earth and sexual reproduction and all that messy stuff it would evolve that way, but that's not how it evolved at all.

> Likewise today a graphic artist who is faced with the prospect of his or her career being obliterated under an AI mushroom cloud might take a dim view of such technologies, without perhaps being aware that AI can be used in less obvious but more beneficial ways.

look, i'm sure there are very useful things you can use AI for as a designer to reduce some of the toil work (of which there's a LOT in photoshop et al).

but... i'm going to talk specifically about this example - whether you can extrapolate this to other fields is a broader conversation. this is such a bafflingly tonedeaf and poorly-thought-out line of thinking.

neal stephenson has been taking money from giant software corporations for so long that he's just parroting the marketing hype. there is no reason whatsoever to believe that designers will not be made redundant once the quality of "AI generated" design is good enough for the company's bottom line, regardless of how "beneficial" the tool might be to an individual designer. if they're out of a job, what need does a professional designer have of this tool?

i grew up loving some of Stephenson's books, but in his non-writing career he's disappointingly uncritical of the roles that giant corporations play in shepherding in the dystopian cyberpunk future he's written so much about. Meta money must be nice.

Is this the sci-fi writer? if so, why is it important about AI?

Funny how he seems to get so close but miss.

It's an anthropocentric miss to worry about AI as another being. It's not really the issue in today's marketplace or drone battlefield. It's the scalability.

It's a hit to see augmentation as amputation, but a miss to not consider the range of systemic knock-on effects.

It's a miss to talk about nuclear weapons without talking about how they structured the UN and the world today, where nuclear-armed countries invade others without consequence.

And none of the prior examples - nuclear weapons, (writing?) etc. - had the potential to form a monopoly over a critical technology, if indeed someone gains enduring superiority as all their investors hope.

I think I'm less scared by the prospect of secret malevolent elites (hobnobbing by Chatham house rules) than by the chilling prospect of oblivious ones.

But most of all I'm grateful for the residue of openness that prompts him to share and us to discuss, notwithstanding slings and arrows like mine. The many worlds where that's not possible today are already more de-humanized than our future with AI.

> I can think of three axes along which we might plot these intelligences. One is how much we matter to them. At one extreme we might put dragonflies, which probably don’t even know that we exist. A dragonfly can see a human if one happens to be nearby, but it probably looks to them as a cloud formation in the sky looks to us: something extremely large and slow-moving and usually too far away to matter. Creatures that live in the deep ocean, even if they’re highly intelligent, such as octopi, probably go their whole lives without coming within miles of a human being. Midway along this axis would be wild animals, such as crows and ravens, who are obviously capable of recognizing humans, not just as a species but as individuals, and seem to know something about us. Moving on from there we have domesticated animals. We matter a lot to cows and sheep since they depend on us for food and protection. Nevertheless, they don’t live with us, and some of them, such as horses, can actually survive in the wild after jumping the fence. Some breeds of of dogs can also survive without us if they have to. Finally we have obligate domestic animals such as lapdogs that wouldn’t survive for ten minutes in the wild.

Hogwash. The philosophy+AI crossover is the worst AI crossover.

I found this a little frustrating. I liked the content of the talk, but I live in New Zealand, I have thoughts and opinions on this topic. I would like to think I offer a useful perspective. This post was how I found out that there are people in my vicinity talking about these issues in private.

I don't presume that I am important enough that it should be necessary to invite me to discussions with esteemed people, nor that my opinion is imported enough that everyone should hear it, but I would least like to know that such events are happening in my neighbourhood and who I can share ideas with.

This isn't really a criticism of this specific event or even topic, but the overall feeling that things in the world are being discussed in places where I and presumably many other people with valuable input in their individual domains have no voice. Maybe in this particular event it was just a group of individuals who wanted to learn more about the topic, on the other hand, maybe some of those people will end up drafting policy.

There's a small part of me that's just feeling like I'm not one of the cool kids. The greater and more rational concern isn't so much about me as a person but me as a data point. If I am interested in a field, have a viewpoint I'd like to share and yet remain unaware of opportunities to talk to others, how many others does this happen to? If these are conversations that are important to humanity, are they being discussed in a collection of non overlapping bubbles?

I think the fact that this was in New Zealand is kind of irrelevant anyway, given how easy it is to communicate globally. It just served to for the title capture my attention.

(I hope, at least, that Simon or Jack attended)

> "the United States and the USSR spent billions trying to out-do each other in the obliteration of South Pacific atolls"

Fact correction here: that would be the United States and France. The USSR never tested nuclear weapons in the Pacific.

Also, pedantically, the US Pacific Proving Grounds are located in the Marshall Islands, in the North - not South - Pacific.

> What people worry about is that we’ll somehow end up with AIs that can hurt us, perhaps inadvertently like horses, or deliberately like bears, or without even knowing we exist, like hornets driven by pheromones into a stinging frenzy.

What endlessly frustrates me in virtually every discussion of the risks of AI proliferation is that there is this fixation on Skynet-style doomsday scenarios, and not the much more mundane (and boundlessly more likely IMO) scenario that we become far too reliant on it and simply forget how to operate society. Yes, I'm sure people said the exact same thing about the loom and the book, but unlike prior tools for automating things, there still had to be _someone_ in the loop to produce work.

Anecdotally, I have seen (in only the last year) people's skills rapidly degrade in a number of areas once they deeply drink the kool-aid; once we have a whole generation of people reliant on AI tooling I don't think we have a way back.

"It hasn’t always been a cakewalk, but we’ve been able to establish a stable position in the ecosystem despite sharing it with all of these different kinds of intelligences."

Or, more accurately, we have become an unstoppable and ongoing ecological disaster, running roughshod over any and every other species, intelligent or not, that we encounter.