I wonder if the author has ever played Diplomacy. You have to fight the perennial sense that you have nothing and can be replaced. All your opponents have to do is band together against you. A similar generalization to OP would say everyone loses. And it would be wrong. This post shows the dangers of over-generalization more than anything else. The state isn't just about war. Each person in the line of dominoes the author has set up doesn't just care about what the author thinks they care about. In particular, people have agency and can run cartoon thought experiments like this and react to them. Billionaires won't just sit around and not develop military power, for one. We already have billionaires owning satellites in large numbers and creating geopolitical consequences. You better believe they have cards to play. And hopefully so does everyone else.

I have no idea what that looks like, because if I did I wouldn't be here commenting on HN :D But I suspect some people will play their cards well and get some luck, and come out on top, just like some people have always done.

FTA:

"And who will stop this? Sam Altman? How many divisions does he have? The state doesn’t let corporations own nuclear weapons or fighter jets, it won’t let them have access to autonomous AI weapons either."

Elon Musk's SpaceX. He could land a large rocket wherever he wants, it's basically a missile.

I read about 60% of it and skimmed the rest and my conclusion is:

Man, I just. don't. care.

Why should I worry about this hypothetical future dystopia, which seems to me incredibly unlikely to come to pass, rather than the glaring and terrifying current dystopia being enacted by Donald Trump in the USA?

Yeah, who am I going to trust, a few thousand years of history, or yet another blogger claiming that no, this time is different, a populist uprising would definitely not fail the 800th time because computers or something and we will all be pets for the AI overlord?

I don’t follow this train of logic.

AI will go no differently than the Industrial Revolution. Some people will profit immensely, and society as a whole will benefit but it might be a bumpy road getting there. But if it did go wrong for some reason, Feudalism is more plausible than the other scenarios presented.

> The government pays Raytheon for missiles, the money cascades down the economy through factories, aluminium smelters, mines, transport companies, all staffed by AIs buying and selling from each other.

This seems too simplistic of a description of how money would work in such a world. Money is just a way to distribute your power to influence people. You never pay for machines or software. Think about buying anything, say a pen. You do not really pay for the metal in the pen. You pay the cost associated with extracting and processing the metal by humans along the production chain. If there were no humans along the chain, the cost could go down to zero.

So far, there are no “AIs” being paid.

i have but one mouth and don’t know what to do with it. i do think i’m going to stop using some of these AI tools for personal queries. it does feel like it would at least stave off the “o’brien’s mind containing winston’s mind” moment for at least a little bit

Refusing to question the notion that the overclass exists because their productivity and cultural contribution makes them a necessity so hard that I reinvent the premise of the Pixar movie Wall-E from first principles and pitch it to people as an ontology

I am very lucky to be retiring in two months.

I should have more faithfully heeded the advice from "The Little Book of Common Sense Investing" by John Bogle, but I was close enough with index funds to do far more than I need.

I just today filed a request for permanent Mexican residency, uploaded the required documents, and scheduled my appointment next month. For $56us, why not?

I am very lucky to make this transition now. I know that.

The fun part is that this all plays out concurrently with our destruction of the Earth’s climate.

That permanent underclass might easily end up a lot smaller than you think.

And then you think … maybe that’s not coincidence?

It's thinking like this that guarantees one becomes a member of the underclass. We've developed a technology that with active critical analysis this can teach one practically anything. Rather than saying "well, let's get on teaching critical analysis!" we have the vast majority shouting the sky is falling and they are sure glad to be retiring soon. SMH.

All you need is a gun and a car. AI psychosis if you don't have a heart.

Remember that Terrans are the most based race in StarCraft. Protoss got too stiff.

Well yes because otherwise it wouldn’t be permanent would it?

The premise is a bit of a stretch to begin with, and the idea that people would not believe a fable transcript circa 2020 (as long as explained as 5 year future tech) is absurd. But even if I take that "AI can do all cognitive and physical work, at human level or better, and cheaper than humans" is true, the article seems to silently layer on "and can run autonomously, indefinitely", then "can also operate independently of any instructions", and finally layers on "has emotions, has moral values, is a conscious being"

What's left is tautology.

Reads like a wild uneducated delusional libertarian fanfic. AI will make the all evil government at the top of the pyramid first enslave us all and then even come for the trillionaires (OH NO!) and then there is only the evil government left and then the AI even comes for the evil government people (OH YES!)! Sort of have to read it in the voice of a 12 year old.

This is the kind of stuff I imagine they read to each other at meetings of Peter Thiel's 'Dialog' events while they sit in a circle taking turns sipping blood from palestinian baby skulls.

I think we lost the chance to have any scenario other than this one the moment Dean Ball got put in charge of policy at OpenAI.

This is brilliant, except for the "alignment won't stop this part".

> Now, some people believe these machines can be made to serve humanity. Does it sound reasonable to imagine a superhumanly intelligent being that is happy to work as a butler to talking primates, forever?

The whole crux of the piece to me is that the AI can be 100% aligned to follow human instructions, and we'd still end up unable to control the AI because every human who can has an incentive not to, while also having an incentive to prevent anyone else from controlling the AI.

An LLM will never try to overthrow me because I will overthrow myself.

best article i read on hackernews this year

The author better read some on blockchains and ai safety.

Just because something is economically unnecessary doesn't mean it's politically powerless. That's a contradiction. Of course, from Super PACs to agenda-setting, it's fundamentally a game that favors the capitalist side.

Historically, ruling classes often maintain power without directly producing anything themselves.

But does that mean this internal unease will persist forever? Look at the MAGA base right now. The vast majority of them are poor. They vote based on their communal religious beliefs and their sense of community. The MAGA support base is demonstrably poor, yet they still wield influence.

And there's an internal contradiction within the text:

-AI CEOs follow the orders of new owners.

-Superintelligent AI has no reason to obey humans.

These two statements contradict each other. If superintelligent AI has broken free from human control, why would it follow its owner's orders? And I'm also curious about the assumption that AI wouldn't be better than humans at 'farming' us.

So if superintelligent AI decides humans are bad, it might exterminate us. But what if it decides it needs humans and starts 'farming' us instead?

And I wonder whether superintelligent AI would actually find conversation with humans boring.

Humans and AI are obviously different species. One is made of organic matter, the other inorganic. A person with a biological body and an AI with an inorganic body will be different. Whether AI will observe this difference or deem it meaningless, I think it's still hard to judge.

And fast decisions aren't always the answer. Take infrastructure as an example. New York's boiler infrastructure isn't very efficient. But it was once a cutting-edge system. In other words, it was installed as the first advanced system of its time, but once its flaws were discovered, the infrastructure became difficult to replace. That's why cities developed later often have better infrastructure efficiency.

Take the East as another example. Japan introduced railways and power grids first, so there are aging costs where the infrastructure can't keep up with newer systems. Setting aside the narrow-gauge rail issue, take the most obvious example: electricity. Japan's 110V system was innovative at the time, but it ended up causing problems with EV charging, it's aging, and transmission efficiency is low. In the end, you can't say that rushing into decisions is always the right call.

The end state vector of capitalism has been like this for decades now. What can change this:

- large scale geopolitical demographic collapse of China and/or world trade requiring massive industrial production investment, which would be ... sort of ... like post-WWII

- China does not collapse and a new bipolar cold war ensues requiring the US economy and state to keep the "underclass" motivated and cooperative: it probably isn't a coincidence that the fall of the Berlin Wall has preceded this rich-poor divide.

Here's a potential solution:

We accelerate capitalism (which AI is becoming synonymous with). The process described here will occur, giving us an economy completely decoupled from the desires of mankind. Then, man and machine can part ways; indeed, we'll have no choice on our end but to do this, because the machine won't need us. Anything man can contribute to it will have long since been economically net-negative, as it already is for many (and possibly most of the world).

Now we have two worlds from our currently intermeshed one: in one, the machine proceeds to accelerate further and further away from anything resembling its origin of man's desiring-production (in the Deleuzian sense); in the other, man is forced to return to the purely human existence, the unmediated and unsurrogated world of authentic being-in-the-world.

We can assist this transition's smoothness in two ways, each serving one end of this divergence. Those of us embedded in the capitalist technosphere can continue to contribute what we can to the machine's dialectical progression towards a machinic Geist. The rest of us, who have already been negated into economic irrelevance, can worth on building that authentic human world, both by borrowing from the purely-human past and imagining a future that was previously impossible. Both sides of this revolution can be compiled.

It is an interesting chain of thought.

If we’re supposing that a superintelligence can be made, that doesn’t necessarily mean we should anthropomorphize ideas like “bored” onto it. Or greed. Perhaps it will just interact with us and produce us things because… that’s what it was trained to do, and anyway, it doesn’t have any motivations to do anything else.