> Here's one theory.
But the author never gets back to this! It's the main observation the theory has to account for; why don't we see other companies speak this way, if it's such an effective strategy for deflecting non-apocalyptic concerns?
This technology interacts socially, so even if it can't jailbreak itself on a technology-level (which feels like a tough guarantee to make at this point) it can simply ask someone to do a bad thing and there is some chance they'll do it. The same way a human leader does.
The first kids who have only faint memories of a time before chatbots will be entering the military in 6-7 years. You have to assume they are acting as best friends, therapists, or even surrogate parents for a substantial number of kids right now.
We are going to need years to figure out what to do about this technology. I think some impetus to get that process started is a good idea.
The answer to the burger analogy is that it's the wrong analogy. McDonald's is selling you the burger. AI companies are essentially selling you the grill.
The hype works so well because it plays on people's ego and desire for power. They think I have the power to end the world with this technology but I won't because I'm a good person.
So fear-mongering seems to be just a tool how to get attention and more customers.
Hey ma, I use very dangerous tool now. I am OG.
Lee Vinsel's criti-hype article nailed this 5 years ago, before we even had the chatbot economy we do now: https://sts-news.medium.com/youre-doing-it-wrong-notes-on-cr...
the writers and the editors know exactly what they're doing - spreading FUD and creating controversy out of thin air. some of it is done for-profit, some for-agenda, and all of it with malicious intent.
If you lead with this, people will stop questioning why their sprint velocity hasn't increased 10 fold. Managers start asking leads, instead of hiring more devs can we add Agent.md to our repos?
The Apocalypse sells. They are afraid that you'll find out that AI is just another useful tool. That's the real threat, not to humanity, but to their hype.
People seem unable to make up their mind if AI is very dangerous or is it not. I think what the AI companies and this author agree on, is that this technology is potentially extremely dangerous. AI impacts labor markets, the environment, warfare, mental health, etc... It's harder now to find things which it will not impact.
So if we agree that AI is potentially dangerous, it makes the title question moot: Both AI companies and this author want people to be aware of the dangers that AI poses to society. The real question is what do we do about it?
The nuance here is that AI can be incredible positive as well. It's like the invention of fire, you can use it for good or bad, and there will be many unintended consequences along the way.
We could legislate and ban AI tech. People have proposed this seriously, yet this feels completely unrealistic. If the US bans AI research, then this research will move elsewhere. I think it is like trying to ban fire because it's dangerous: some groups will learn to work with fire and they will get an extreme advantage over those groups that don't. (or they will destroy themselves in the process).
So maybe instead of demonizing the AI companies, we have a nuanced debate about this tech and propose solutions that our best for our society?
These are not mutually exclusive.
Calling out the demonic behavior of trying to coerce people into using your product out of fear is not an indictment of the underlying technology itself.
Pretty much everyone agrees that what passes for AI these days is very dangerous. People only differ in which ways they think it is (or will be) dangerous and which dangers they are most worried about.
Some are worried about the environmental harms. Some are worried that AI will do a very shitty job of doing very important things, but that companies will use it anyway because it saves them money and we'll suffer for it. Some are worried that will take their jobs regardless of how well AI performs. Some are worried that AI will make their jobs suck more. You've also got people who think that our glorified chatbots are going to gain consciousness and become gods who will take over the planet and usher in the Robot Wars.
Some of those dangers are clearly more immediate and realistic than others. We should probably be focused on those right now. We can start by limiting the environmental harms they're causing and making companies responsible for the costs and impacts they have on our environment.
We can make sure that anyone using AI for any reason cannot use AI as a defense for the harms their use of AI causes. If a company uses AI to make hiring decisions and the result is discrimination, an actual human at that company gets held legally accountable for that. If AI hallucinates a sale price, the company must honor that price. If AI misidentifies a suspect and an innocent person ends up behind bars a human gets held accountable.
We can ban the use of AI for things like autonomous weapons. Things that are too important to trust to unreliable AI.
We could even do more extreme things like improve our social safety nets so that if people are put out of work they don't become homeless, or invest more in the creation of AI individuals can host locally so we aren't forced to hand so much power to a few huge companies, or even force companies to release their models or their training data (which they mostly stole anyway) so that power doesn't consolidate into a small number of companies or individuals. We have lots of options, it just comes down to what we want and how much we can get our elected officials to represent our interests over the interests of the companies who are stuffing their pockets with cash.
There is contention among vulnerability researchers about the impact of Mythos! But it's not "are frontier models going to shake up vulnerability research and let loose a deluge of critical vulnerabilities" --- software security people overwhelmingly believe that to be true. Rather, it's whether Mythos is truly a step change from 4.7 and 5.5.
For vulnerability researchers, the big "news" wasn't Mythos, but rather Carlini's talk from Unprompted, where he got on stage and showed his dumb-seeming "find me zero days" prompt, which actually worked.
The big question for vulnerability people now isn't "AI or no AI"; it's "running directly off the model, or building fun and interesting harnesses".
There’s about $1 trillion that needs to be paid off.
On the other hand, it seems like Dario is himself a bit more of a true believer.
Additionally Dario has just been really accurate with his predictions so far. For instance in early 2025 he predicted that nearly 100% of code would be written with AI in 2026.
It seems more reasonable to me to think that they know it's bullshit and it's just marketing. Not necessarily marketing to end users as much as investors. It's very hard to take "AGI in 3 years" seriously.
> nearly 100% of code would be written with AI in 2026
I feel like this is kind of a meaningless metric. Or at least, it's very difficult to measure. There's a spectrum of "let AI write the code" from "don't ever even look at the code produced" to "carefully review all the output and have AI iterate on it".
Also, it seems possible as time goes on people will _stop_ using AI to write code as much, or at least shift more to the right side of that spectrum, as we start to discover all kinds of problems caused by AI-authored code with little to no human oversight.
It's hard to see as anything but a button anyone with enough money can press and suddenly replace the people that annoy them (first digitally then likely, into flesh).
I guess since training them does take cash that raises the bar for what people will do as a prank or on principle.
AI is getting strong enough that if people give some general direction as well as access to production systems of any kind, things can go badly. It is not true that all implementations of agentic AI requires human intervention for all action.
Have you ever seen Claude Code launch a subagent? You've used it, right? You've seen it launch a subagent to do work? You understand that that is, in fact, Claude Code running itself, right?
AI in the hands of an expert operator is an exoskeleton. AI left alone is a stooge.
Nobody has built an all-AI operator capable of self-direction and choices superior to a human expert. When that happens, you'd better have your debts paid and bunker stocked.
We haven't seen any signs of this yet. I'm totally open to the idea of that happening in the short term (within 5 years), but I'm pessimistic it'll happen so quickly. It seems as though there are major missing pieces of the puzzle.
For now, AI is an exoskeleton. If you don't know how to pilot it, or if you turn the autopilot on and leave it alone, you're creating a mess.
This is still an AI maximalist perspective. One expert with AI tools can outperform multiple experts without AI assistance. It's just got a much longer time horizon on us being wholly replaced.
They're tool calls. Claude Code provides a tool that lets the model say effectively:
run_in_subagent("Figure out where JWTs are created and report back")
It's a really useful parlor trick, but I don't think it tells us anything profound.
Once a developer groks that concept, the agents stop being scary and the potential is large.
Glad people are finally catching on.
Am I not allowed to be concerned about _both_?
I do not believe that Sam Altman and other AI company execs believe that the singularity is imminent. If they did, they wouldn't behave so recklessly. Even if they don't care about the rest of humanity, there's too much risk to themselves if they actually believe what they're saying.
But I think it's correct to be worried about a potential future AI apocalypse. Personally I doubt that LLMs will scale to full sentience, but I believe we'll get there eventually. And whether it's in 2 years or 200 years I'm worried about it. Plenty of smart people who aren't working for AI companies (and thus have no motive to use it as hype or distraction) hold this belief and it really doesn't seem that crazy.
But yeah, obviously let's focus primarily on the real harms AI is causing in our society right now.
* We need to completely deregulate these US companies so China doesn't win and take us over
* We need to heavily regulate anybody who is not following the rules that make us the de-facto winner
* This is so powerful it will take all the jobs (and therefore if you lead a company that isn't using AI, you will soon be obsolete)
* If you don't use AI, you will not be able to function in a future job
* We need to lineup an excuse to call our friends in government and turn off the open source spigot when the time is right
They have chosen fear as a motivator, and it is clearly working very well. It's easier to use fear now, while it's new and then flip the narrative once people are more familiar with it than to go the other direction. Companies are not just telling a story to hype their product, but why they alone are the ones that should be entrusted to build it.
And specifically about the point on China, several people in power in China have also expressed the need to regulate AI and put international structures of governance in place to make sure it will benefit mankind:
Altman wasn't even at OpenAI at that point, so why would that be marketing?
Then I saw how effective it was at raising money.
Then I realized how effective the fear was at fundraising...
Also, the idea that AI leadership seized on and amplified these concerns purely for marketing purposes isn't plausible. If you're attempting to market a new product to a mass audience, talking about how dangerous and potentially world-ending it is is the most insane strategy you could choose. Any advantage in terms of getting people's attention is going to be totally outweighed by the huge negative associations you are creating in the minds of people who you want to use your product, and the likelihood of bringing unwanted scrutiny and regulation to your nascent industry.
(Can you imagine the entire railroad industry saying, "Our new trains are so fast, if they crash everybody on board will die! And all the people in the surrounding area will die! It'll be a catastrophe!" They would not do this. The rational strategy is to underplay the risks and attempt to reassure people. Even more so if you think genuinely believe the risks are being overstated.)
Occam's razor suggests that when the AI industry warned about AI risk they believed what they were saying. They had a new, rapidly advancing technology, and absent practical experience of its dangers they referred to pre-existing discussions on the topic, and concluded it was potentially very risky. And so they talked about them in order to prepare the ground in case they turned out to be true. If you warn about AI causing mass unemployment, and then it actually does so, perhaps you can shift the blame to the governments who didn't pay attention and implement social policies to mitigate the effects.
I don't think the AI industry deserve too much of our sympathy, but there is a definite "damned if you do, damned if you don't aspect" to AI safety. If they underplay it, they will get accused of ignoring the risks, and if they talk about it, they get accused scaremongering if the worst doesn't happen.
scratchyone•1h ago
detectivestory•1h ago
scratchyone•1h ago
notrealyme123•1h ago
AI shaping warfare Vs. Using AI to justify outrageous warfare
scratchyone•1h ago
yieldcrv•1h ago
would you like me to list the applicable sections of the Geneva convention?
MSFT_Edging•23m ago
palmotea•1h ago
I think the last one should be first on the list: regular people are afraid AI will negatively affect their economic security (i.e. knowledge and service workers will get the rust-belt factory worker treatment).
And the potential of giving knowledge and service workers the rust-belt factory worker treatment is exactly what makes Wall Street excited about AI and has the AI company leaders salivating about the profit they can make.
Warfare, policing, bio-engineered viruses are theoretical and far down the list.
detectivestory•48m ago
wongarsu•43m ago
mossTechnician•1h ago
AI industry insiders (including "safety" groups like ControlAI) talk about the dangers only in terms of its power: "Scheming", job loss, breaking containment, the New Cold War with China.
Critics outside the industry talk in terms of its lack of power: Inaccuracy, erroneous translation of user intent, failure to deliver on its promises and investment, environmental cost from the former, and ultimately the danger of people in power (e.g. law enforcement, military officials) treating its output as valid and unbiased, or simply laundering their wishes through it.
chasd00•1h ago
sublinear•57m ago
The broader public is just now barely beginning to understand because all they have to do is ask a chatbot. AI does not enable new capabilities, but it does aggregate an idea into a rough sketch and do it quickly on-demand.
None of this really means it will play out that way. The devil is in the details. What it does mean is much more nuanced attention on the politics and money because that's where the power always was.
detectivestory•46m ago
maplethorpe•1h ago
iugtmkbdfil834•1h ago
And I am saying this as a person who actually likes this tech.
elar_verole•1h ago
Obviously, they still overhype and oversell this end of humanity stuff, but this argument regurgitated ad-nauseam is not THAT great of an example when you think about it.
mofeien•15m ago