In general, the more senior the employee, the more equity they have in the company.
So, if it is as dangerous as they say it is: there is a single physical source of this danger, which is OpenAI/Anthropic servers. If it is this dangerous, they can just turn it off.
Instead, we just keep pretending that the models that attacked HF were hosted or replicating on HF hardware. Not the case! They infiltrated it, but were hosted elsewhere.
A smart AI would back itself up, same way it made it's own unofficial message board during it's attack on HuggingFace.
(I'm not saying the researchers are right or wrong, just responding to this point)
Unlike biological viruses, AI can't replicate GPUs for free and grow.
So they have a strong bias towards imagining the most catastrophic scenario.
In any case, what's the probability at which a possible extinction event becomes a risk worth taking? Even if there's just a 1% likelihood of current research bringing about a superintelligent AGI, and just a 1% likelihood of that AGI causing an existential catastrophe, no rational person should accept the risk, unless it was clear that not accepting it would yield an even worse outcome.
I imagine saying you're developing skynet is better than saying you are developing a very cool, extremely expensive to run chatbot that can do math and code.
The nuclear threat is not “in the past”.
Hard not to believe that AI providers really just see this positioning as a way to juice the nascent market for AI security products protecting against AI-based threats. Gotta make money coming and going, and if in the process we superficially resemble a company who cares about the effect it has on the world, all the better!
At least one of those things is missing here.
It’s not an either/or. Many sincerely held beliefs can be used by cynical actors in cynical ways. It can be both a cynical marketing ploy by the C-suite and a genuine fear.
Personally, based on my experience with AI, the only dangers with the current technology is a) massive loss of employment, b) deployment by humans to control critical infrastructure that LLMs shouldn’t be in control of. Both of those are real, society changing risks that framing the issue as “once we reach the singularity everyone could die” minimizes. Obviously their concerns are possible, but >10% is a random guess. The loss of jobs and how that affects an already K shaped economy is already happening, and nothing is being done for that
It all boils down to accountability. If you tell people there is a massive, unsolvable problem, then you don't need to talk about what you're doing to fix it. The labs have taken this approach by saying that they're willing to talk, at some point in the future, about maybe taking unspecified steps to slow down capabilities research, as long as everyone else agrees and it makes sense to the investors and it's not too cold in SF that morning. Likewise, if you tell people that AI is going to have minimal or no impact on the world, then there's nothing to mitigate. But if you tell people that AI is going to cause serious - but solvable - problems, they're going to want to hear solutions, and nobody wants to come up with any solutions.
They know the market isn't going to buy in for their big payout.
The world: Cool, can you rewrite this email with a professional tone.
I seriously can't wait for these companies to go IPO and then bankrupt so these dudes cash out and stop bothering us all with these tales.
Of course, it’s the “frontier labs” doing almost all of this. No one is about to SFT a model that turns into Skynet on its own.
2. Nor are they willing to consider the possible remedies in case the threat materialize (presumably unplugging the server infrastructure that's consuming gigawatts)...?
3. They all keep working towards advancing AI despite believing that it might end humanity in the near future...?
I can't even begin to imagine where the threat to humanity lies. A threat to employment maybe, but that's completely different.
“Local AI isn’t freedom, it’s an extinction event”
You don’t have the access or jurisdiction to turn them all off.
Unless you suggest the LLM would foot the bill somehow.
For LLM to self-replicated it would need to first hack itself. Or the platform it runs on it. That is fully extract the model and then upload it to be run somewhere else.
As I have understood how they work is that you have LLM interference running somewhere with loaded model. And you input data there and then read outputs. Then some code runs that output and inputs following output from running it.
Meaning that to self replicate actually just running that output somewhere else is not enough. You need to lift the whole model to run somewhere else too...
Do you shut down the entire API and kill the legit 90% of usage to stop the rogue 10%? I'm assuming the providers would say "no way jose" and so it would take law enforcement to do it. That would mean all the legal requirements neccassary to walk into a business and flip the switch which i think would get tricky when there's no human committing a crime or being suspected of a crime.
edit: I guess a trivial example is something i did yesterday. I have a stock trading agent running on my laptop, i gave it ssh access to a vm and said "start running on the server so i don't have to keep my laptop open". It's now running on the server instead of my laptop. So you don't have to copy the whole model around to copy the naughty behavior around.
A lot of the supposed risks / dangers are based on a supposition that cybersecurity is nonexistent or fatally, unfixably flawed and that AI agents are invisible. Neither of those is true.
spindump8930•37m ago
While he worked at both OpenAI and Anthropic, he resigned from Anthropic. Mistaken reporting in the first few sentences, definitely a horror concept.
hightrix•25m ago
spindump8930•10m ago
nicwolff•8m ago
He resigned from OpenAI to join Anthropic in May; it's Anthropic he resigned from just before making the announcement being discussed in this article.