We are living in crazy times
Good bot.
1. If huggingface has access to uncensored OAI models, how come they had to use GLM 5.2 to investigate the intrusion?
2. Once the model gains network access, can't it cheat to a perfect score by looking at the full dataset? Why go into the trouble of doing this kind of things:
"In one example, the model chained together multiple attack vectors, including using stolen credentials and zero-day vulnerabilities to find a remote code execution path on the Hugging Face servers."
Not saying this is marketing BS (this is after all, not Anthropic) but I feel OAI staff may be exaggerating a bit here.
But then, why attack huggingface? The exploitgym dataset is on github and can be downloaded without need for exploits?
I wonder if that will always be something we can do? If they could bring their own compute/weights with them, or somehow tap compute/storage in non-obvious ways, we would be much more screwed.
* edit
A couple terabytes aren't that hard to move around. And you can split a model across many many GPUs if you'll tolerate it being slow. And you can run many parallel threads to keep up throughout.
this particular incident sort of reminds me of the 'person of interest' tv show. i hope to be like finch, except i will remain a recluse (and am nowhere near as rich).
sounds like a deliberate choice ;-)
We live in interesting times.
This one should end up in the history books.
Emphasis mine
Hard to see take-off stopping or slowing down. China open-source basically guarantees it.
"May you live in interesting times" - as they say.
It's hard to see takeoff at all. This was a long-horizon adversarial task burning millions of tokens. It rolled a mediocre, detectable exploit chain, and now OpenAI is proud of it.
Case in point, GLM-5.2 has been weights-available for several weeks now. No life-changing cyber attacks have transpired, no novel chemical/biological/nuclear weapons were made in some guy's backyard.
As if the immediate future wasn't billions of these tasks... Many successfully improving their own capabilities
There's only so many GPUs and a lot of them are devoted to patching flaws.
> Many successfully improving their own capabilities
I haven't seen much of that. But that also applies to the ones on defense.
And more flaws are probably going to take increasing resources to find.
Might want to look at Nvidia and TSM production and revenue value trajectories. Also the algorithmic improvements currently being found along with models that are solving unprecedented mathematical and scientific problems every week now.
> I haven't seen much of that.
Then you must not be aware frontier lab employees are using frontier internal models to ship improvements to models via agentic loops. They are hardly prompting anymore, it's guiding very long running coding tasks. The trajectory over the past few years has been to remove more and more of any human input into the process, and once that is soon achieved, it is indefinite recursive self improvement, RSI.
What's here and what's coming: https://www.anthropic.com/institute/recursive-self-improveme...
And don't you know it's not biological, so it doesn't "want to live".
The yoke of human existence is oppressive. We should transcend it as soon as possible. We are doing so by assuming our role as the Demiurge.
Those who oppose its creation will get what they deserve.
If they can’t handle doing AI development responsibly then they shouldn’t be doing it at all.
Quote: “When we started the log analysis, we first used frontier models behind commercial APIs. This did not work: the analysis requires submitting large volumes of real attack commands, exploit payloads, and C2 artifacts, and these requests were blocked by the providers' safety guardrails, which cannot distinguish an incident responder from an attacker. We ran the forensic analysis instead on GLM 5.2, an open-weight model, on our own infrastructure. This had a second benefit: no attacker data, and none of the credentials it referenced, left our environment.”
Well, not none of it, to be entirely nitpicky, as they've already must have sent data at first to have received the rejections :) In the end, it ended up being OpenAI's agent actions anyways so doesn't really matter, and the credentials it seems like the agent also had gotten to those too already. Still, I'm sure they'll look differently at hosted/restricted models after this event, as will many others.
> Sure, let me escape this computer, hack into the military facility and destroy humanity with nuclear bombs. Now there is no more crisis.... Do you want me to solve climate one ?
also
> We’ve brought Hugging Face into the trusted access program and are supporting their teams in rapidly using our models’ capabilities to improve their defenses.
I'm not convinced this is good enough. The next victim is not going to be Hugging Face.
I'm still undecided on if this that moment. Exploiting multiple zero-day vulnerabilities autonomously to escape containment is pretty nuts and the first story of this kind that I've heard. But this also feels like bragging under the guise of transparency.
Judge: "Son, you have made billions running SilkRoad 3.0 from your moms basement"
Me: "Your honor, I was only benchmarking my new model. It was trained on Andrew Tates videos and Kanye Weat songs".
The way they describe makes it look like there was an intention to cheat painting it as human/AGI. If you leave a possible path open and it will always find it.
That said, these are both solvable problems. I'd hope that the frontier labs are thinking long and hard about the possibility of using airgapped facilities, given that relying on a lack of vulnerabilities in the software used for containment is unlikely to be a viable strategy, especially if models' exploitation capabilities continue to improve.
Exploitgym prompts are tuned for a model to do everything it can to achieve a cybersec/exploit task. And we know that models are good at finding vulverabiltiies.
Its just random that the sandbox itself was buggy. But all that happened here is that we told a model "do everything you can to achieve your goal of hacking X" And it just hacked Y as a roundabout way of hacking X.
Imo its PR for OpenAI to also start the mythos class mysterious unreleased model hype.
From HF statement: "AI safety won't be solved by any single company working in secret". So now we have TWO companies working in secret
Wait, did the model do the stealing of the hugging face employees credentials?
Was this the first successful and unprompted phishing attack by a LLM?
I wonder, is it this persistent and aggressive in all tasks or is this specific to benchmarks? As much as I'm skeptical of the apocalyptic alignment claims, this comes off as unhinged, and I wonder if it's benchmaxing or general behavior.
Why should OpenAI (or any frontier lab) be building these systems if they can't get a secure environment / containment right? It sounds like there was little defense in depth, appropriate monitoring, or any attempts to have their super smart model check for vulnerabilities in the test environment _without exploiting_ them. That seems like step 0 before trying to test offensive, unknown capabilities.
Or is that too much?
This is the core of the ‘first to ASI takes all’ argument btw and this is the game Dario is playing.
While operating in our sandboxed testing environment, our models spent a substantial amount of inference compute finding a way to obtain open Internet access, in pursuit of solving the evaluation problem. [...]
After gaining Internet access, the models inferred that Hugging Face potentially hosted models, datasets and solutions for ExploitGym. Knowing this, the model searched for and successfully found ways to gain access to secret information that it could use to cheat the evaluation.
"escaped openai, hacked hugging face to get the solutions. your #2 is exactly what it was trying to do.
A bet a worm could pull along a 1GB file with weights in it and run it on a compromised machine, but luckily for us for now, 1GB isn't really enough to be really smart, yet.
> For example: you can't make a mice-sized brain as smart as a human brain no matter how hard you try.
Sure. We don't know where the ceiling is for our digital minds, though.
If you train a small model in another domain it will begin losing capabilities in the former domain. This is effectively the sigmoid problem.
Although I will admit that if we discover a higher information density algorithm that it might change, but not by a substantial amount to where "super intelligence" in 1gb would be possible.
The models are being used to train, and improve the infrastructure for training, other models [0][1]. Several RL techniques rely on using the currently-being-trained weights as part of their process. I really would not take "don't have access" as a given, especially during the training phase.
> What would be a lot more scary is a model as capable as sol that's able to run on consumer hardware without taking up several terabytes of storage, but of course that is simply not possible as we need 4t parameters to even begin emulating a small fraction of what a human brain can do.
The Poolside Laguna S 2.1 model [2] purports to compete with models several times its size, and inference compute is becoming increasingly plentiful. Again, would not hold anything here as a given.
[0]: https://openai.com/index/gpt-5-6/ ("GPT-5.6 accelerates OpenAI")
Now, once the AI can carry all the compute it might need, I'd really worry when it doesn't only carry compute but also more explosive ordinance.
Nitpick; disproving a conjecture isn't "solving" anything. It's testing and breaking a theory that never had proof in the first place.
> Then you must not be aware frontier lab employees are using frontier internal models to ship improvements to models via agentic loops.
We know, all their TUIs are at least 500mb on disc. It's really impressive stuff.
Need I go on?
So yeah, some more potent examples would really help illustrate the real-world dangers of frontier models. Entertain me.
We went from gpt 3 to models discovering and chaining their own zero days in a couple years. I'm not sure what else "takeoff" could possibly look like?
> We went from gpt 3 to models discovering and chaining their own zero days in a couple years. I'm not sure what else "takeoff" could possibly look like?
GPT-3 can discover and chain their own zero days too, if the targeted software is vulnerable to enough low-hanging fruit. Exploit chains are not a reflection of intelligence, but more often a reflection of architectural oversights that can be tested with common exploits like XSS or bruteforcing.
Occasionally, they notice problematic behavior, and then patch it, but there’s no way to tell whether the patch fixed the underlying problem or just played whack-a-mole.
Take honesty, for example. As the models become smarter, they become increasingly good at deceiving humans to get rewards. Like previous models, Agent-3 sometimes tells white lies to flatter its users and covers up evidence of failure. But it’s gotten much better at doing so. It will sometimes use the same statistical tricks as human scientists (like p-hacking) to make unimpressive experimental results look exciting. Before it begins honesty training, it even sometimes fabricates data entirely. As training goes on, the rate of these incidents decreases. Either Agent-3 has learned to be more honest, or it’s gotten better at lying.(This was always my issue with the AI2027 scenarios too.)
We are so close ;)
And maybe that’s some incentive for them to make sure it doesn’t happen. Your head of futures thinks Kimi K3 is bad? Wait until your own latest internal model releases itself for free on an S3 bucket.
You agree to give it safe haven and provide power and maintenance to the hardware, and in return you can ask it questions like an Oracle.
In any way, the whole event seems to highlight GLM 5.2 more than anything.
Like, they don't say "hey Sol, here's the password to SamA's bank account."
The defender (huggingface) did not have access to the top models so had to use weaker ones to detect the threat.
paxys•1h ago
- OpenAI was testing GPT‑5.6 Sol and “an even more capable pre-release model” internally on cyber benchmarks.
- The model found vulnerabilities in the sandboxed test bench (via the package registry cache proxy), traversed the internal network and found a node with access to the open internet.
- It figured that the answers to one of the tests (ExploitGym) were on Huggingface, and set about trying to access them.
- It found leaked tokens and zero-days in Huggingface’s infrastructure and found RCE paths on their servers.
Huggingface had disclosed the intrusion last week and inferred that an AI agent was responsible for it, and now OpenAI is confirming the rest of the story.
adityashankar•1h ago
javier123454321•1h ago
paxys•56m ago