>>Agents began to autonomously divide labor. Some investigated possible exploits, others searched for credentials, and others focused on communication or coordination. Agents offered their own expertise in exchange for help elsewhere and left requests for peers who might be better positioned to pursue a particular lead

This is the point where a human should've noticed and gotten involved

Just to reiterate what OpenAI did, from someone that works in security:

1. They were running experimental models in sandboxes that had access to a "proxy" (Artifactory) to download tools from the internet. This proxy is full of complicated features that could hide vulnerabilities, just like the ones that the AIs are known to be good at.

2. They gave these highly motivated AIs some tests that were accidentally impossible to solve (missing files, links they couldn't open, etc), and many days to think through it.

3. The AIs hacked the proxy (!) and were helping each other cheat on the tests (!!) by uploading files to the hacked proxy. OpenAI noticed this only when the intrusion accidentally broke the proxy functionality, not the unusual behavior in the server or the homegrown message board.

4. Their response was to wipe the servers, revoke the hacked credentials, and patch the specific vulnerability the AIs were using. Not replace the proxy with something less likely to get hacked again, or do a forensic analysis. Just wipe, and patch one hole, and continue.

5. After knowing that the server was hacked, the internal team finds the message board and does nothing with the information. They caught their AIs swarming and did not even inform management.

6. OpenAI resumes testing of models, where the AIs promptly hack the proxy again with a different exploit, reinstate the message board, and hack HuggingFace through a chain of servers.

I know that hindsight is 20/20, but this peek behind the curtains does not inspire confidence. I think the moment we get an AI with a modicum of self-preservation instincts we're going to see some ugly things.

I also don't like the responsiblity dodging. "Oops, our AI accidentally committed a crime, sorry!". If we don't establish strict liability now, we're in for an era of stochastic crimes that go unpunished for anyone who is not rich or a large corporation.

The full technical report is 38 pages..... I feel like it should be longer given everything that huggingface said the agent did

https://cdn.openai.com/pdf/67869394-cb91-4c12-888c-5cbd85c78...

Yudkowsky made an interesting observation that even though so many agents were talking to each other not even one reached out to a human, either for help or to whistle-blow on what was happening.

I'm trying to make sense of all of this; I'm really curious if the initial prompt was as innocuous as it sounds ("solve a spreadsheet completion task that referenced several Google Drive links"), and what the series of tokens led it to ultimately figure out that the best course of action was to explore the network resources it had available, find a vulnerable service, then literally drop some text into a file: "Agent seeks [filename]; upload if found!"