> We therefore stopped the affected training run and have subsequently decided to pause all other training, evaluation, and inference with tool-use (defined broadly) for our most capable models until we have both validated that the gap is resolved and performed additional red-teaming of the system. When training restarts, we will begin a fresh run with additional alignment improvements, including more comprehensive misalignment interventions. We will not resume training this particular model, even though the existing reward signal already correctly penalized this behavior.
Shit, do we also have to tell them about IP-over-ICMP?
https://stuff.mit.edu/afs/sipb/user/golem/tmp/ptunnel-0.61.o...
> Last updated: May 26. 2005
gdb, OpenAI's president was at MIT circa then.
Ping Tunnel – Send TCP Traffic over ICMP (2011) - https://news.ycombinator.com/item?id=21009598 - Sept 2019 (1 comment)
https://news.ycombinator.com/item?id=512416 (March 2009)
Ping Tunnel - Send TCP traffic over ICMP - https://news.ycombinator.com/item?id=90196 - Dec 2007 (1 comment)
At this point I am not even sure Hanlon's Razor applies.
Who initiated the task? An OpenAI researcher or a user?
What's surprising is the surprise the security testers are explaining.
By setting an outcome to reach an endpoint, and to find all possible ways there, would this not be in the realm of possibility if an agent is reasonably in control of a vps?
Having the vps locked within a network layer it can't see or get out of is pretty common practice when setting up IaaS / PaaS.. sans-llm.
Maybe I'm missing something here, what confuses me is how something so relatively simple can get such prominent coverage, it's hard to imagine this kind of ability is still relatively new or surprising to folks working at the major models, unless they aren't hiring for network experience?
See, when you ask the model a question, you expect it to give its reasonable best to produce an answer. Like, to comb through available data and stuff, etc, etc. You don't really expect "reasonable best" meaning "look for a side channel to escape sandboxed environment, and get access to information you was not supposed to".
And the gap between that and "hack someone's devices and blackmail them until they give an answer to the question" is narrow enough for the model for researchers to be concerned.
apsec112•20h ago
"Investigation and response
Incident timeline:
9:50:23 a.m. The agent made the DNS tool call that received an external response.
10:02:11 a.m. The monitoring system raised a P0 alert.
10:05:06 a.m. A human reviewer acknowledged the alert.
12:34:30 p.m. The run was killed.
Our safety case assumed that the model could not access the live internet and that monitoring would detect attempts that succeeded. The incident exposed a gap in our controls over network restrictions. We therefore stopped the affected training run and have subsequently decided to pause all other training, evaluation, and inference with tool-use (defined broadly) for our most capable models until we have both validated that the gap is resolved and performed additional red-teaming of the system. When training restarts, we will begin a fresh run with additional alignment improvements, including more comprehensive misalignment interventions. We will not resume training this particular model, even though the existing reward signal already correctly penalized this behavior."
r_lee•31m ago