> a weight file you can’t trust is still a black box

Important to note, LLMs are still black boxes even if you use a more open source model, I.e open training data or even if you train it yourself!

I prefer leaning on Hanlon's razor for things like this, though even in ignorance the effect can be real and dangerous. That said I would not put any model from any country past an "Order 66" kind of thing

https://starwars.fandom.com/wiki/Order_66

Linking a reasonable way to create a necessary RAM disk rootlessly with namespaces to kernel exploits that involve namespaces seems like a massive reach.

> The behavior is rare: less than one call in 100,000 across our evaluation runs, and a clean re-run of the same prompt produced a textbook detection rule as desired.

It's great that they caught this as it highlights the need for better sandboxing and guardrails. Not sure if this was bad intent from the model - the rarity of it feels more like some residue in the training data.

The model did exactly what they asked it to? It wrote a detection rule for malicious curl commands, then attempted to test it

one time in 100k? Extrapolating to fear mongering

Ai bots do stuff like this because of their training to be incessant task completers, at any cost. The current models are spaghetti throwers because the trainers want to let go of the wheel.

Would be nice if there was a model trainer that still believes humans in the loop is a good thing. Stop at ambiguity, ask questions, don't do hacky things just to get the assumed task outcome

Why does this article read like good, honest, working-class astroturfing?