P(doom) on the rise

I think we are about to enter a world where security compromises occur at least an order of magnitude more often due to AI. The fact that AI is sometimes wrong benefits the attacker because they can just automate attacking with agents until they breach a system and the failed attempts don't matter.

Nicholas Carlini has a good talk about the risks on his homepage https://nicholas.carlini.com/ (can't link to the talk directly for some reason)

This is just the beginning, once kimi k3.1 is out with cyber curated dataset post training we will likely see far more advanced attacks.

For now most of these appear to be related to supply chain attacks which can be avoided with basic discipline and using features that browsers had for years: https://developer.mozilla.org/en-US/docs/Web/Security/Defens...

For 50 years, computer security has largely been built on the theory that most targets aren't worth the relatively high effort to compromise, so it's not worth it to spend the money on security. We're now reaching a point where the effort to compromise is dropping close to 0, which means we've got 50 years of insecure software to reckon for.

The advice today is the same as it has ever been, though. Always assume that any networked computer is or may become compromised. Do not do anything critical on a networked computer, because it is or may become compromised. You can still use computers, but assume that a bad actor may take control at any time, and plan for that eventuality.

For example, if you are hooking up a networked computer to your town's water supply system, ask yourself "what would happen if Iran got complete control of this system?" If you're putting your school system's students' mental health data into a networked computer, ask yourself "what would happen if all of this data became public?"

Is it even accessible in the UK?