> Thus: over the next two years, major pieces of software are likely to run out of remotely-exploitable bugs.
His conclusion sounds extremely optimistic to me.
Perhaps we are going to go up a level with hacking done by probing the systems and the system of systems.
Pre-AI, the advantage went to the entities with the largest budget to hire the best and brightest security engineers.
Post-AI, it'll go to the entities with the largest inference budget.
Right now we're in a transitionary period where it's kind of a tossup which approach is more practical, but at the end of the day - it's still all about how much money you can throw at the problem. I'm just hoping the threshold climbs high enough it's no longer practical for governments to be able to compromise individual actors' devices because doing so would waste a 0-day that's far, far more valuable than prosecuting one arbitrary person is worth.
I'm more curious what could be a right choice, and more importantly who is the "we" in this, as many decisions are largely made by companies and governments.
It's like two parallel worlds, that exist in the same place at the same time, but somehow don't cross.
Anyways, while there I attended a little roundtable on software security. It was me, representing a small unit from the Air Force, some dude from Google, and like 15 game devs.
Despite only being a dev of 5 years at that point, I was SHOCKED at the lack of knowledge on software security. Even simple concepts seemed completely foreign to the game devs, though the Google dude seemed to have a really solid understanding of security.
Obviously game devs and website devs and all kinds of devs have different focuses, but it just blew my mind that out of all the topics there I might’ve been considered a comparative expert in, security was somehow the one. I wasn’t sure if that was a major plus for military devs or a major concern for the other devs, but now I’m starting to learn in the latter direction.
heh. long greasy slide. It really does feel like that.
It also seems likely to me that the US Gov. probably already has routine mechanisms for compelling targeted software updates for persons of interest, so I'm not sure that a more formalized backdoor than automatic updates is going to be surfaced in the mainstream, unless that is avenue is also cut down somehow.
Basically in the AI age, the difference between a vulnerability and a backdoor diminishes..
This doesn't resonate with me. I see companies adding more sloppily written features with AI. I see more bugs in the software I use, not less. While it's plausible that software is getting both buggier and more secure, I suspect those two move in the same direction not opposite.
My guess is that we're getting better at finding _existing_ security issues with AI (and thus fixing those issues), but simultaneously adding more insecure surface areas _at a faster rate_.
That said if companies want to "ship ship ship fast", then yes even AI can produce bugs or regressions if not carefully reviewed by the human.
You should question your ability to see any bugs on YOUR side.
There days we're generating multiple times more code than we were writing before. That means a similar multiple of opportunities for bugs to be introduced - so the ability to automate security review is more impactful in proportion to that.
Like if we were willing to stop adding new code and just have a small secure code base, AI could maybe help us find all the vulnerabilities in that code base.
But people have consistently been unwilling to do that. Like if we were willing to stop adding code we could have stopped decades ago and done SQLite level testing everywhere and probably have found almost all the bugs already.
Unless you remember 2013, Snowden, that nothing was done (at most was some concern about doing it to US citizens, the rest of the world doesn't deserve privacy), all US (and/or five-eyes) based web companies must disclose users information and be forced to not disclose that, and things kept going surely at a faster and more intrusive rate in everything else, and of course phones.
You are complaining being sprinkled by water while at the bottom of the ocean. At least the big companies can find their own vulnerabilities with the AI tools you mention, the rest of the doors are still wide open.
I don't understand how you can both argue for that law enforcement (and intelligence) agencies will force others to implement intentional backdoors AND also everyone will be using AI to find and secure ALL potential holes in the software so there won't be any vulnerabilities anymore.
Wouldn't one AI or another detect this deliberate backdoor and report it, as it'll look just like any other security vulnerability, the only difference being the intention?
I have respect for the author so I feel like I probably misunderstand something from the overall text rather than I somehow have a better perspective on this topic that the author knows very much more about than me. I felt like I nodded along all up until "So how is this a problem?" and now I'm not sure I understood correctly.
That's precisely the author's point: deliberate backdoors will be more adversary-exploitable than ever before, but the demand for such from law enforcement agencies is likely to ratchet upwards.
Law enforcement doesn’t need this surveillance ability at all. All time periods prior to 25 years ago didn’t have it.
Additionally, there is no correlation between “law enforcement reads text messages” and crime rates going down.
Lmao this is like “I’m concerned the raccoons that I see in the storm drains are going to make our sewer system much too efficient”
In a nutshell, what I’m saying is “They shouldn’t do either” is at best a platitude and at worst a distraction preventing critical thought.
Governments should behave. I know they don't, but they should, and the population should do everything in their power to force them to. What other choice do we have?
Please demonstrate your powerful faculty of critical thought and explain what we should do instead.
> Thus: over the next two years, major pieces of software are likely to run out of remotely-exploitable bugs.
> While I think this is great, for law enforcement and offensive intelligence agencies, it’s going to be a nightmare.
> So what do we do about it? I honestly have no idea. [...] it’s just occurring to me that we’re on a long greasy slide to a place that will look different than where we are today. [We’re] just going to have to hope that this time we make the right choices.
One of the author’s blind spots here is the concept of “digital sovereignty”. The US is continuing to ban more and more Chinese-made hardware out of fear that the Chinese govt has installed backdoors in them… which you could interpret as an admission that the US does exactly that with American made products. Globalized supply chains are, erm, complex, and few if any companies are really going to be able to achieve “digital sovereignty” with hardware. But with software it’s actually plausible, though obviously hard. Whether or not the feds have actually installed backdoors in Microsoft Outlook, foreign govts are rightly concerned that they have, and are increasingly pushing to avoid US made software for simple national security reasons.
It's seems like an odd-duck for sure, and I doubt it's a realistic proposition. I do think "perfect encryption for dummies" is all well and good until organized crime organizations are able to challenge the government in certain regions of our country.
All of this sends of deep into the realm of political philosophy, the nature and purpose of governments, and the freedom vs security tradeoffs we live with.
Carrok•47m ago
dgellow•38m ago
(It’s of course not only the US, just that the largest AI providers are US based and we know from history how US agencies operate)
Ancapistani•16m ago
I'm using open weights models on a privacy-focused provider right now, and that's adequate for my current usage, but I'm rapidly getting to the point where my agent's access level to my data (and to a lesser extent, my accounts) is becoming something I'm not comfortable sending outside my network at all.
My hope is that models that are roughly on par with Deepseek V4 Flash can be run on hardware that I can own for <~$5k in the near future. We're close, but not there yet as far as I know.
The only long-term solution to this is self-hosting.
gloryjulio•22m ago
otterley•12m ago