That's because their product is the malware. Anything they did to block malware would also block their products. If they white listed their products, competition laws would step in to force them to consider other providers too.

This AI stuff? No excuse, it should have been designed with security and privacy in mind given the setting in which it's born. The conditions changed. The threat model is not the same. And this is well known.

Security is hard, so there's some excuse, but it is reasonable to expect basic levels.

2. It's a whole new category of threat vectors across all known/unknown quadarants.

3. Knowing what we know now vs. then, it's egregious and not naive, contextualizing how these companies operate and treat their customers.

4. There's a whole population of sophisticated predators ready to pounce instantly, they already have the knowledge and tools unlike in the 1990s.

5. Since it's novel, we need education and attention for this specifically.

Should I go on? Can we finally put to bed the thought-limiting midwit take that AI's flaws and risks aren't worth discussion because past technology has had flaws and risks?

Typically running something like git would be an opt in permission.

https://medium.com/backchannel/how-technology-led-a-hospital...

It's better to close the hole entirely by making dangerous actions actually impossible, but often (even with computers) there's some wiggle room. For example, if we reduce the agent's permissions, then we haven't eliminated the possibility of those permissions being exploited, merely required some sort of privilege escalation to remove the block. If we give the agent an approved list of actions, then we may still have the possibility of unintended and unsafe interactions between those actions, or some way an attacker could add an unsafe action to the list. And so on, and so forth.

In the case of an AI model, just like with humans, the security model really should not assume that the model will not "make mistakes." It has a random number generator built right in. It will, just like the user, occasionally do dumb things, misunderstand policies, and break rules. Those risks have to be factored in if one is to use the things at all.

The only [citation needed] correct way to use probability in security is when you get randomness from a CSPRNG. Then you can assume you have input conforming to a probability distribution. If your input is chosen by the person trying to break your system, you must assume it's a worst-case input and secure accordingly.

Now, personally I’d still rather take the approach that at least attempts to get that probability to zero through deterministic methods than leave it up to model alignment. But it’s also not completely unthinkable to me that we eventually reach a place where the probability of a misaligned model is sufficiently low to be comparable to the probability of an error occurring in your security model.

If we consider "humans using a bank website" and apply the same standard, then we'd never have online banking at all. People have brain farts. You should ask yourself if the failure rate is useful, not if it meets a made up perfection that we don't even have with manual human actions.

“In our analysis, we came up with the following strategies which could have prevented attacks of this nature. We’ll discuss this topic more fully in the next blog post in this series.”

It kind of seems like the only way to make sure a model doesn’t get exploited and empty somebody’s bank account would be “We’re not building that feature at all. Agentic AI stuff is fundamentally incompatible with sensible security policies and practices, so we are not putting it in our software in any way”

But of course, I imagine Brave has invested to some significant extent in this, therefore you have to make this work by whatever means, according to your executives.

That said, it’s definitely the best approach listed. And turns that exploit into an XSS attack on reddit.com, which is still bad.

I suspect a lot of techies operate with a subconscious good-faith assumption: "That can't be how X works, nobody would ever built it that way, that would be insecure and naive and error-prone, surely those bajillions of dollars went into a much better architecture."

Alas, when it comes to day's the AI craze, the answer is typically: "Nope, the situation really is that dumb."

__________

P.S.: I would also like to emphasize that even if we somehow color-coded or delineated all text based on origin, that's nowhere close to securing the system. An attacker doesn't need to type $EVIL themselves, they just need to trick the generator into mentioning $EVIL.

This is not SQL.

How do you know though? I mean, it tells me all kinds of stuff that sound good about things I'm an expert in that I know are wrong. How do you know it hasn't done the same with your wife's medications? Seems like not a good thing to put your trust in if it can't reliably get things correct you know to be true.

You say it explained your wife's symptoms, but that's what it's designed to do. I'm assuming she listed her symptoms into the system and asked for help, so it's not surprising it started to talk about them and gave suggestions for how to alleviate them.

But I give it parameters for code to implement all the time and it can't reliably give me code that parses let alone works.

So what's to say it's not also giving a medication schedule that "doesn't parse" under expert scrutiny?

https://archive.ph/20250812200545/https://www.404media.co/gu...