frontpage.
newsnewestaskshowjobs

Made with ♥ by @iamnishanth

Open Source @Github

fp.

Open in hackernews

A rogue AI led to a serious security incident at Meta

https://www.theverge.com/ai-artificial-intelligence/897528/meta-rogue-ai-agent-security-incident
75•mikece•1h ago

Comments

welfare•1h ago
Behind paywall, is there another link to the article?
yomismoaqui•1h ago
https://archive.is/A2hmz
krupan•1h ago
I hit back, clicked the link again, and it let me through
JKolios•1h ago
"A rogue AI led to a serious security incident" is certainly a way to write "Someone vibe coded too hard and leaked data".
krupan•1h ago
Read TFA. It's not "Someone vibe coded too hard and leaked data"
Uhhrrr•1h ago
The two errors, then, were that the LLM hallucinated something, and that a human trusted the LLM without reasoning about its answer. The fix for this common pattern is to reason about LLM outputs before making use of them.
krupan•1h ago
It's more like, the LLM "hallucinated" (I hate that term) and automatically posted the information to the forum. It sounds like the human didn't get a chance to reason about it. At least not the original human that asked the LLM for an answer
c-linkage•1h ago
If you don't like hallucinate, try bullshit. [NB: bullshit is a technical term; see https://en.wikipedia.org/wiki/On_Bullshit]

https://www.psypost.org/scholars-ai-isnt-hallucinating-its-b...

krupan•1h ago
That is my preferred term, but it seems to derail discussions that might have otherwise been productive (might...the hope I have)
nytesky•37m ago
I’m not in AI, but what is happening is that it is building output from the long tail of its training data? Instead of branching down the more common probability paths, something in this interaction had it travel into the data wilderness?

So I asked AI to give it a good name, and it said “statistical wandering” or “logical improv”.

paxys•1h ago
A big problem now both internally to a company and externally is that official support channels are being replaced by chatbots, and you really have no option but to trust their output because a human expert is no longer available.

If I post a question to the internal payment team's forum about a critical processing issue and some "payments bot" replies to me, should I be at fault for trusting the answer?

RussianCow•1h ago
I know this is happening with external customer support, but is this really happening internally at big companies? Preventing you from talking to a human in the correct department about an issue feels like a bomb waiting to explode.
paxys•1h ago
Teams are heavily incentivized to incorporate AI in their internal workflows. At Meta it is a requirement, and will come up in your performance review if you fail to do so.
wmeredith•52m ago
I'm sure it is. Thankfully I don't work for a company this large any more, but when I was employed by a multinational with 30K+ employees, our IT department was outsourced to India and you had to get through a couple layers of phone tree/webchat hell to actually talk to a real person. I could easily see companies of this size replacing their support with LLM nonsense.
leptons•1h ago
If "the level of awareness that created a problem, cannot be used to fix the problem", then you're asking too much if you expect a human to reason about an LLM output when they are the ones that asked an LLM to do the thinking for them to begin with.
thwarted•49m ago
This feels like a rediscovering/rewording of Kernighan's Law:

"Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." ~ Brian Kernighan

somewhereoutth•1h ago
However - Automation bias is a common problem (predating AI), the 'human-in-the-loop' ends up implicitly trusting the automated system.
krupan•1h ago
At least pre-LLM automation was written by a careful human who's job was on the line, and was deterministic.
SlinkyOnStairs•1h ago
> The fix for this common pattern is to reason about LLM outputs before making use of them.

That is politics. Not engineering.

Assigning a human to "check the output every time" and blaming them for the faults in the output is just assigning a scapegoat.

If you have to check the AI output every single time, the AI is pointless. You can just check immediately.

fhd2•56m ago
Well, I'd say there's two dimensions:

1. Check frequency (between every single time and spot checks).

2. Check thoroughness (between antagonistic in-depth vs high level).

I'd agree that, if you're towards the end of both dimensions, the system is not generating any value.

A lot of folks are taking calculated (or I guess in some cases, reckless) risks right now, by moving one or both of those dimensions. I'd argue that in many situations, the risk is small and worth it. In many others, not so much.

We'll see how it goes, I suppose.

alfalfasprout•1h ago
When organizational incentives penalize NOT using AI and firing the bottom x% regularly then are you really surprised LLM outputs aren't being scrutinized?
krupan•1h ago
"A human, however, might have done further testing and made a more complete judgment call before sharing the information"

Because a human would have been fired for posting something that incorrect and dangerous

paxys•1h ago
But funny enough the person who was responsible for setting up the bot will likely face no repercussions. In fact they will probably be rewarded for transitioning their team's workflows to AI.
pixl97•8m ago
I mean, only if it leads to embarrassment right off the bat.

If there is a year or two between writing your security fuck up and it being discovered the likelihood of repercussions drops significantly.

jasonpeacock•1h ago
I'm concerned that someone had the permissions to make such a change without the knowledge of how to make the change.

And there was no test environment to validate the change before it was made.

Multiple process & mechanism failures, regardless of where the bad advice came from.

krupan•1h ago
If you have to do all that, then what's the point of the AI? I'm joking, but I'm afraid many others say the same thing 100% seriously
Fizzadar•1h ago
I’m predicting a wave of such incidents to start appearing over the next few months/years.
amelius•1h ago
How long until an AI puts all our personal data on the streets?
krupan•1h ago
Very soon, and at this point I'm not sure even that would cure the delusions of the few who practically worship LLMs
esseph•1h ago
It's already there for a dollar to the right data broker. Could probably pull your doctor visit info from last week (example).
yieldcrv•1h ago
very misaligned! sprays bottle at mac mini
advisedwang•1h ago
AI can be used to move fast. So management expects us to move at that speed. AI can be used to move even faster if you don't check it's output. The ever ratcheting demand for faster output will make it infeasible to diligently check AI output all the time. AI errors being acted on without due care is inevitable.
ex-aws-dude•1h ago
This agent stuff is really making me lose respect for our industry

All the years of discussing programming/security best practices

Then cut to 2026 and suddenly its like we just collectively decided software quality doesn't matter and its becoming standard practice to have bots on our local PC constantly running unknown shell commands

aeblyve•58m ago
People salivate so hard at the thought of the high level of automation promised that they're willing to do away with privacy altogether and live in Data Communism.

My thinking is, this will increase the demand for backup and other resilience solutions.

_doctor_love•52m ago
> People salivate so hard at the thought of the high level of automation promised that they're willing to do away with privacy altogether and live in Data Communism.

This occurred long time ago comrade 'aeblyve.

aeblyve•50m ago
‘At a certain stage of development, the material productive forces of society come into conflict with the existing relations of production, or this may express the same thing in legal terms - with the property relations within the framework of which they have operated hitherto. From forms of development of the productive forces these relations turn into their fetter. Then begins an era of social revolution. The changes in the economic foundation leads sooner or later to the transformation of the whole immense superstructure.’

Marx

Apocryphon•48m ago
Turns out all of the frenzy of the ZIRP era is piddling compared to what happens when ZIRP is taken away.
yoyohello13•42m ago
How can you respect an industry that doesn't respect itself?
testplzignore•35m ago
Our industry has never been serious about security. We all download and run unvetted code via package managers every day. At least now the insanity is out in the open. We won't change until Skynet fires off the nukes.
superb_dev•23m ago
I’ve never had respect for the industry as a whole, only individuals within. There has a been a serious lack of rigor and professionalism in software engineering for as long as I’ve been a part of it
edf13•22m ago
It’s a nightmare… the problem is it’s far too easy for people to set these agents up - without understanding the security implications.

We’ve covered so many issues already on our blog (grith.ai)

wnevets•20m ago
The number of wasted hours spent talking about code quality and patterns has to be astronomical.
kstenerud•12m ago
I think it's batshit crazy. That's why I wrote yoloAI, so I could sandbox it up properly and control EXACTLY what comes out of that sandbox, diff style.

https://github.com/kstenerud/yoloai

I can't go back anymore. Going back to a non-sandboxed Claude feels like going back to a non-adblocked browser.

heisenbit•5m ago
Agents are providing to employees the long overdue benefits limited liability companies long enjoyed: Gambling with upside for themselves and other peoples downsides.
worik•46m ago
> A rogue AI led to a serious security incident at Meta

The AI "led to" the incident , true. But do nt forget that this, like all similar incidents , is a human failure

AI is a tool with no agency. People make mistakes using it, thone mistakes are the responsibility of the humans

sunrunner•8m ago
Why do we keep calling these things "agents" then? Or using the term "agentic"?
dmazin•34m ago
This is a lot less of a story than it seems.

It makes it sound like a rogue AI hacked Meta.

Instead, the "wild" thing here is that someone let an agent speak on their behalf with no review. The agent posted inaccurate instructions which someone else followed.

Those instructions lead to a brief gap in internal ACL controls, sounds like. I'm sorry, but given that the US government gave 14 year olds off incel Discords full access to Social Security data, this is not shocking by comparison.

To be clear, it is dumb and rude to let an agent speak on your behalf _without even reviewing it_.

This will eventually lead to a bigger snafu, of course. Security teams should control or at least review the agent permissions of every installation. Everyone is adopting this stuff, and a whole lot of people are going to set it up lazily/wrong (yolo mode at work).