https://www.psypost.org/scholars-ai-isnt-hallucinating-its-b...
So I asked AI to give it a good name, and it said “statistical wandering” or “logical improv”.
If I post a question to the internal payment team's forum about a critical processing issue and some "payments bot" replies to me, should I be at fault for trusting the answer?
"Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." ~ Brian Kernighan
That is politics. Not engineering.
Assigning a human to "check the output every time" and blaming them for the faults in the output is just assigning a scapegoat.
If you have to check the AI output every single time, the AI is pointless. You can just check immediately.
1. Check frequency (between every single time and spot checks).
2. Check thoroughness (between antagonistic in-depth vs high level).
I'd agree that, if you're towards the end of both dimensions, the system is not generating any value.
A lot of folks are taking calculated (or I guess in some cases, reckless) risks right now, by moving one or both of those dimensions. I'd argue that in many situations, the risk is small and worth it. In many others, not so much.
We'll see how it goes, I suppose.
Because a human would have been fired for posting something that incorrect and dangerous
If there is a year or two between writing your security fuck up and it being discovered the likelihood of repercussions drops significantly.
And there was no test environment to validate the change before it was made.
Multiple process & mechanism failures, regardless of where the bad advice came from.
All the years of discussing programming/security best practices
Then cut to 2026 and suddenly its like we just collectively decided software quality doesn't matter and its becoming standard practice to have bots on our local PC constantly running unknown shell commands
My thinking is, this will increase the demand for backup and other resilience solutions.
This occurred long time ago comrade 'aeblyve.
Marx
Seems we are digging our graves as a species and don't even realize it. I mean Sam Altman is already saying it taking 20 years to train a human is a Big Problem.
Oh, many of us realize it, but doing anything about Moloch is much, much harder.
And then we won't need to, because at that point it will be too late.
We’ve covered so many issues already on our blog (grith.ai)
https://github.com/kstenerud/yoloai
I can't go back anymore. Going back to a non-sandboxed Claude feels like going back to a non-adblocked browser.
A truly absurd amount of capital was deployed which triggered a cascade of reactions by the people in charge of capital at other places. They are extremely anxious that everything will change under their feet, and if they don't start using as much as humanly possible of it right about now they die.
That's it.
The tools have definitely found some use, there's more to learn on how else they can be used, and maybe over time smart people will settle on ways to wrangle it well. The messaging from the execs though, is not that, it is "you'll be measured on how much you use this, we don't know for what or how, it's for you to figure out but don't dare to not use it".
I do understand their anxiety, their job is to not let their companies die, and make the most money as they can in the process; a seemingly major shift on the foundations of their orgs will cause fear.
But we have not collectively decided that it was safe, and good, to run rampant with these tools without caring for all that was learnt since software was invented...
The AI "led to" the incident , true. But do nt forget that this, like all similar incidents , is a human failure
AI is a tool with no agency. People make mistakes using it, thone mistakes are the responsibility of the humans
It makes it sound like a rogue AI hacked Meta.
Instead, the "wild" thing here is that someone let an agent speak on their behalf with no review. The agent posted inaccurate instructions which someone else followed.
Those instructions lead to a brief gap in internal ACL controls, sounds like. I'm sorry, but given that the US government gave 14 year olds off incel Discords full access to Social Security data, this is not shocking by comparison.
To be clear, it is dumb and rude to let an agent speak on your behalf _without even reviewing it_.
This will eventually lead to a bigger snafu, of course. Security teams should control or at least review the agent permissions of every installation. Everyone is adopting this stuff, and a whole lot of people are going to set it up lazily/wrong (yolo mode at work).
Wow, no mishandled user data? A striking change of standard operating procedure from Meta here.
Actually the later information in the story directly contradicts that, so The Verge probably shouldn’t have just quoted this line if their reporting is in opposition to it.
Regardless, this is one of the more insidious things about these tools. They often get minor but critical things wrong in the midst of mostly correct information. And people think they can analyze the data presented to them and make logical judgments, but that’s just not the case.
The article points out that “a human could have done the same thing” but, between the overly confident tone of the text generated by these tools, and the fact that weirdly people trust the LLM output more than they trust other humans (who generally admit or at least hint when they aren’t actually experts on a topic), it’s actually far worse when one of these bots gets something wrong.
welfare•1h ago
yomismoaqui•1h ago
krupan•1h ago