ISTR someone else round here observing how much more effective it is to ask these things to write short scripts that perform a task than doing the task themselves, and this is my experience as well.
If/when AI actually gets much better it will be the boss that has the problem. This is one of the things that baffles me about the managerial globalists - they don't seem to appreciate that a suitably advanced AI will point the finger at them for inefficiency much more so than at the plebs, for which it will have a use for quite a while.
It's no different from those on HN that yell loudly that unions for programmers are the worst idea ever... "it will never be me" is all they can think, then they are protesting in the streets when it is them, but only after the hypocrisy of mocking those in the street protesting today.
Unionized software engineers would solve a lot of the "we always work 80 hour weeks for 2 months at the end of a release cycle" problems, the "you're too old, you're fired" issues, the "new hires seems to always make more than the 5/10+ year veterans", etc. Sure, you wouldn't have a few getting super rich, but it would also make it a lot easier for "unionized" action against companies like Meta, Google, Oracle, etc. Right now, the employers hold like 100x the power of the employees in tech. Just look at how much any kind of resistance to fascism has dwindled after FAANG had another round of layoffs..
I guess if managers get canned, it'll be just marketing types left?
But don't count on it.
I mean, apart from anything else, that's still a bad outcome.
[1] https://fortune.com/article/jamie-dimon-jpmorgan-chase-ceo-a...
(Has anyone tried an LLM on an in-basket test? [1] That's a basic test for managers.)
On the other hand, trying to do something "new" is lots of headaches, so emotions are not always a plus. I could make a parallel to doctors: you don't want a doctor to start crying in a middle of an operation because he feels bad for you, but you can't let doctors doing everything that they want - there needs to be some checks on them.
And also where people believe that others believe it resides. Etc...
If we can find new ways to collectively renegotiate where we think power should reside we can break the cycle.
But we only have time to do this until people aren't a significant power factor anymore. But that's still quite some time away.
Our best technology at current require teams of people to operate and entire legions to maintain. This leads to a sort of balance, one single person can never go too far down any path on their own unless they convince others to join/follow them. That doesn't make this a perfect guard, we've seen it go horribly wrong in the past, but, at least in theory, this provides a dampening factor. It requires a relatively large group to go far along any path, towards good or evil.
AI reduces this. How greatly it reduces this, if it reduces it to only a handful, to a single person, or even to 0 people (putting itself in charge), seems to not change the danger of this reduction.
Perhaps because most of the smartest people I know are regularly irrational or impulsive :)
I hear that. Then I try to use AI for simple code task, writing unit tests for a class, very similar to other unit tests. If fails miserably. Forgets to add an annotation and enters in a death loop of bullshit code generation. Generates test classes that tests failed test classes that test failed test classes and so on. Fascinating to watch. I wonder how much CO2 it generated while frying some Nvidia GPU in an overpriced data center.
AI singularity may happen, but the Mother Brain will be a complete moron anyway.
Linda is 31 years old, single, outspoken, and very bright. She majored in philosophy. As a student, she was deeply concerned with issues of discrimination and social justice, and also participated in anti-nuclear demonstrations.
Hm, I'm listening, let's see.
> Software vulnerabilities are caused by mistakes in the code
That's not exactly true. In regular software, the code can be fine and you can still end up with vulnerabilities. The platform in which the code is deployed could be vulnerable, or the way it is installed make it vulnerable, and so on.
> Bugs in the code can be found by carefully analysing the code
Once again, not exactly true. Have you ever tried understanding concurrent code just by reading it? Some bugs in regular software hide in places that human minds cannot probe.
> Once a bug is fixed, it won’t come back again
Ok, I'm starting to feel this is a troll post. This guy can't be serious.
> If you give specifications beforehand, you can get software that meets those specifications
Have you read The Mythical Man-Month?
> these claims mostly hold, but they break down when applied to distributed systems, parallel code, or complex interactions between software systems and human processes
The claims the GP quoted DON’T mostly hold, they’re just plain wrong. At least the last two, anyway.
> I’m also going to be making some sweeping statements about “how software works”, these claims mostly hold, but they break down when applied to distributed systems, parallel code, or complex interactions between software systems and human processes.
I'd argue that this describes most software written since, uh, I hesitate to even commit to a decade here.
If you include analog computers, then there are some WWII targeting computers that definitely qualify (e.g., on aircraft carriers).
He is trying to lax the general public perception around AIs shortcomings. He's giving AI a break, at the expense of regular developers.
This is wrong on two fronts:
First, because many people foresaw the AI shortcomings and warned about them. This "we can't fix a bug like in regular software" theatre hides the fact that we can design better benchmarks, or accountability frameworks. Again, lots of people foresaw this, and they were ignored.
Second, because it puts the strain on non-AI developers. It blamishes all the industry, putting together AI with non-AI in the same bucket, as if AI companies stumbled on this new thing and were not prepared for its problems, when the reality is that many people were anxious about the AI companies practices not being up to standard.
I think it's a disgraceful take, that only serves to sweep things under a carpet.
The fact is, we kind of know how to prevent problems in AI systems:
- Good benchmarks. People said several times that LLMs display erratic behavior that could be prevented. Instead of adjusting the benchmarks (which would slow down development), they ignored the issues.
- Accountability frameworks. Who is responsible when an AI fails? How the company responsible for the model is going to make up for it? That was a demand from the very beginning. There are no such accountability systems in place. It's a clown fiesta.
- Slowing down. If you have a buggy product, you don't scale it. First, you try to understand the problem. This was the opposite of what happened, and at the time, they lied that scaling would solve the issues (when in fact many people knew for a fact that scaling wouldn't solve shit).
Yes, it's kind of different. But it's a different we already know. Stop pushing this idea that this stuff is completely new.
'we' is the operative word here. 'We', meaning technical people who have followed this stuff for years. The target audience of this article are not part of this 'we' and this stuff IS completely new _for them_. The target audience are people who, when confronted with a problem with an LLM, think it is perfectly reasonable to just tell someone to 'look at the code' and 'fix the bug'. You are not the target audience and you are arguing something entirely different.
This also is a misunderstanding.
The LLM can be fine, the training and data can be fine, but because the LLMs we use are non-deterministic (at least in regard to their being intentional attempts at entropy to avoid always failing certain scenarios) current algorithms are inherently by-design not going to always answer every question correctly that it potentially could have if the values that fall within a range had been specific values for that scenario. You roll the dice on every answer.
Huh? If I need to sort the list of integer number of 3,1,2 in ascending order the only correct answer is 1,2,3. And there are multiple programming and mathematical questions with only one correct answer.
If you want to say "some programming and mathematical questions have several correct answers" that might hold.
1, 2, 3
1,2,3
[1,2,3]
1 2 3
etc.
"1 2 3" is another
"After sorting, we get `1, 2, 3`" yet another
etc.
At least, that's how I understood GP's comment.
Neither do we.
> They will give a solution that seems correct under their heuristic reasoning, but they arrived at that result in a non-logical way.
As do we, and so you can correctly reframe the issue as "there's a gap between the quality of AI heuristics and the quality of human heuristics". That the gap is still shrinking though.
And no, just because you can imagine a human stupid enough to make the same mistake, doesn't mean that LLMs are somehow human in their flaws.
> the gap is still shrinking though
I can tell this human is fond of extrapolation. If the gap is getting smaller, surely soon it will be zero, right?
Related opposing data point to this statement: https://news.ycombinator.com/item?id=45529587
Honestly this feels like a true statement to me. It's obviously a new technology, but so much of the "non-deterministic === unusable" HN sentiment seems to ignore the last two years where LLMs have become 10x as reliable as the initial models.
Of course LLMs aren't people, but an AGI might behave like a person.
LLMs don't learn from a project. At best, you learn how to better use the LLM.
They do have other benefits, of course, i.e. once you have trained one generation of Claude, you have as many instances as you need, something that isn't true with human beings. Whether that makes up for the lack of quality is an open question, which presumably depends on the projects.
I guessed the URL based on the Quartz docs. It seems to work but only has a few items from https://boydkane.com/essays/
Presumably it's a phrase you might hear from a boss who sees AI as similar to (and as benign/known/deterministic as) most other software, per TFA
>Presumably it's a phrase you might hear from a boss who sees AI as similar to (and as benign/known/deterministic as) most other software, per TFA
Yeah I get that, but I think that given the content of the article, "can't you just fix the code?" or the like would have been a better fit.
I didn't read it that way. I read "your boss" as basically meaning any non-technical person who may not understand the challenges of harnessing LLMs compared to traditional, (more) deterministic software development.
[1] https://www.economist.com/leaders/2025/09/25/how-to-stop-ais...
"The worst effects of this flaw are reserved for those who create what is known as the “lethal trifecta”. If a company, eager to offer a powerful AI assistant to its employees, gives an LLM access to un-trusted data, the ability to read valuable secrets and the ability to communicate with the outside world at the same time, then trouble is sure to follow. And avoiding this is not just a matter for AI engineers. Ordinary users, too, need to learn how to use AI safely, because installing the wrong combination of apps can generate the trifecta accidentally."
This sounds a little dramatic. The capabilities of ChatGPT are known. It generates text and images. The qualities of the content of the generated text and images is not fully known.
Likewise what to ask it for how to make some sort of horrific toxic chemical, nuclear bomb, or similar isn't much good if you cannot recognize it and dangerous capability depends heavily on what you have available to you. Any idiot can be dangerous with C4 and detonator or bleach and ammonia. Even if ChatGPT could give entirely accurate instructions on how to build an atomic bomb it wouldn't do much good because you wouldn't be able to source the tools and materials without setting off red flags.
Human thought is analog. It is based on chemical reactions, time, and unpredictably (effectively) random physical characteristics. AI is an attempt to turn that which is purely digital into an rational analog thought equivalent.
No matter how much effort, money, power, and rare mineral eating TPUs will - ever - produce true analog data.
Or they have, but chose to exploit or stockpile it rather than expose it.
Me: Ask me later.
Then it says the shop sign looks like a “Latin alphabet business name rather than Spanish or Portuguese”. Uhhh… what? Spanish and Portuguese use the Latin alphabet.
I think Apple execs genuinely underestimated how difficult it would be to get LLMs to perform up to Apple's typical standards of polish and control.
kazinator•2h ago
:)
Was that a humam Freudian slip, or artificial one?
Yes, old software is often more reliable than new.
joomla199•2h ago
giancarlostoro•2h ago
1313ed01•2h ago
eptcyka•2h ago
DSMan195276•2h ago
prasadjoglekar•2h ago
shakna•2h ago
LeifCarrotson•1h ago
Old software is typically more reliable, not because the developers were better or the software engineering targeted a higher reliability metric, but because it's been tested in the real world for years. Even more so if you consider a known bug to be "reliable" behavior: "Sure, it crashes when you enter an apostrophe in the name field, but everyone knows that, there's a sticky note taped to the receptionist's monitor so the new girl doesn't forget."
Maybe the new software has a more comprehensive automated testing framework - maybe it simply has tests, where the old software had none - but regardless of how accurate you make your mock objects, decades of end-to-end testing in the real world is hard to replace.
As an industrial controls engineer, when I walk up to a machine that's 30 years old but isn't working anymore, I'm looking for failed mechanical components. Some switch is worn out, a cable got crushed, a bearing is failing...it's not the code's fault. It's not even the CMOS battery failing and dropping memory this time, because we've had that problem 4 times already, we recognize it and have a procedure to prevent it happening again. The code didn't change spontaneously, it's solved the business problem for decades... Conversely, when I walk up to a newly commissioned machine that's only been on the floor for a month, the problem is probably something that hasn't ever been tried before and was missed in the test procedure.
freetime2•48m ago
And more often than not the issue is a local configuration issue, bad test data, a misunderstanding of what the code is supposed to do, not being aware of some alternate execution path or other pre/post processing that is running, some known issue that we've decided not to fix for some reason, etc. (And of course sometimes we do actually discover a completely new bug, but it's rare).
To be clear, there are certainly code quality issues present that make modifications to the code costly and risky. But the code itself is quite reliable, as most bugs have been found and fixed over the years. And a lot of the messy bits in the code are actually important usability enhancements that get bolted on after the fact in response to real-world user feedback.
hatthew•1h ago
E.g. you might fix a bug by adding a hacky workaround in the code; better product, worse code.
kube-system•1h ago
The author is talking about the maturity of a project. Likewise, as AI technologies become more mature we will have more tools to use them in a safer and more reliable way.
izzydata•2h ago
wsc981•2h ago
wvenable•1h ago
New code is the source of new bugs. Whether that's an entirely new product, a new feature on an existing project, or refactoring.
kazinator•1h ago
Yes, I did that.
james_marks•1h ago
glitchc•1h ago
kstrauser•1h ago
If you think modern software is unreliable, let me introduce you to our friend, Rational Rose.
kazinator•1h ago
noir_lord•1h ago
Or debuggers that would take out the entire OS.
Or a bad driver crashing everything multiple times a week.
Or a misbehaving process not handing control back to the OS.
I grew up in the era of 8 and 16 bit micros and early PCs, they where hilariously less stable than modern machines while doing far less, there wasn’t some halcyon age of near perfect software, it’s always been a case of things been good enough to be good enough but at least operating systems did improve.
malfist•1h ago
kazinator•1h ago
Windows is only stabilizing because it's basically dead. All the activity is in the higher layers, where they are racking their brains on how to enshittify the experience, and extract value out of the remaining users.
wlesieutre•44m ago
krior•42m ago
ClimaxGravely•18m ago
Yoric•43m ago
There were plenty of other issues, including the fact that you had to adjust the right IRQ and DMA for your Sound Blaster manually, both physically and in each game, or that you needed to "optimize" memory usage, enable XMS or EMS or whatever it was at the time, or that you spent hours looking at the nice defrag/diskopt playing with your files, etc.
More generally, as you hint to, desktop operating systems were crap, but the software on top of it was much more comprehensively debugged. This was presumably a combination of two factors: you couldn't ship patches, so you had a strong incentive to debug it if you wanted to sell it, and software had way fewer features.
Come to think about it, early browsers kept crashing and taking down the entire OS, so maybe I'm looking at it with rosy glasses.
binarymax•55m ago
cjbgkagh•49m ago