> require them to be majority hand written.
We should specify the outcome not the process. Expecting the contributor to understand the patch is a good idea.
> Juniors may be encouraged/required to elide LLM-assisted tooling for a period of time during their onboarding.
This is a terrible idea. Onboarding is a lot of random environment setup hitches that LLMs are often really good at. It's also getting up to speed on code and docs and I've got some great text search/summarizing tools to share.
I’ve never heard of this cliff before. Has anyone else experienced this?
I can make the problem input bigger as I want.
Each LLM have a different thresholf for each problem, when crossed the performance of the LLM collapse.
One feature others have noted is that the Opus 4 context buffer rarely "wears out" in a work session. It can, and one needs to recognize this and start over. With other agents, it was my routine experience that I'd be lucky to get an hour before having to restart my agent. A reliable way to induce this "cliff" is to let AI take on a much too hard problem in one step, then flail helplessly trying to fix their mess. Vibe-coding an unsuitable problem. One can even kill Opus 4 this way, but that's no way to run a race horse.
Some "persistence of memory" harness is as important as one's testing harness, for effective AI coding. With the right care having AI edit its own context prompts for orienting new sessions, this all matters less. AI is spectacularly bad at breaking problems into small steps without our guidance, and small steps done right can be different sessions. I'll regularly start new sessions when I have a hunch that this will get me better focus for the next step. So the cliff isn't so important. But Opus 4 is smarter in other ways.
People love to justify big expenses as necessary.
Snipping out the flailing in this way seems to help.
And one of the things with current generators is that they tend to make things more complex over time, rather than less. It's always me prompting the LLM to refactor things to make it simpler, or doing the refactoring once it's gotten to complex for the LLM to deal with.
So at least with the current generation of LLMs, it seems rather inevitable that if you just "give LLMs their head" and let them do what they want, eventually they'll create a giant Rube Goldberg mess that you'll have to try to clean up.
ETA: And to the point of the article -- if you're an old salt, you'll be able to recognize when the LLM is taking you out to sea early, and be able to navigate your way back into shallower waters even if you go out a bit too far. If you're a new hand, you'll be out of your depth and lost at sea before you know it's happened.
Imagine that you have your input to the context, 10000 tokens that are 99% correct. Each time the LLM replies it adds 1000 tokens that are 90% correct.
After some back-and-forth of you correcting the LLM, its context window is mostly its own backwash^Woutput. Worse, the error compounds because the 90% that is correct is just correct extrapolation of an argument about incorrect code, and because the LLM ranks more recent tokens as more important.
The same problem also shows up in prose.
This is also can be made much worse by thinking models, as their CoT is all in context, and if there thoughts really wander it just plants seeds of poison feeding the rot. I really wish they can implement some form of context pruning, so you can nip irrelevant context when it forms.
In the meantime, I make summaries and carry it to a fresh instance when I notice the rot forming.
The solve was to define several Cursor rules files for different views of the codebase - here's the structure, here's the validation logic, etc. That and using o3 has at least gotten me to the next level.
I suspect it has something more to do with the model producing too many tokens and becoming fixated on what it said before. You'll often see this in long conversations. The only way to fix it is to start a new conversation.
That said, I do think it would be nice for people to note in pull requests which files have AI gen code in the diff. It's still a good idea to look at LLM gen code vs human code with a bit different lens, the mistakes each make are often a bit different in flavor, and it would save time for me in a review to know which is which. Has anyone seen this at a larger org and is it of value to you as a reviewer? Maybe some tool sets can already do this automatically (I suppose all these companies report the % of code that is LLM generated must have one if they actually have these granular metrics?)
> The article opens with a statement saying the author isn't going to reword what others are writing, but the article reads as that and only that.
Hmm, I was just saying I hadn't seen much literature or discussion on trust dynamics in teams with LLMs. Maybe I'm just in the wrong spaces for such discussions but I haven't really come across it.
While on the other hand real nation-state threat actors would face no such limitations.
On a more general level, what concerns me isn't whether people use it to get utility out of it (that would be silly), but the power-imbalance in the hand of a few, and with new people pouring their questions into it, this divide getting wider. But it's not just the people using AI directly but also every post online that eventually gets used for training. So to be against it would mean to stop producing digital content.
This often is brought up that if you don't use LLMs now to produce so-so code you will somehow magically completely fall off when the LLMs all of a sudden start making perfect code as if developers haven't been learning new tools constantly as the field as evolved. Yes, I use old technology, but also yes I try new technology and pick and choose what works for me and what does not. Just because LLMs don't have a good place in my work flow does not mean I am not using them at all or that I haven't tried to use them.
The blog itself is using Alpine JS, which is a human-written framework 6 years ago (https://github.com/alpinejs/alpine), and you can see the result is not good.
Two completely unnecessary request to: jsdelivr.net and net.cdn.cloudflare.net
Never actually expected it to be posted on HN. Working on getting a static version up now.
It might not solve every problem, but it solves enough of them better enough it belongs in the tool kit.
That comparison kind of makes my point though. Sure you can bury your face into Tik Tok for 12hrs a day and they do kind of suck at Excel but smartphones are massively useful and used tools by (approximately) everyone.
Someone not using a smartphone in this day and age is very fairly a 'luddite'.
A computer is a bicycle for the mind; an LLM is an easy-chair.
AI may reach that point - that it's enough better than us thinking that we don't think much anymore, and get worse at thinking as a result. Well, is that a net win, or not? If we get there for that reason, it's probably a net win[1]. If we get there because the AI companies are really good at PR, that's a definite net loss.
All that is for the future, though. I think that currently, it's a net loss. Keep your ability to think; don't trust AI any farther than you yourself understand.
[1] It could still not be a net win, if AI turns out to be very useful but also either damaging or malicious, and lack of thinking for ourselves causes us to miss that.
3 have obviously only read the title, and 3 comments how the article require JS.
Well played HN.
Otherwise please use the original title, unless it is misleading or linkbait.
This title counts as linkbait so I've changed it. It turns out the article is much better (for HN) than the title suggests.
Good change btw.
Most of the current discourse on AI coding assistants sounds either breathlessly optimistic or catastrophically alarmist. What’s missing is a more surgical observation: the disruptive effect of LLMs is not evenly distributed. In fact, the clash between how open source and industry teams establish trust reveals a fault line that’s been papered over with hype and metrics.
FOSS project work on a trust basis - but industry standard is automated testing, pair programming, and development speed. That CRUD app for finding out if a rental car is available? Not exactly in need for a hand-crafted piece of code, and no-one cares if Junior Dev #18493 is trusted within the software dev organization.
If the LLM-generated code breaks, blame gets passed, retros are held, Jira tickets multiply — the world keeps spinning, and a team fixes it. If a junior doesn’t understand their own patch, the senior rewrites it under deadline. It’s not pretty, but it works. And when it doesn’t, nobody loses “reputation” - they lose time, money, maybe sleep. But not identity.
LLMs challenge open source where it’s most vulnerable - in its culture. Meanwhile, industry just treats them like the next Jenkins: mildly annoying at first, but soon part of the stack.
The author loves the old ways, for many valid reasons: Gabled houses are beautiful, but outside of architectural circles, prefab is what scaled the suburbs, not timber joints and romanticism.
I understand the frustration: meaning reduced to metadata, debate replaced with reaction, and the richness of human thought lost in the echo of paraphrased content. If there is an exit to this timeline, I too would like to request the coordinates.
[ai]: rewrote the documentation ...
This is helps us to put another set of "glasses" as we later review the code.
If you use AI as tab-complete but it's what you would've done anyway, should you flag it? I don't know, plenty to think about when it comes to what the right amount of disclosure is.
I certainly wish that with our company, people could flag (particularly) large commits as coming from a tool rather than a person, but I guess the idea is that the person is still responsible for whatever the tool generates.
The problem is that it's incredibly enticing for over-worked engineers to have AI do large (ie. diffs) but boring tasks that they'd typically get very little recognition for (eg. ESLint migrations).
The HN submission has been editorialised since it was submitted, originally said "Yes, I will judge you for using AI..." and a lot of the replies early on were dismissive based on the title alone.
If someone uses an LLM and produces bug-free code, I'll trust them. If someone uses an LLM and produces buggy code, I won't trust them. How is this different from when they were only using their brain to produce the code?
The problem is often really one of miscommunication, the task may be clear to the person working on it, but with frequent context resets it's hard to make sure the LLM also knows what the whole picture is and they tend to make dumb assumptions when there's ambiguity.
The thing that 4o does with deep research where it asks for additional info before it does anything should be standard for any code generation too tbh, it would prevent a mountain of issues.
All of these make mistakes (there are documented incidents).
And yes, we can counter with "the journalists are dumb for not verifying", "the lawyers are dumb for not checking", etc., but we should also be open for the fact that these are intelligent and professional people who make mistakes because they were mislead by those who sell LLMs.
As if that is a somehow exonerating sentence.
LLMs are a tool, just like any number of tools that are used by developers in modern software development. If a dev doesn’t use the tool properly, don’t trust them. If they do, trust them. The way to assess if they use it properly is in the code they produce.
Your premise is just fundamentally flawed. Before LLMs, the proof of a quality dev was in the pudding. After LLMs, the proof of a quality dev remains in the pudding.
Indeed it does, however what the "proof" is has changed. In terms of sitting down and doing a full, deep review, tracing every path validating every line etc. Then for sure, nothing has changed.
However, at least in my experience, pre LLM those reviews were not EVERY CASE there were many times I elided parts of a deep review because i saw markers in the code that to me showed competency, care etc. With those markers there are certain failure conditions that can be deemed very unlikely to exist and therefore the checks can be skipped. Is that ALWAYS the correct assumption? Absolutely not but the more experienced you are the less false positives you get.
LLMs make those markers MUCH harder to spot, so you have to fall back to doing a FULL indepth review no matter what. You have to eat ALL the pudding so to speak.
For people that relied on maybe tasting a bit of the pudding then assuming based on the taste the rest of the pudding probably tastes the same its rather jarring and exhausting to now have to eat all of it all the time.
That was never proof in the first place.
If anything, someone basing their trust in a submission on anything other than the code itself is far more concerning and trust-damaging to me than if the submitter has used an LLM.
I mean, it's not necessarily HARD proof but it has been a reliable enough way to figure out which corners to cut. You can of course say that no corners should ever be cut and while that is true in an ideal sense. In the real world things always get fuzzy.
Maybe the death of cutting corners is a good thing overall for output quality. Its certainly exhausting on the people tasked with doing the reviews however.
Ultimately I don't think the heuristics would change all that much, though. If every time you review a person's PR, almost everything is great, they are either not using AI or they are vetting what the AI writes themselves, so you can trust them as you did before. It may just take some more PRs until that's apparent. Those who submit unvetted slop will have to fix a lot of things, and you can crank up the heat on them until they do better, if they can. (The "if they can" is what I'm most worried about.)
It’s like if someone started bricking up tunnel entrances and painting ultra realistic versions of the classic Road Runner tunnel painting on them, all over the place. You’d have to stop and poke every underpass with a stick just to be sure.
Precisely, in the age where it is very difficult to ascertain the type or quality of skills you are interacting with say in a patch review or otherwise you frankly have to "judge" someone and fallback to suspicion and full verification.
There's so much more than "works well". There are many cues that exist close to code, but are not code:
I trust more if the contributor explains their change well.
I trust more if the contributor did great things in the past.
I trust more if the contributor manages granularity well (reasonable commits, not huge changes).
I trust more if the contributor picks the right problems to work on (fixing bugs before adding new features, etc).
I trust more if the contributor proves being able to maintain existing code, not just add on top of it.
I trust more if the contributor makes regular contributions.
And so on...
Spot on, there are so many little things that we as humans use as subtle verification steps to decide how much scrutiny various things require. LLMs are not necessarily the death of that concept but they do make it far far harder.
A good rule of thumb is to simply reject any work that has had involvement of an LLM, and ignore any communication written by an LLM (even for EFL speakers, I'd much rather have your "bad" English than whatever ChatGPT says for you).
I suspect that as the serious problems with LLMs become ever more apparent, this will become standard policy across the board. Certainly I hope so.
This is standard for any activity where accuracy / safety is paramount - you validate the process. Hence things like maintenance logs for airplanes.
Precisely this, and this is hardly a unique to software requirement. Process audits are everywhere in engineering. Previously you could infer the process of producing some code by simply reading the patch and that generally would tell you quite a bit about the author itself. Using advanced and niche concepts with imply a solid process with experience backing it. Which would then imply that certain contextual bugs are unlikely so you skip looking for them.
My premise in the blog is basically that "Well now I have go do a full review no matter what the code itself tells me about the author."
What? How on god's green earth could you even pretend to know how all people are using these tools?
> LLMs are not calculators, nor are they the internet.
Umm, okay? How does that make them less useful?
I'm going to give you a concrete example of something I just did and let you try and do whatever mental gymnastics you have to do to tell me it wasn't useful:
Medicare requires all new patients receiving home health treatment go through a 100+ question long form. This form changes yearly, and it's my job to implement the form into our existing EMR. Well, part of that is creating a printable version. Guess what I did? I uploaded the entire pdf to Claude and asked it to create a print-friendly template using Cottle as the templating language in C#. It generated the 30 page print preview in a minute. And it took me about 10 more minutes to clean up.
> I suspect that as the serious problems with LLMs become ever more apparent, this will become standard policy across the board. Certainly I hope so.
The irony is that they're getting better by the day. That's not to say people don't use them for the wrong applications, but the idea that this tech is going to be banned is absurd.
> A good rule of thumb is to simply reject any work that has had involvement of an LLM
Do you have any idea how ridiculous this sounds to people who actually use the tools? Are you going to be able to hunt down the single React component in which I asked it to convert the MUI styles to tailwind? How could you possibly know? You can't.
How are you going to know?
This is okay for platitudes, but for emails that really matter, having this messy watercolor kind of writing totally destroys the clarity of the text and confuses everyone.
To your point, I’ve asked everyone on my team to refrain from writing words (not code) with ChatGPT or other tools, because the LLM invariably leads to more complicated output than the author just badly, but authentically, trying to express themselves in the text.
The LLMs struggle with both but REALLY struggle with figuring out what NOT to say.
Surely it's less work to put the words you want to say into an email, rather than craft a prompt to get the LLM to say what you want to say, and iterate until the LLM actually says it?
There's nothing wrong with using LLMs to save time doing trivial stuff you know how to do yourself and can check very easily. The problem is that (very lazy) people are using them to do stuff they are themselves not competent at. They can't check, they won't learn, and the LLM is essentially their skill ceiling. This is very bad: what plus-value are you supposed to bring over something you don't understand? AGI won't have to improve from the current baseline to surpass humans if we're just going to drag ourselves down to its level.
What you're seeing now is people who once thought and proclaimed these tools as useless now have to start to walk back their claims with stuff like this.
It does amaze me that the people who don't use these tools seem to have the most to say about them.
For what it's worth I do actually use the tools albeit incredibly intentionally and sparingly.
I see quite a few workflows and tasks that they can be a value add on, mostly outside of the hotpath of actual code generation but still quite enticing. So much so in fact I'm working on my own local agentic tool with some self hosted ollama models. I like to think that i am at least somewhat in the know on the capabilities and failure points of the latest LLM tooling.
That however doesn't change my thoughts on trying to ascertain if code submitted to me deserves a full indepth review or if I can maybe cut a few corners here and there.
How would you even know? Seriously, if I use Chatgpt to generate a one-off function for a feature I'm working on that searches all classes for one that inherits a specific interface and attribute, are you saying you'd be able to spot the difference?
And what does it even matter it works?
What if I use Bolt to generate a quick screen for a PoC? Or use Claude to create a print-preview with CSS of a 30 page Medicare form? Or converting a component's styles MUI to tailwind? What if all these things are correct?
This whole OS repos will ban LLM-generated code is a bit absurd.
> or what it's worth I do actually use the tools albeit incredibly intentionally and sparingly.
How sparingly? Enough to see how it's constantly improving?
I don't know, thats the problem. As a result, because I can't know I have to now do full in depth reviews no matter what. Which is the "judging" I tongue in cheek talk about in the blog.
> How sparingly? Enough to see how it's constantly improving?
Nearly daily, to be honest I have not noticed too much improvement year over year in regards to how they fail. They still break in the exact same dumb ways now as they did before. Sure they might generate correct syntactic code reliably now and it might even work. But they still consistently fail to grok the underlying reasoning for things existing.
But I am writing my own versions of these agentic systems to use for some rote menial stuff.
You're kidding, right? Most people who don't use the tools and write about it are responding to the ongoing hype train -- a specific article, a specific claim, or an idea that seems to be gaining acceptance or to have gone unquestioned among LLM boosters.
I recently watched a talk by Andrei Karpathy. So much in it begged for a response. Google Glass was "all the rage" in 2013? Please. "Reading text is laborious and not fun. Looking at images is fun." You can't be serious.
Someone recently shared on HN a blog post explaining why the author doesn't use LLMs. The justification for the post? "People keep asking me."
Essentially the premise is that in medium trust environments like very large teams or low trust environments like an open source project.
LLMs make it very difficult to make an immediate snap judgement about the quality of the dev that submitted the patch based solely on the code itself.
In the absence of being able to ascertain the type of person you are dealing with you have to fall back too "no trust" and review everything with a very fine tooth comb. Essentially there are no longer any safe "review shortcuts" and that can be painful in places that relied on those markers to grease the wheels so to speak.
Obviously if you are in an existing competent high trust team then this problem does not apply and most likely seems completely foreign as a concept.
That's the core of the issue. It's time to say goodbye to heuristics like "the blog post is written in eloquent, grammatical English, hence the point its author is trying to make must be true" or "the code is idiomatic and following all code styles, hence it must be modeling the world with high fidelity".
Maybe that's not the worst thing in the world. I feel like it often made people complacent.
For sure, in some ways perhaps reverting to a low trust environment might improve quality in that it now forces harsher/more in depth reviews.
That however doesn't make the requirement less exhausting for people previously relying heavily on those markers to speed things up.
Will be very interesting to see how the industry standardizes around this. Right now it's a bit of the wild west. Maybe people in ten years will look back at this post and think "what do you mean you judged people based on the code itself that's ridiculous"
You said "hence the point its author is trying to make must be true" and "hence it must be modeling the world with high fidelity".
But it's more like "hence the author is likely competent and likely put in a reasonable effort."
When those assumptions hold, putting in a very deep review is less likely to pay off. Maybe you are right that people have been too complacent to begin with, I don't know, but I don't think you've framed it fairly.
And isn't dyslexic, and is a native speaker etc. Some will gain from this shift, some will lose.
When someone follows standard conventions it means that they A) have a baseline level of knowledge to know about them, and B) care to write the code in a clear and approachable way for others.
“unconventional” or “fancy” is in the eye of the beholder. Whose conventions are we talking about? Code is bad when it doesn't look the way you want it to? How convenient. I may find code hard to read because it's formatted “conventionally”, but I wouldn't be so entitled as to call it bad just because of that.
Literally not: a language defines its own conventions, they're not defined in terms of individual users/readers/maintainers subjective opinions.
> Whose conventions are we talking about?
The conventions defined by the language.
> Code is bad when it doesn't look the way you want it to?
No -- when it doesn't satisfy the conventions established by the language.
> I may find code hard to read because it's formatted “conventionally”,
If you did this then you'd be wrong, and that'd be a problem with your personal evaluation process/criteria, that you would need to fix.
The heuristic is "this submission doesn't even follow the basic laws of grammar, therefore I can safely assume incompetence and ignore it entirely."
You still have to do verification for what passes the heuristic, but it keeps 90% of the crap away.
a dev can write piece of good, and piece of bad code. so per code, review the code. not the dev!
In my experience they very much are related. High quality devs are far more likely to output high quality working code. They test, they validate, they think, ultimately they care.
In that case that you are reviewing a patch from someone you have limited experience with, it previously was feasible to infer the quality of the dev from the context of the patch itself and the surrounding context by which it was submitted.
LLMs make that judgement far far more difficult and when you can not make a snap judgement you have to revert your review style to very low trust in depth review.
No more greasing the wheels to expedite a process.
I could not disagree more. The quality of the dev will always matter, and has as much to do with what code makes it into a project as the LLM that generated it.
An experienced dev will have more finely tuned evaluation skills and will accept code from an LLM accordingly.
An inexperienced or “low quality” dev may not even know what the ideal/correct solution looks like, and may be submitting code that they do not fully understand. This is especially tricky because they may still end up submitting high quality code, but not because they were capable of evaluating it as such.
You could make the argument that it shouldn’t matter who submits the code if the code is evaluated purely on its quality/correctness, but I’ve never worked in a team that doesn’t account for who the person is behind the code. If its the grizzled veteran known for rarely making mistakes, the review might look a bit different from a review for the intern’s code.
That may be true, but the proxy for assessing the quality of the dev is the code. No one is standing over you as you code your contribution to ensure you are making the correct, pragmatic decisions. They are assessing the code you produce to determine the quality of your decisions, and over time, your reputation as a dev is made up of the assessments of the code you produced.
The point is that an LLM in no way changes this. If a dev uses an LLM in a non-pragmatic way that produces bad code, it will erode trust in them. The LLM is a tool, but trust still factors in to how the dev uses the tool.
Yes, the quality of the dev is a measure of the quality of the code they produce, but once a certain baseline has been established, the quality of the dev is now known independent of the code they may yet produce. i.e. if you were to make a prediction about the quality of code produced by a "high quality" dev vs. a "low quality" dev, you'd likely find that the high quality dev tends to produce high quality code more often.
So now you have a certain degree of knowledge even before you've seen the code. In practice, this becomes a factor on every dev team I've worked around.
Adding an LLM to the mix changes that assessment fundamentally.
> The point is that an LLM in no way changes this.
I think the LLM by definition changes this in numerous ways that can't be avoided. i.e. the code that was previously a proxy for "dev quality" could now fall into multiple categories:
1. Good code written by the dev (a good indicator of dev quality if they're consistently good over time)
2. Good code written by the LLM and accepted by the dev because they are experienced and recognize the code to be good
3. Good code written by the LLM and accepted by the dev because it works, but not necessarily because the dev knew it was good (no longer a good indicator of dev quality)
4. Bad code written by the LLM
5. Bad code written by the dev
#2 and #3 is where things get messy. Good code may now come into existence without it being an indicator of dev quality. It is now necessary to assess whether or not the LLM code was accepted because the dev recognized it was good code, or because the dev got things to work and essentially got lucky.
It may be true that you're still evaluating the code at the end of the day, but what you learn from that evaluation has changed. You can no longer evaluate the quality of a dev by the quality of the code they commit unless you have other ways to independently assess them beyond the code itself.
If you continued to assess dev quality without taking this into consideration, it seems likely that those assessments would become less accurate over time as more "low quality" devs produce high quality code - not because of their own skills, but because of the ongoing improvements to LLMs. That high quality code is no longer a trustworthy indicator of dev quality.
> If a dev uses an LLM in a non-pragmatic way that produces bad code, it will erode trust in them. The LLM is a tool, but trust still factors in to how the dev uses the tool.
Yes, of course. But the issue is not that a good dev might erode trust by using the LLM poorly. The issue is that inexperienced devs will make it increasingly difficult to use the same heuristics to assess dev quality across the board.
Only because you already trust them to know that the code is indeed bug-free. Some cases are simple and straightforward -- this routine returns a desired value or it doesn't. Other situations are much more complex in anticipating the ways in which it might interact with other parts of the system, edge cases that are not obvious, etc. Writing code that is "bug free" in that situation requires the writer of the code to understand the implications of the code, and if the dev doesn't understand exactly what the code does because it was written by an LLM, then they won't be able to understand the implications of the code. It then falls to the reviewer to understand the implications of the code -- increasing their workload. That was the premise.
That being said, the prediction engine still can't do any real engineering. If you don't specifically task them with using things like Python generators, you're very likely to have a piece of code that eats up a gazillion memory. Which unfortunately don't set them appart from a lot of Python programmers I know, but it is an example of how the LLM's are exactly as bad as you mention. On the positive side, it helps with people actually writing the specification tasks in more detail than just "add feature".
Where AI-agents are the most useful for us is with legacy code that nobody prioritise. We have a data extractor which was written in the previous millennium. It basically uses around two hunded hard-coded coordinates to extact data from a specific type of documents which arrive by fax. It's worked for 30ish years because the documents haven't changed... but it recently did, and it took co-pilot like 30 seconds to correct the coordinates. Something that would've likely taken a human a full day of excruciating boredom.
I have no idea how our industry expect anyone to become experts in the age of vibe coding though.
So they’re even more confident in their wrongness
Every time I tell claude code something it did is wrong, or might be wrong, or even just ask a leading question about a potential bug it just wrote, it leads with "You're absolutely correct!" before even invoking any tools.
Maybe you've just become used to ignoring this. I mostly ignore it but it is a bit annoying when I'm trying to use the agent to help me figure out if the code it wrote is correct, so I ask it some question it should be capable of helping with and it leads with "you're absolutely correct".
I didn't make a proposition that can be correct or not, and it didn't do any work yet to to investigate my question - it feels like it has poisoned its own context by leading with this.
I’d love to hear more about your workflow and the code base you’re working in. I have access to Amazon Q (which it looks like is using Claude Sonnet 4 behind the scenes) through work, and while I found it very useful for Greenfield projects, I’ve really struggled using it to work on our older code bases. These are all single file 20,000 to 100,000 line C modules with lots of global variables and most of the logic plus 25 years of changes dumped into a few long functions. It’s hard to navigate for a human, but seems to completely overwhelm Q’s context window.
Do other Agents handle this sort of scenario better, or are there tricks to making things more manageable? Obviously re-factoring to break everything up into smaller files and smaller functions would be great, but that’s just the sort of project that I want to be able to use the AI for.
Not they don't. This is 100% a made up statistic.
I found out very early that under no circumstances you may have the code you don't understand, anywhere. Well, you may, but not in public, and you should commit to understanding it before anyone else sees that. Particularly before sales guys do.
However, AI can help you with learning too. You can run experiments, test hypotheses and burn your fingers so fast. I like it.
Sorry about the JS stuff I wrote this while also fooling around with alpine.js for fun. I never expected it to make it to HN. I'll get a static version up and running.
Happy to answer any questions or hear other thoughts.
Edit: https://static.jaysthoughts.com/
Static version here with slightly wonky formatting, sorry for the hassle.
Edit2: Should work on mobile now well, added a quick breakpoint.
I think what the author misses here is that imperfect, probabilistic agents can build reliable, deterministic systems. No one would trust a garbage collection tool based on how reliable the author was, but rather if it proves it can do what it intends to do after extensive testing.
I can certainly see an erosion of trust in the future, with the result being that test-driven development gains even more momentum. Don't trust, and verify.
> but rather if it proves it can do what it intends to do after extensive testing.
Author here: Here I was less talking about the effectiveness of the output of a given tool and more so about the tool itself.
To take your garbage collection example, sure perhaps an agentic system at some point can spin some stuff up and beat it into submission with test harnesses, bug fixes etc.
But, imagine you used the model AS the garbage collector/tool, in that say every sweep you simply dumped the memory of the program into the model and told it to release the unneeded blocks. You would NEVER be able to trust that the model itself correctly identifies the correct memory blocks and no amount of "patching" or "fine tuning" would ever get you there.
With other historical abstractions like say jvm, if the deterministic output, in this case the assembly the jit emits is incorrect that bug is patched and the abstraction will never have that same fault again. not so with LLMs.
To me that distinction is very important when trying to point out previous developer tooling that changed the entire nature of the industry. It's not to say I do not think LLMs will have a profound impact on the way things work in the future. But I do think we are in completely uncharted territory with limited historical precedence to guide us.
An even more important question: who tests the tests themselves? In traditional development, every piece of logic is implemented twice: once in the code and once in the tests. The tests checks the code, and in turn, the code implicitly checks the tests. It's quite common to find that a bug was actually in the tests, not the app code. You can't just blindly trust the tests, and wait until your agent finds a way to replicate a test bug in the code.
Verification has a high cost and trust is the main way to lower that cost. I don't see how one can build trust in LLMs. While they are extremely articulate in both code and natural language, they will also happily go down fractal rabbit holes and show behavior I would consider malicious in a person.
This new world of having to verify every single thing at all points is quite exhausting and frankly pretty slow.
I don't agree with this at all. Writing new code is trivially easy, to do a full in depth review takes significantly more brain power. You have to fully ascertain and insert yourself into someone elses thought process. Thats way more work than utilizing your own thought process.
They basically achieve over 80% agreement with human evaluators [1]. This level of agreement is similar to the consensus rate between two human evaluators, making LLM-as-a-judge a scalable and reliable proxy for human judgment.
[1] https://arxiv.org/abs/2306.05685 (2023)
Oh goodness that's like trusting one kid to tell you whether or not his friend lied.
In matters where trust matters, it's a recipe for disaster.
Give it another year and HN comments will be very different.
Writing tests already works now. It's usually easier to read tests than to read convoluted logic.
Mmmhmm. And you think this "growing up" doesn't have biases to lie in circumstances where it matters? Consider politics. Politics matter. It's inconceivable that a magic algorithm would lie to us about various political concerns, right? Right...?
A magic algorithm lying to us about anything would be extremely valuable to liars. Do you think it's possible that liars are guiding the direction of these magic algorithms?
The classic on the subject.
You'll have to elaborate on that. How much trust was there in electricity, flight and radioactivity when we discovered them?
In science, you build trust as you go.
> As the use of AC spread rapidly with other companies deploying their own systems, the Edison Electric Light Company claimed in early 1888 that high voltages used in an alternating current system were hazardous, and that the design was inferior to, and infringed on the patents behind, their direct current system.
> In the spring of 1888, a media furor arose over electrical fatalities caused by pole-mounted high-voltage AC lines, attributed to the greed and callousness of the arc lighting companies that operated them.
The moment a technology reaches trust at scale, it becomes a non-innovation in people's mind.
Happened for TVs, electrical light in homes, AI for chess, and Google. Will happen with LLM-based assistants.
True not only in innovation, but in business settings.
I don't think there's anyone who works in any business long enough who doesn't have problems getting their job done simply because someone else with a key part of the project doesn't trust that you know what you're doing.
Not much.
Plenty of people were against electricity when it started becoming common. They were terrified of lamps, doorbells, telephones, or anything else with an electric wire. If they were compelled to use these things (like for their job) they would often wear heavy gloves to protect themselves. It is very occasionally mentioned in novels from the late 1800's.
There are still people afraid of electricity today. There is no shortage of information on the (ironically enough) internet about how to shield your home from the harmful effects of electrical wires, both in the house and utility lines.
Flight? I dunno about back then, but today there's plenty of people who are afraid to fly. If you live in Las Vegas for a while, you start to notice private train cars occasionally parked on the siding near the north outlet mall. These belong to celebrities who are afraid to fly, but have to go to Vegas for work.
Radioactivity? There was a plethora of radioactive hysteria in books, magazines, comics, television, movies, and radio. It's not hard to find.
It's just too useful to ignore, and trust is always built, brick by brick. Let's not forget humans are far from reliable anyway. Just like with driving cars, I imagine producing less buggy code (along predefined roads) will soon outpace humans. Then it is just blocking and tackling to improve complexity.
I once had a member of my extended family who turned out to be a con artist. After she was caught, I cut off contact, saying I didn’t know her. She said “I am the same person you’ve known for ten years.” And I replied “I suppose so. And now I realized I have never known who that is, and that I never can know.”
We all assume the people in our lives are not actively trying to hurt us. When that trust breaks, it breaks hard.
No one who uses AI can claim “this is my work.” I don’t know that it is your work.
No one who uses AI can claim that it is good work, unless they thoroughly understand it, which they probably don’t.
A great many students of mine have claimed to have read and understand articles I have written, yet I discovered they didn’t. What if I were AI and they received my work and put their name on it as author? They’d be unable to explain, defend, or follow up on anything.
This kind of problem is not new to AI. But it has become ten times worse.
I’m someone who put in my "+10,000 hours" programming complex applications, before useful LLMs were released. I spent years diving into documentation and other people's source code every night, completely focused on full-stack mastery. Eventually, that commitment led to severe burnout. My health was bad, my marriage was suffering. I released my application and then I immediately had to walk away from it for three years just to recover. I was convinced I’d never pick it up again.
It was hearing many reports that LLMs had gotten good at code that cautiously brought me back to my computer. That’s where my experience diverges so strongly from your concerns. You say, “No one who uses AI can claim ‘this is my work.’” I have to disagree. When I use an LLM, I am the architect and the final inspector. I direct the vision, design the system, and use a diff tool to review every single line of code it produces. Just recently, I used it as a partner to build a complex optimization model for my business's quote engine. Using a true optimization model was always the "right" way to do it but would have taken me months of grueling work before, learning all details of the library, reading other people’s code, etc. We got it done in a week. Do I feel like it’s my work? Absolutely. I just had a tireless and brilliant, if sometimes flawed, assistant.
You also claim the user won't "thoroughly understand it." I’ve found the opposite. To use an LLM effectively for anything non-trivial, you need a deeper understanding of the fundamentals to guide it and to catch its frequent, subtle mistakes. Without my years of experience, I would be unable to steer it for complex multi-module development, debug its output, or know that the "plausibly good work" it produced was actually wrong in some ways (like N+1 problems).
I can sympathize with your experience as a teacher. The problem of students using these tools to fake comprehension is real and difficult. In academia, the process of learning, getting some real fraction of the +10,000hrs is the goal. But in the professional world, the result is the goal, and this is a new, powerful tool to achieve better results. I’m not sure how a teacher should instruct students in this new reality, but demonizing LLM use is probably not the best approach.
For me, it didn't make bad work look good. It made great work possible again, all while allowing me to have my life back. It brought the joy back to my software development craft without killing me or my family to do it. My life is a lot more balanced now and for that, I’m thankful.
I do not lightly say that I don't trust the work of someone who uses AI. I'm required to practice with LLMs as part of my job. I've developed things with the help of AI. Small things, because the amount of vigilance necessary to do big things is prohibitive.
Fools rush in, they say. I'm not a fool, and I'm not claiming that you are either. What I know is that there is a huge burden of proof on the shoulders of people who claim that AI is NOT problematic-- given the substantial evidence that it behaves recklessly. This burden is not satisfied by people who say "well, I'm experienced and I trust it."
It seems that LLMs, as they work today, make developers more productive. It is possible that they benefit less experienced developers even more than experienced developers.
More productivity, and perhaps very large multiples of productivity, will not be abandoned due roadblocks constructed by those who oppose the technology due to some reason.
Examples of the new productivity tool causing enormous harm (eg: bug that brings down some large service for a considerable amount of time) will not stop the technology if it being considerable productivity.
Working with the technology and mitigating it's weaknesses is the only rational path forward. And those mitigation can't be a set of rules that completely strip the new technology of it's productivity gains. The mitigations have to work with the technology to increase its adoption or they will be worked around.
Even now I have this refusal to use GPT where as my coworkers lately have been saying "ChatGPT says" or this code was created by chatGPT idk, for me I take pride writing code myself/not using GPT but I also still use google/stackoverflow which you could say is a slower version of GPT.
Think this strongly depends on the developer and what they're attempting to accomplish.
In my experience, most people who swear LLMs make them 10x more productive are relatively junior front-end developers or serial startup devs who are constantly greenfielding new apps. These are totally valid use cases, to be clear, but it means a junior front-end dev and a senior embedded C dev tend to talk past each other when they're discussing AI productivity gains.
> Working with the technology and mitigating it's weaknesses is the only rational path forward.
Or just using it more sensibly. As an example: is the idea of an AI "agent" even a good one? The recent incident with Copilot[0] made MS and AI look like a laughingstock. It's possible that trying to let AI autonomously do work just isn't very smart.
As a recent analogy, we can look at blockchain and cryptocurrency. Love it or hate it, it's clear from the success of Coinbase and others that blockchain has found some real, if niche, use cases. But during peak crypto hype, you had people saying stuff like "we're going to track the coffee bean supply chain using blockchain". In 2025 that sounds like an exaggerated joke from Twitter, but in 2020 it was IBM legitimately trying to sell this stuff[1].
It's possible we'll look back and see AI agents, or other current applications of generative AI, as the coffee blockchain of this bubble.
[0] https://www.reddit.com/r/ExperiencedDevs/comments/1krttqo/my...
[1] https://www.forbes.com/sites/robertanzalone/2020/07/15/big-c...
I agree with this quite a lot. I also think that those greenfield apps quickly become unmanageable by AI as you need to start applying solutions that are unique/tailored for your objective or you want to start abstracting some functionality into building components and base classes that the AI hasn't seen before.
I find AI very useful to get me to a from beginner to intermediate in codebases and domains that I'm not familiar with but, once I get the familiarity, the next steps I take mostly without AI because I want to do novel things it's never seen before.
But this doesn't mean that the model/human combo is more effective at serving the needs of users! It means "producing more code."
There are no LLMs shipping changesets that delete 2000 lines of code -- that's how you know "making engineers more productive" is a way of talking about how much code is being created...
gblargg•10h ago
(Works on older browsers and doesn't require JavaScript except to get past CloudSnare).