I think where things get dicey is being able to write in any language. I write and review code in many languages and frameworks I'm not fluent in, so it's hard for me to distinguish between working code and great code. I can spot when the fundamental logic is wrong, but when it comes to "best fit" choices I'm clueless.
At work, with a team and code reviews, the 50%-100% figure seems much more likely.
This can probably move towards the more spectacular productivity gains as the AI's output becomes more reliable, people realize this, and less time is spend on code review and cleaning up the output.
The thing is, if you follow SWE best practices indiscriminately, then you ll have a shit code base in no time.
There is no silver bullet, and no replacement for experience and mindfulness.
This is something I have been trying to get right as well. I've attempted to use lots of linting and things like strong typing, duplicate checks, cyclomatic complexity, and robust tests. However, I still happen to find issues, which requires me to look at the code (at least at a high level)
For example, I can say "Don't repeat yourself, and don't re-write helper functions" and I will even have a duplicate linter check, but inevitably the LLM will always want to re-write a similar yet slightly different helper function. Like it will always want to re-write something small like a trim() or a toString() function in every file.
I have better luck telling it positive things rather than lots of "never do X" style things.
But if I add a separate post-implementation review/fix pass by the agent, it'll usually find and fix the issues.
So I've started doing it for everything from naming conventions to duplicate code to other problems. It does cost more tokens, but now I get less frustrated at having to fix basic issues in the PRs.
1. they don't care
2. the rest of the team doesn't care
3. the powers that be actively discourage it because velocity.
LLMs let you move faster.
But it's not as if introducing them is the only reason your codebase isn't high quality.
I’m doubtful that the author’s recommendation always work, but I do some similar things and they do seem to help.
There also exist other good reasons why projects don't want AI-generated code, in particular
- because of unclarity of copyright status and consequences of AI-generated code
- because the project leader simply made the observation than many programmers who hand in AI-generated code care more about "getting things done" and "pushing through their changes" (possibly to boost their CV) instead of deeply caring about code quality
I'm happily vibing my own toy projects, but would prefer if the tech in hospitals is not vibe coded.
And I don't think it's plausible that the gap between those two is "well you just need to use it right".
No LLM will destroy any code base in any time frame without permission from an human operator. That person is responsible for allowing the code base being destroyed.
And technical quality gates do not help if the human side lacks defense against slop code. If you don't have the right managers in place, the 2 years of experience vibecoder who ships a feature in 4 hours will always win against the 20+ year senior who actually looks at the code he is about to ship.
If the compiler that I write produces lousy code, I get bugs that I fix until it doesn’t.
And that is the most annoying thing about this revolution. It’s obviously powerful and transformative and I use in my job all the time.
But many, perhaps even most, purveyors seem intent on blaming their users when they have issues, rather than fixing their own bugs.
General model improvement is going a long way here, but basic things like “ensure you use good style and programming practices” really shouldn’t be a thing users need to put in any .md file.
AIs are stochastic/probabilistic machines. Their big potential is in how they take malformed, incomplete, ambiguous inputs and come up with valuable and usable solutions.
Good defaults are expected in pretty much every other tool.
And “You just have to set it up carefully and properly” is pretty much saying that the defaults are never good enough.
Eh. I wouldn't focus on unit test coverage.
I think it's true that good, well tested code will have higher code coverage than crappy code.
But, above a certain point (which will vary from codebase to codebase), unit tests aren't meaningfully increasing confidence that the code is working.
I'd recommend focusing instead on the code being written in a pure 'functional core, imperative shell' to the extent that's possible. For that pure/functional part, 100% code coverage is attainable (& so not worth remarking on). For the impure parts, unit tests are probably using "mocks" just to get the code to compile anyway.
When implementing new features or making large refactoring changes; I use the superpowers:brainstorming skill. That has consistent process which has worked really well. I alway review the code before merging, but most of the time there are few issues to correct.
I don't do 95% coverage, but I have increased it from 65% to about +80% and that is sufficient.
Kindly share the metrics by which you evaluate the changes.
As a simple experiment, try giving AI a high level goal for your software and let it iterate on it by just repeatedly prompting it to continue, it will happily churn forever on the goal, turning the codebase into a useless spaghetti mess with very high probability, and growing it more and more without ever cutting anything back. That's what happens without human intervention regarding system state and manipulation. The main issues here are most prompts that are extremely underspecified ("fix the issue with the buttons on the main page") so AI will ingest context data it likely generated itself in a previous step and assumptions from its own training data, then act on that to produce a new state. Think of it like a random walk, the AI makes a small step in one random direction to achieve a goal, that brings the system to a new state which is now the basis for the next step, and so on. If there's no (or not enough) corrective action that pulls the system back to a known good reference state it will keep wandering in random directions.
That's the main issue, people have a hard time steering recursive, probabilistic systems, especially when they never look at the output of the system after each step and correct it. And let's be real, if you examine AI generated output in great detail after each iteration you're often better off writing the code yourself, so I would argue that the promised speed up of agentic development can only be realized if you stop inspecting every output of the system. And it seems we still haven't figured out how to specify the steering instructions that keep a system close to a given ideal state that allow unsupervised, recursive work on most codebases. I think some codebases are by themselves better suited for this as they provide a more rigid harness for AI development and exist in the training data (e.g. CRUD apps using RoR), whereas complex software that doesn't use rigid frameworks is at much higher risk of destruction by AI as there's no reference point in the training data that would hold the AI back from randomly walking to a garbage state.
And that's why people have such different views on agentic software development, some work on codebases that are better represented in the training data and so have great success using agentic tools on them, others work on software that isn't represented so well so AI does poorly on it. I don't think it's an issue with quality management, from my own experiments no amount of hand-written rules or system prompts will keep AI from destroying a codebase for which it doesn't have a strong idea how the code is supposed to look from its own training data in the first place. As another experiment, try giving AI strict rules about how to change code or introduce new features, it will always find a way around them or appropriate them in a maliciously funny way that you haven't anticipated. That's also an artefact of the training process, these systems aren't designed to say no or do nothing, they produce outputs to achieve goals and they will bend your rules to the greatest amount possible if it helps with goal fulfilment.
However, I just don't think that's realistic. It's asking an author to suddenly become an editor. It's asking somebody who writes code to now read and debug others code.
It can actually be harder to find the the bug in a tricky piece of code than it can be to write your own correct code from scratch. I see AI introduce all sorts of bugs all the time in my personal projects that I would never introduce, and would never think to test for, especially around anything graphical.
I'd like to think the time and practice I've put into software engineering has made me better at it. If that's not true, then there's no reason to prefer senior or principal engineers with years of experience over newcomers.
You don't think crash will happen because XYZ. You _wish_ for the crash because you are hateful of progress that you are not part of.
Though arguably more of a process and judgement issue than skill.
What makes LLM-generated code a bit special there is that misjudging how to deal with it seems to be what most people do. So the default is broken.
Whereas in prior iterations of "skill issue", the default was working.
Over the last year, LLM coding agents gotten pretty good. It's no longer "if your results suck, you gotta try the latest and greatest model". You can get capable results on a wide variety of tasks, with a wide variety of models, used in a wide variety of ways.
zwaps•47m ago