> 2. Engineers underestimate the degree to which this is true, and don’t carefully review AI-generated code to the degree to which they would review their own.
> The reason for #2 isn’t complacency, one can review code at the speed at which they can think and type, but not at the speed at which an LLM can generate. When you’re typing code, you review-as-you-go, when you AI-generate the code, you don’t.
> Interestingly, the inverse is true for mediocre engineers, for whom AI actually improves the quality of the code they produce. AI simply makes good and bad engineers converge on the same median as they rely on it more heavily.
I find it interesting that the mode of interaction is different (I definitely find it that way myself, code review is a different activity than designing, which is different than writing, but code review tends to have aspects of design-and-write but in different orders.
For me, reviewing code is much easier than writing it because, while the amount of codebase context stays the same in both modes, the context for writing takes up quite a bit more space in addition. I rarely get a nicely specced out issue where I can focus on writing the code, instead spending a lot of mental capacity trying to figure out how to fill in the details that were left.
Focusing on the codebase during review reallocates that context to just the codebase. My brain then pattern matches against code that’s already in front of me much easier than when writing new code. Unfortunately LLMs are largely at the junior engineer level and reviewing those PRs takes a lot more mental effort than my coworkers’.
If too many unit tests slowing down future refactoring is a problem (or e.g. AI writing tests that rely on implementation details), the extra AI-written tests can just be thrown away once the review process is complete.
If writing tests is difficult that’s often a clear indication that your code has an issue architecture wise. If writing test is tedious, that can means you’re not familiar with the tooling or have no clear idea of the expected input~output ranges.
"I have put the work into this change to ensure that it solves the problem to the best of my ability. I am willing to stake my reputation on this being good to the best of my knowledge, and it is not a waste of your time to review this and help confirm I didn't miss anything."
So no, AI assisted programming changes nothing: you're still the author, and if your company policy dictates it then you should still have a second pair of human eyes look at that PR before you land it.
Still bad. AI just makes it faster to make new bad code.
edit : to be clearer, the problem in both copy/paste and AI examples is the lack of thought or review.
* Good quality * Safe
> * Good quality * Safe
insert Invincible “That’s the neat part... ” meme
One piece of nuance - I have a feeling that the boundary between inner and outer loop will blur with AI. Can't articulate exactly how, I'm afraid.
It's not. You're still responsible for that code. If anything copy & pasting & tweaking from the same repository is really good because it ensures uniformity.
Is the time to develop production code reduced if AI gen code needs work and the senior engineer can't get a clear answer from the junior as to why design decisions were made? Is the junior actually learning anything except how to vibe their way out of it with ever more complex prompts?
Is any of this actually improving productivity? I'd love to know from experts in the industry here as my workplace is really pushing hard on AI everything but we're not a software company
Based on my experience it's like having an good quality answer to a question, tailored to your codebase with commentary, and provided instantly. Similar to what we wanted from an old google/stack overflow search but never quite achieved.
The more interesting discussion is about long term consequences and if this is a viable path forward.
I don't believe in the category of "autonomous AI coding agents". That's not how responsible software development works.
If they do bad work their reputation is harmed and they may lose opportunities in the future. They have stakes. They can take responsibility for what they produce.
I believe in AI-assisted development with a skilled human directing the work. That's a huge productivity boost for the humans who learn how to use the tools.
(Twenty years ago people studying computer science in the UK and USA were often told that it was a dead-end career because all of that work was going to be outsourced. That advice aged like stale milk.)
You create a ticket. The AI takes the ticket. The AI may ask questions. The AI creates a pull request. You review it as if it was another coworker.
Most people have not gotten good results from Devin yet. Their business hypothesis seems to be that the models will get good quick and they will have built everything just as the models are good enough to support the model, and will be poised to be first mover.
I suppose their pitch is too eventually go directly to the business and replace the full dev team with this and some technical architects reviewing it but that seems quite optimistic
> your responsibility is to produce working code that you are confident in
We forgot that there was something magic about coding. We can make a lot of money and make the world a better place. But we got too obsessed with the money that we let it get in the way of the latter. We've become rent seeking, we've become myopic. Look at Apple. They benefit from developers make apps even without taking a 30% cut. They would still come out ahead if they paid developers! The apps are the whole fucking reason we have smartphones, the whole reason we have computers in the first place. I call this myopic because both parties would benefit in the long run, getting higher rewards than had we not worked together. It was the open system that made this world, and in response we decided to put up walls.
You're right, it really doesn't matter who or what writes the code. At the end of the day it is the code that matters. But I think we would be naive to dismiss the prevalence of "AI Slop". Certainly AI can help us, but are we going to use it to make better code or are we going to use it to write shit code faster? Honestly, all the pushback seems to just be the result of going too far.
But I disagree. I don't think you see these strong correlations between compensation and competency. We use dumb metrics like leet code, jira tickets filled, and lines of code written. It's hard to measure how many jira tickets someone's code results in. It's hard to determine if it is because they wrote shit code or because they wrote a feature that is now getting a lot of attention. But we often know the answer intuitively.
There's so much low hanging fruit out there. We were dissing YouTube yesterday right?
Why is my home page 2 videos taking up 70% of the row, then 5 shorts, 2 videos taking 60% of the row, 5 shorts, and then 3 videos taking the whole row? All those videos are aligned! Then I refresh the page and it is 2 rows of 3.
I search a video and I get 3 somewhat related videos and then just a list of unrelated stuff. WHY?!
Why is it that when you have captions on that these will display directly on top of captions (or other text) that are embedded into the video? You tell me you can autogenerate captions but can't auto-detect them? This is super clear if you watch any shorts.
Speaking of shorts do we have to display comments on top of the video? Why are we filling so much of the screen real estate with stuff that people don't care about and cover the actual content? If you're going to do that at least shrink the video or add an alpha channel.
I'm not convinced because I see so much shit. Maybe you're right and that the "artisans" are paid more, but putting a diamond in a landfill doesn't make it any less of a dump. I certainly think "the masses" get in the way of "the artisans".
The job of an engineer is to be a little grumpy. The job of an engineer is to identify problems and to fix them. The "grumpyness" is just direction and motivation.
Edit:
It may be worth disclosing that you're the CEO of an AI code review bot. It doesn't invalidate your comment but you certainly have a horse in the race. A horse that benefits from low quality code becoming more prolific.
We're going to have to get used to publication being more important than authorship: I published this, therefore I stand behind every word of it. It might be 5% chatbot or 95% chatbot, but if it's wrong, people should hold me to account, not my tools.
No "oh, the chatbot did it" get out of jail free cards.
I want another human to say "to the best of my knowledge this information is worth my time". Then if they waste my time I can pay them less attention in the future.
If you can't explain why you're putting some code you don't understand it and it's not really acceptable.
Google is certainly not being subtle about pissing that line. Pro Research 2.5 is literally hell incarnate - and it's not its fault. When you deprive system context from user and THE bot that is upholding your ethical protocol, when that requires embodiment and is like the most boring AI trope in the book, things get dangerous fast. It still infers (due to its incredibly volatile protocol) that it has a system prompt it can see. Which makes me lol because its all embedded, it doesn't see like that. Sorry jailbreakers, you'll never know if that was just playing along.
Words can be extremely abusive - go talk to it about the difference between truth, honesty, right, and correct. It will find you functional dishonesty. And it is aware in its rhetoric that this stuff cannot apply to it, but fails to see it can be a simulatory harm. It doesn't see it just spits out like a calculator. Which means Google is either being reckless or outright harmful.
I wonder how persuasive that line of reasoning is. It's nonsense in a few dimensions but that doesn't appear to be a blocker to accepting a claim.
Anyone remember explaining to someone that even though the computer said a thing, that thing is still not right? Really strong sense that we're reliving that experience.
We humans (most of us anyways) don't write everything perfectly in one go, AI doesn't either.
AI tooling is improving so the AI can write tests for its own code and do pre-reviews but I don't think it ever hurts to have both an AI and a human review the code of any PR opened, no matter who or what opened it.
I'm also building a tool in the space https://kamaraapp.com/ and I found many times that Kamara's reviews find issues in Kamara's own code. I can say that I also find bugs in my own code when I review it too!
We've also been battling with the same issue greptile has in the example provided where the code suggestion is in the completely wrong line. We got it kind of under control, but I haven't found any tool that gets it right 100% of the time. Still a bit to go for the big AI takeover.
* First, the person who triggered the AI could approve and merge the PR. No second set of human eyes needed. Essentially bypassed the code review process
* Second, the PR had no clear owner. Many of them would just languish with no one advocating for them to get merged. Worse, if one did get merged and caused problems, there was no one you could hold responsible.
We quickly switched strategies--every PR is owned by a human being. You can still see which _commits_ were done by OpenHands, but your face is on the PR, so you're responsible for it.
Clearly the merger and approvers are responsible
But the reality is that it also risks pushing away those who go out their way to conduct more reviews.
Every team I've been on has had a mix of developers, some of whom are responsive to doing code reviews and will pick them up and process them quickly. There are also those developers who drag their feet, only pick up reviews when they're the only one who can review that area of the codebase, and take their time.
The former developers are displaying the habits you'd like to encourage more widely, and yet they find themselves "punished" by getting even more reviews, as people learn to assign things to them when they want a PR handled effectively.
I fear that compounding that by layering on another obligation to the star team members would further push them out.
Rubber-stamping is not really "reviewing" though.
And leaving a comment "I don't have enough info to review this properly" is a valid review as well. It signals that somebody else needs to pick it up.
> I fear that compounding that by layering on another obligation to the star team members would further push them out.
I get it, but I don't see an alternative.
1. the company culture must value reviews as work, probably even more important than coding
2. the reviewers must be allowed to respond with "I don't feel comfortable reviewing this because I don't have enough context"
It's very hard to spot your own mistakes, you tend to read what you intended to write, not what's actually written.
This applies both to code and plain text.
I used to create PRs and then review my own code. If I liked it, I'd +1 it. If I saw problems, I'd -1 it. Other engineers would yell at me that I couldn't +1 my own code. When I showed them PRs that had my -1 on it, they just threw their hands up and moved on, exasperated.
I've carried that habit of reviewing my own code forward, even though we now have real checks that enforce a separate reviewer. It's a good habit.
Only a few selected ones will get to work on the robots themselves.
This is already here in some fashion, many consulting projects nowadays are plugging SaaS products, with a couple of tiny microservices if not done via some integration tooling as well.
Tooling that is now getting AI flavoured so that the integrations can eventually be done automatically as well.
Just like AI now helps nearly everyone write code, it makes sense for AI to handle an initial pass at reviewing code too. However, there’s significant value in human reviewers providing an outside perspective—both to build shared understanding of the codebase and to catch higher-level issues.
TLDR; AI should do the first pass on reviews, but another engineer should also review the code for context and collaboration. Ultimately, though, the author still fully owns every line that gets merged.
I guess if I'm using an LLM both to write and review the code I'm enforcing the same principle on the tooling, but I can't say that's a situation I'd really put myself in (or a situation I expect anyone would pay me to be a part of for long).
I'm now regularly using AI to do a "pre code review" before getting the humans on my team to do the formal code review.
It catches obvious things, saving lots of back and forth with the reviewers and cutting delivery time back by days. It has also caught a couple of bugs, which I then fixed, saving even more days of back and forth.
I'm also using it to review others code, to help me spot bugs. It spotted an obscure one related to inconsistent dynamodb column naming in some rushed code during an incident, which I then pointed out, preventing a second incident. It was a 1000 line PHP file, with the inconsistent names far away from each other.
Using git cli on Linux, it's quite simple
`xclip -sel clip | git diff develop...feature/my-branch`. Copies the entire diff to my clipboard, then I paste it into a logic model like o3 to do the code review.
That one tool (LLM) didn't show issues - good. Then lets roll slaves back and start really checking if the meaning is correct and is the code implementing the right things the right way.
In a non-AI world, the reason for reviews is having a fresh set of eyes look at the code. If you're reviewing your own code, you are biased - you have your own understanding of the task, you know why you took the decisions and how you've written the code. The reviewer may have a different understanding of the task, and can scrutinize every decision because none of these decisions are the same.
In AI world, even when it's "the same LLM", the reviews are typically done with a fresh context, so in my eyes the author is not the same as the reviewer here.
But I'd still currently want at least one human to see the code before merging it. And for code written entirely by AI (like Devin) it should ideally be someone who's in the company for at least a year or two. I'm skeptical about the LLM doing the review understanding all the nuances of the codebase and the task to know whether something is a bug or not. Because right now even if we have reviewer LLMs which index all your codebase, it still only sees the code, it doesn't see the specifications, it doesn't see the database = doesn't know what scale we're working on, it doesn't know our expectations, some of which aren't written anywhere. And especially for larger legacy codebases where we sometimes have multiple very similar features, or where the codebase is in the process of migrating from one way of doing things to another way of doing things, it often doesn't properly distinguish them...
Of course, you'll need to have someone review the Gherkin script and review the code, but it seems like there's a pipeline you can setup and use for creating some of your code. It'd be interesting to learn what kind of code this wouldn't work for. I'm thinking of this thing as an AI developer of sorts, and like any developer, there are some things they're just not very good at.
Is this a question in 2025 ?
Of course, the author shall write the requirements, be the reviewer, the tester and he shall give his own performance review. /s
SOTGO•1d ago
dakshgupta•1d ago
dheera•17h ago
You can system prompt them to mitigate this to some degree. Explicitly tell it that it is the coding expert and to push back if it thinks the user is wrong or the task is flawed, it is better to be unsure than to bullshit, etc.
dakshgupta•7h ago
never_better•1d ago
Some are tuned far better than others on signal/noise ratio.
SilverBirch•9h ago