This is an undergraduate level persuasive essay masquerading as an academic paper.

The title implies some novel research or a review of existing research that that clearly shows agents are better at code review than humans but then provides this single paragraph on the review capabilities of agents:

> Beyond general software engineering, several strands of work speak specifically to the capabilities that code review re- quires. Pornprasit and Tantithamthavorn evaluate LLM-based automated review in industrial settings and find that agents detect the same categories of defect that human reviewers target: correctness errors, security weaknesses, performance inefficiencies, and style violations [12]. Li et al. demonstrate that CodeReviewer produces actionable inline comments at quality that is at least comparable to those of trained human reviewers on a significant fraction of the evaluation set [11].

This feels like some combination of monumentally stupid and incredibly naive.

> "To support the claim that coding agents can displace human code review, we survey evidence of agent capability across three dimensions: benchmark performance on softwareengineering tasks, review-specific capabilities, and developer productivity with deployed tools"

Not sure I can agree with this premise, especially since there seems to be a complete lack of "real-world results" in this evaluation. This strikes me as being written by a theorist, who's only experience with Quality Assurance exists in studies or papers.

Big eye roll from me.

"Can't scale due to too many PRs" neglects answering questions like: Are these PRs valuable? Are they just additional PRs to right the wrongs of previous ill-conceived PRs? How much churn is going on here? Is the influx of PRs a permanent state, or something that we'll only live through temporarily because we have a lot of little things we can set our agents upon, but after they're done we'll return to a normal work cadence?

I don't believe that a human being wrote this paper. The "review-specific capabilities" section is obviously the only one that matters to the thesis, and it does not actually point towards any data indicating that coding agents supersede human inspection. An LLM, though, could easily be distracted or prompted into making the leap from "same categories" + "comparable on a significant fraction of the evaluation set" to "superior".

This seems to take the view that code review is essentially linting for simple issues. Our team is fairly small so "code review" usually involves QA and everything else you might want to do before something is pushed to production.

But yeah - I can have one LLM check another LLMs work. Kind of a waste of tokens for most PRs.

What in the LLM psychosis is this

This is honestly the biggest battle with AI driven development right now. You have these extremely potent tools that can output a ton of really great code if they are wielded correctly but there is simply no way to keep up with their output at human review pace (which was already slower than human code creation pace).

I think the only real solution is to add increasingly strict guardrails that can be enforced with a combination of more AI agents and actual executable contracts. The other aspect is through using languages and tools that densify correctness. i.e languages like Rust that have very rich type system so both review and design can be focused on a small by volume slice which is the core types. The other main tools for densifying correctness are formal methods, (model checking, etc), fuzzing/property based testing and static analysis.

All of these tools are cheaper to use than they once were because of lot of the minutiae can be handled AI agents while core invariants can receive heavy human scrutiny.

IMO generative AI is here to stay in development so may as well get ahead of the game and start using these tools to try get the best out of it.

>the naive integration in which agents write code and humans remain the mandatory reviewers is a dead end because it neither provides meaningful assurance nor scales with AI-assisted throughput.

Who said it has to "scale with AI-assisted throughput"? AI can produce code all day, the goal is not to fill storage with AI code, is to make products, following product tradeoffs, timelines, and decisions.

Ahhhhhhhhhh hahahahahahahahahahaha. No. We have all seen how bad these LLMs are at coding, let's not have it review it's own failed coding attempt, too.

Here is my take:

- I disagree with the contributions

- I suspect most of the paper was written / edited by "generative AI"

It is a shame that "researchers" have started using generative AI to such a large degree as it now masks the voice of the person. Generative AI tends to claim things that are not true and tends to use words that are unsuitable. This text leaves a bad taste in my mouth.

(Edits: formatting)

Ironically, the number of sentences in bold tells me that an AI wrote this with no human review.

And here I am huffing and puffing because the latest models keep adding nonsense to my codebase.

Have you seen the decisions LLMs make? They write code like the worst developers I know. They're lazy, short-sighted, and impossible to teach.

It took 6 turns for opus 4.8 to attempt to make an illegal move in chess against me.

Yes you can setup guardrails and validations and other things, but as long its primary brain is demonstrably full of holes, I feel obligation to at very least be always be capable to take over the code. To me this means it should stay human-maintainable, and that's for humans to decide what is and isn't, one review at a time.

I do think there's some real limits to code review as we do it and that we need to figure out what is next, to some degree. That's just my belief here. Orgs keeping to the past are going to be kind of miserable; leads already had a ton of responsibility for understanding it all, and it feels like that just isn't feasible to keep on with.

That's a pretty bike take, but I think a bit less than what this paper is saying.

A more mild sub-argument I really enjoyed recently was just that we have basically an obligation to have AI code review:

> Nuclear opinion: I don’t care if you use AI to write code,¹ but making a human review a change before an LLM is disrespectful.

> Why am I catching your bugs if a computer could have done it? It’s like asking to review code that doesn’t compile or pass CI.

https://bsky.app/profile/filippo.abyssdomain.expert/post/3mo...

This makes me also really appreciate a thread that shows up recently, where folks were talking about their code/repo forges. Using tickets/PRs as rendezvous for where people and agents come together had very high appeal to me. Being able to surface discussion in a shared context seems crucial.