This is an undergraduate level persuasive essay masquerading as an academic paper.

The title implies some novel research or a review of existing research that that clearly shows agents are better at code review than humans but then provides this single paragraph on the review capabilities of agents:

> Beyond general software engineering, several strands of work speak specifically to the capabilities that code review re- quires. Pornprasit and Tantithamthavorn evaluate LLM-based automated review in industrial settings and find that agents detect the same categories of defect that human reviewers target: correctness errors, security weaknesses, performance inefficiencies, and style violations [12]. Li et al. demonstrate that CodeReviewer produces actionable inline comments at quality that is at least comparable to those of trained human reviewers on a significant fraction of the evaluation set [11].

This feels like some combination of monumentally stupid and incredibly naive.

> "To support the claim that coding agents can displace human code review, we survey evidence of agent capability across three dimensions: benchmark performance on softwareengineering tasks, review-specific capabilities, and developer productivity with deployed tools"

Not sure I can agree with this premise, especially since there seems to be a complete lack of "real-world results" in this evaluation. This strikes me as being written by a theorist, who's only experience with Quality Assurance exists in studies or papers.

Big eye roll from me.

"Can't scale due to too many PRs" neglects answering questions like: Are these PRs valuable? Are they just additional PRs to right the wrongs of previous ill-conceived PRs? How much churn is going on here? Is the influx of PRs a permanent state, or something that we'll only live through temporarily because we have a lot of little things we can set our agents upon, but after they're done we'll return to a normal work cadence?

I don't believe that a human being wrote this paper. The "review-specific capabilities" section is obviously the only one that matters to the thesis, and it does not actually point towards any data indicating that coding agents supersede human inspection. An LLM, though, could easily be distracted or prompted into making the leap from "same categories" + "comparable on a significant fraction of the evaluation set" to "superior".

This seems to take the view that code review is essentially linting for simple issues. Our team is fairly small so "code review" usually involves QA and everything else you might want to do before something is pushed to production.

But yeah - I can have one LLM check another LLMs work. Kind of a waste of tokens for most PRs.

What in the LLM psychosis is this

This is honestly the biggest battle with AI driven development right now. You have these extremely potent tools that can output a ton of really great code if they are wielded correctly but there is simply no way to keep up with their output at human review pace (which was already slower than human code creation pace).

I think the only real solution is to add increasingly strict guardrails that can be enforced with a combination of more AI agents and actual executable contracts. The other aspect is through using languages and tools that densify correctness. i.e languages like Rust that have very rich type system so both review and design can be focused on a small by volume slice which is the core types. The other main tools for densifying correctness are formal methods, (model checking, etc), fuzzing/property based testing and static analysis.

All of these tools are cheaper to use than they once were because of lot of the minutiae can be handled AI agents while core invariants can receive heavy human scrutiny.

IMO generative AI is here to stay in development so may as well get ahead of the game and start using these tools to try get the best out of it.

>the naive integration in which agents write code and humans remain the mandatory reviewers is a dead end because it neither provides meaningful assurance nor scales with AI-assisted throughput.

Who said it has to "scale with AI-assisted throughput"? AI can produce code all day, the goal is not to fill storage with AI code, is to make products, following product tradeoffs, timelines, and decisions.

Ahhhhhhhhhh hahahahahahahahahahaha. No. We have all seen how bad these LLMs are at coding, let's not have it review it's own failed coding attempt, too.

Here is my take:

- I disagree with the contributions

- I suspect most of the paper was written / edited by "generative AI"

It is a shame that "researchers" have started using generative AI to such a large degree as it now masks the voice of the person. Generative AI tends to claim things that are not true and tends to use words that are unsuitable. This text leaves a bad taste in my mouth.

(Edits: formatting)

Ironically, the number of sentences in bold tells me that an AI wrote this with no human review.

And here I am huffing and puffing because the latest models keep adding nonsense to my codebase.

Have you seen the decisions LLMs make? They write code like the worst developers I know. They're lazy, short-sighted, and impossible to teach.