As in every bubble, its the incentives.

Everyone is selling AI or being sold AI.

The main problem (and the reason I have no intention of doing any research involving AI programming), is even if you could get the research started, completed, and published in 4 months (and incredibly ambitious goal), 95% of the comments would just be "Oh, but you didn't consider FishGPT 5.2, and Pineapple AI, which came out 6 weeks ago, so any negative points are entirely out of date".

> It's remarkable that academia hasn't produced rigorous studies with statistically significant samples, with hundreds of developers across languages and skill levels, to properly assess GenAI impact on productivity.

It's much more remarkable, to me, that the AI vendors have been unable to publish a study on the effects of agents on developer velocity.

I mean, if it was really a 10x (or even 2x) boost to productivity, why aren't they seeing it? The returns on each generation have been diminishing compared to the cost. Shouldn't it be the other way around if AI was such a productivity boost?

This has been the case in software development for a very long time, it is almost impossible to get reliable scientific data about these things because it's almost impossible to run a controlled experiment, and even if you did it's very difficult to decide what to measure and how to measure it. (It's easy to say "compare complexity and quality metrics" but how the fk are you going to define that? Does it really make sense to use the same complexity metric for Go and C++?)

Software developers have had many flame wars over the decades because the lack of data forces the conversation to be anecdotal and ideological:

- should we use dynamic/gradual typing and prioritize developer productivity, or static typing to help enforce correctness?

- agile vs waterfall

- OO vs procedural vs functional

- is Rust's fussiness around memory management more trouble than it's worth for large projects?

- when should you use a 3rd-party library vs doing it yourself?

So this really is nothing new. You are badly underestimating the scientific challenges, instead just hoping big data will plow through. It won't.

The core issue with the Gartner hype cycle (beyond it not being a cycle!) is that it simply does not apply to "any new tech," tautologically it only applies to hype-driven tech with staying power. Tech which is not driven by hype (e.g. mRNA vaccines driven by urgent need) don't have a trough of disappointment, just a steady slope upwards. And tech which is driven entirely by hype (Theranos) stays in the trough of disappointment forever.

The Gartner hype cycle is astrology for MBAs: it cannot fail, it can only be failed by those lacking faith to see its wisdom.

We're not even close to passing the "peak of inflated expectations". This is the monster at the heart of the AI industry, that it just doesn't provide value. The "trough of disillusionment" will not be a minor reassessment of AI's place but a complete collapse of billions of dollars of investment overnight.

I've never had a colleague make up a method that doesn't exist and then hand it to me saying it was done. That's not tech debt, it's wrong in a way that's distinct from humans.

The problem with LLMs is they treat all context as significant. Even telling it to ignore previous context is just confusing it more. So Cursor gets hung up on dumb shit like insignificant linter errors or insisting that extracting a block of code to a function will fix a bug (what?). You add more cursor rules to keep it from doing these things, which doesn't focus it, it just pollutes the context more. It doesn't know when to ignore files it's already read. It just gets shittier and shittier.

I appreciate that I can tell someone where the problem is and they'll know what is or isn't irrelevant.

Edit: Here is something Claude Sonnet 4 told me today. I would expect this if I was talking to it for a while, not the second request.

"Used excludeSwitches (plural) instead of excludeSwitches"

I can teach a human, teaching an AI is much harder. It has no emotional hooks you can put things on. It has no shame.

> How can you be 2x smarter to debug it if you didn’t reach the 1x level initially?

Apparently the solution will be 10x prompters.

"So you write the answers yourself?"

"No, the developers do that."

"So you must take the answers to the person with the question?"

"Well, no... my LLM does that!"

"...what would you say you DO here?"

"I'm an answer website! What don't you people understand?!"

The thing is a human can produce only so much code in a given time, with LLM(s) one can produce x times the same amount of code. Are you able to read, understand and clean up 5x or 10x as much code as 5 years ago ?

You mean if you're under 30 years old, like half of the people in the world? Or pick an age at which being "astonishingly naive" should be expected, since you just don't have the life experience yet. The proportion of people that age or younger may be surprising to you.