I switched back to Rails for my side project a month ago and ai coding when doing not too complex stuff has been great. While the old NextJS code base was in shambles.
Before I was still doing a good chunk of the NextJS coding. I’m probably going to be directly coding less than 10% of the code base from here on out. I’m now spending time trying to automate things as much as possible, make my workflow better, and see what things can be coded without me in the loop. The stuff I’m talking about is basic CRUD and scraping/crawling.
For serious coding, I’d think coding yourself and having ai as your pair programmer is still the way to go.
1) Abstract data showing an increase in "productivity" ... CHECK
2) Completely lacking in any information on what was built with that "productivity" ... CHECK
Hilarious to read this on the backend of the most widely publicized AWS failure.
And guess what happens? Reality doesn't match expectations and everyone ends up miserable.
Good engineering orgs should have engineers deciding what tools are appropriate based on what they're trying to do.
This sounds a lot like Tesla's Fake Self Driving. It self drives right up to the crash, then the user is blamed.
Part of being a mature engineer is knowing when to use which tools, and accepting responsibility for your decisions.
It's not that different from collaborating with a junior engineer. This one can just churn out a lot more code, and has occasional flashes of brilliance, and occasional flashes of inanity.
These guys actually seem rattled now.
"The Arithmetic of AI-Assisted Coding Looks Marginal" would be the more honest article title.
The way to code going forward with AI is Test Driven Development. The code itself no longer matters. You give the AI a set of requirements, ie. tests that need to pass, and then let it code whatever way it needs to in order to fulfill those requirements. That's it. The new reality us programmers need to face is that code itself has an exact value of $0. That's because AI can generate it, and with every new iteration of the AI, the internal code will get better. What matters now are the prompts.
I always thought TDD was garbage, but now with AI it's the only thing that makes sense. The code itself doesn't matter at all, the only thing that matters is the tests that will prove to the AI that their code is good enough. It can be dogshit code but if it passes all the tests, then it's "good enough". Then, just wait a few months and then rerun the code generation with a new version of the AI and the code will be better. The humans don't need to know what the code actually is. If they find a bug, write a new test and force the AI to rewrite the code to include the new test.
I think TDD has really found its future now that AI coding is here to stay. Human code doesn't matter anymore and in fact I would wager that modifying AI generated code is as bad and a burden. We will need to make sure the test cases are accurate and describe what the AI needs to generate, but that's it.
It's sort of like a director telling an AI the high level plot of a movie, vs giving an AI the actual storyboards. The storyboards will better capture the vision of the director vs just a high level plot description, in my opinion.
writing comprehensive tests is harder than writing the code
oh no! another bug!
The reason AI code generation works so well is a) it is text based- the training data is huge and b) the output is not the final result but a human readable blueprint (source code), ready to be made fit by a human who can form an abstract idea of the whole in his head. The final product is the compiled machine code, we use compilers to do that, not LLMs.
Ai genereted code is not suitable to be directly transferred to the final product awaiting validation by TDD, it would simply be very inefficient to do so.
Congratulations, you invented end-to-end testing.
"We have yellow flags when the build breaks!"
Congratulations! You invented backpressure.
Every team has different needs and path dependencies, so settles on a different interpretation of CI/CD and software eng process. Productizing anything in this space is going to be an uphill battle to yank away teams' hard-earned processes.
Productizing process is hard but it's been done before! When paired with a LOT of spruiking it can really progress the field. It's how we got the first CI/CD tools (eg. https://en.wikipedia.org/wiki/CruiseControl) and testing libraries (eg. pytest)
So I wish you luck!
Waiting to see anyone show even a month ahead of schedule after 6 months.
AI can't keep up because its context window is full of yesteryear's wrong ideas about what next month will look like.
Madmallard•2h ago
now AWS guy doing it !
"My team is no different—we are producing code at 10x of typical high-velocity team. That's not hyperbole - we've actually collected and analyzed the metrics."
Rofl
"The Cost-Benefit Rebalance"
In here he basically just talks about setting up mock dependencies and introducing intermittent failures into them. Mock dependencies have been around for decades, nothing new here.
It sounds like this test system you set up is as time consuming as solving the actual problems you're trying to solve, so what time are you saving?
"Driving Fast Requires Tighter Feedback Loop"
Yes if you're code-vomiting with agents and your test infrastructure isn't rock solid things will fall apart fast, that's obvious. But setting up a rock solid test infrastructure for your system involves basically solving most of the hard problems in the first place. So again, what? What value are you gaining here?
"The communication bottleneck"
Amazon was doing this when I worked there 12 years ago. We all sat in the same room.
"The gains are real - our team's 10x throughput increase isn't theoretical, it's measurable."
Show the data and proof. Doubt.
Yeah I don't know. This reads like complete nonsense honestly.
Paraphrasing: "AI will give us huge gains, and we're already seeing it. But our pipelines and testing will need to be way stronger to withstand the massive increase in velocity!"
Velocity to do what? What are you guys even doing?
Amazon is firing 30,000 people by the way.
p1necone•1h ago
lispisok•58m ago
CharlesW•51m ago
Can you point me to anyone who knows what they're talking about declaring that LOC is the best productivity metric for AI-assisted software development?
chipsrafferty•29m ago
Can you point me to where the author of this article gives any proof to the claim of 10x increased productivity other than the screenshot of their git commits, which shows more squares in recent weeks? I know git commits could be net deleting code rather than adding code, but that's still using LOC, or number of commits as a proxy to it, as a metric.
CharlesW•15m ago
Yes, I'm also reading that the author believes commit velocity is one reflection of the productivity increases they're seeing, but I assume they're not a moron and has access to many other signals they're not sharing with us. Probably stuff like: https://www.amazon.science/blog/measuring-the-effectiveness-...
blibble•15m ago
and this guy didn't survive there for a decade by challenging it