I completely agree, getting an objective measure for the developer experience from these various tools is not easy. On one hand, you have a series of benchmarks from LLM providers. While reflecting some degree of fitness to specific tasks, they often fail to translate to real-world usage. On the other hand, you have the tool providers with different features and product claims, and user anecdotes for very different use-cases.
The attempt with this post was to summarize my experience across some of these tools and highlight some specific features which worked better for me vs others. Given how quickly things are changing in this space, the primary conclusion is that using a tool day-to-day, discovering its strengths and deficiencies and working to eliminate the ones with high hit-rate is best at this point.
I agree with you. I feel the challenge is that using AI coding tools is still an art, and not a science. That's why we see many qualitative studies that sometimes conflict with each other.
In this case, we found the following interesting. That's why we nudged Shikhar to blog about his experience and put a disclaimer at the top.
* Our codebase is in Ruby and follows a design pattern uncommon industry * We don't have a horse in this game * I haven't seen an evaluation that evaluates coding tools in (a) coding, (b) testing, and (c) debugging dimension
Good: I can prototype things very quickly thanks to these tools
Bad: After couple of vibe coding iterations, I don't have a mental model of the project.
Good: When I open my past projects where I have very good mental models, I can come up with a nice prompt and build anything quickly again.
Bad: After couple of iterations I become lazy, and eventually my mental models break.
There's definitely a use for these tools. But be careful, job of engineers are not only coding but also training their memory to build solutions and bridge real world problem with software solution. If you lose this skill of thinking, you will be obsolete quickly
When I'm vibe coding something from scratch I don't have the mental model, I don't always review everything closely, and eventually it becomes an "AI project" that I'm just making requests against to hopefully achieve my goal.
CuriouslyC•1h ago