According to this, we'll all be reading the feelings journals of our LLM children and scolding them for cheating on our carefully crafted exams instead of, you know, making things. We'll read psychology books, apparently.
I like reading and tinkering directly. If this is real, the field is going to leave that behind.
It's just that we designed this iteration of technology foundationally on people's fickle and emotional reddit posts among other things.
It's a designed-in limitation, and kind of a happy accident it's capable of writing code at all. And clearly carries forward a lot of baggage...
That's fine.
Perhaps we can RL away some of this or perhaps there's something else we need. Idk, but this is the problem when engineers are the customer, designer, and target audience.
either your business isn't successful, so you're coding when you shouldn't be, or cosplaying coding with Claude, or you're lying, or you're telling us about your expensive and unproductive hobby.
How much do you spend on AI? What's your annual profit?
edit: oh cosplaying as a CEO. I see. Nice WPEngine landing page Mr AppBind.com CEO. Better have Claude fix your website! I guess that agent needs therapy...
I hate managing people.
What are we doing?
Is Claude really "learning new skills" when you feed it a book, or does it present it like that because you're prompting encourages that sort of response-behavior. I feel like it has to demo Claude with the new skills and Claude without.
Maybe I'm a curmudgeon but most of these types of blogs feel like marketing pieces with the important bit is that so much is left unsaid and not shown, that it comes off like a kid trying to hype up their own work without the benefit of nuance or depth.
The most challenging part when working with coding agents is that they seem to do well initially on a small code base with low complexity. Once the codebase gets bigger with lots of non-trivial connections and patterns, they almost always experience tunnel vision when asked to do anything non-trivial, leading to increased tech debt.
I made a crude first stab at an approach that at least uses similar steps and structure to compare the effectiveness of AI agents. My approach was used on a small toy problem, but one that was complex enough the agents couldn't one-shot and required error correction.
It was enough to show significant differences, but scaling this to larger projects and multiple runs would be pretty difficult.
https://mattwigdahl.substack.com/p/claude-code-vs-codex-cli-...
"We should have rigorous evaluations of whether or not [thing] works." seems like an incredibly obvious thought.
But in the realm of LLM-enabled use cases they're also expensive. You'd need to recruit dozens, perhaps even hundreds of developers to do this, with extensive observation and rating of the results.
So rather than actually try to measure the efficacy, we just get blog posts with cherry-picked example of "LLM does something cool". Everything is just anecdata.
This is also the biggest barrier to actual LLM adoption for many, many applications. The gap between "it does something REALLY IMPRESSIVE 40% of the time and shits the bed otherwise" and "production system" is a yawning chasm.
It applies to using LLMs too. I guess the one largest difference here is that LLM has few enough companies with abundant enough money pushing it to make it trivial for them to run a test like this. So the fact that they aren't doing that also says a lot.
> "We should have rigorous evaluations of whether or not [thing] works." seems like an incredibly obvious thought.
Heh, I'd rephrase the first part to:
> What you're getting at is the heart of the problem with software development though, isn't it?
> Trial participants saved an average of 56 minutes a working day when using AICAs
That feels accurate to me, but again I'm just going on vibes :P
I agree. I think there are too many resources, examples, and live streams out there for someone to credibly claim at this point that these tools have no value and are all hype. I think the nuance is in how and where you apply it, what your expectations and tolerances are, and what your working style is. They are bad at many things, but there is tremendous value to be discovered. The loudest people on both sides of this debate are typically wrong in similar ways imo.
You absolutely can quantify the results of a chaotic black box, in the same way you can quantify the bias of a loaded die without examining its molecular structure.
If that's what we need to do, don't we already have the answer to the question?
I’ve similarly been using spec.md and running to-do.md files that capture detailed descriptions of the problems and their scoped history. I mark each of my to-do’s with informational tags: [BUG], [FEAT], etc.
I point the LLM to the exact to-do (or section of to-do’s) with the spec.md in memory and let it work.
This has been working very well for me.
https://gist.github.com/JacobBumgarner/d29b660cb81a227885acc...
On top of that, it doesn't sound enjoyable. Anti slop sessions? Seriously?
Lastly, the largest problem I have with LLMs is that they are seemingly incapable of stopping to ask clarifying questions. This is because they do not have a true model of what is going on. Instead they truly are next token generators. A software engineer would never just slop out an entire feature based on the first discussion with a stakeholder and then expect the stakeholder to continuously refine their statement until the right thing is slopped out. That's just not how it works and it makes very little sense.
If asking clarifying questions is plausible output text for LLMs, this may work effectively.
1. Context isn't infinite
2. Both Claude and OpenAI get increasingly dumb after 30-50% of context had been filled
Even the "million token context window" becomes useless once it's filled to 30-50% and the model starts "forgetting" useful things like existing components, utility functions, AGENTS.md instructions etc.
Even a junior programmer can search and remember instructions and parts of the codebase. All current AI tools have to be reminded to recreate the world from scratch every time, and promptly forget random parts of it.
If that's your idea of trivial then you and I have very different standards in terms of what's a trivial change and what isn't.
Yes, the core of that pull requests is an hour or two of thinking, the rest is ancillary noise. The LLM took away the need for the noise.
If your definition of trivial is signal/noise ratio, then, sure, relatively little signal in a lot of noise. If your definition of "trivial" hinges on total complexity over time, then this kicks the pants of manual writing.
I'd assume OP did the classic senior engineer stick of "I can understand the core idea quickly, therefore it can't be hard". Whereas Mitchel did the heavy lifting of actually shipping the "not hard" idea - still understanding the core idea quickly, and then not getting bogged down in unnecessary details.
That's the beauty of LLMs - it turns the dream of "I could write that in a weekend" into actually reality, where it before was always empty bluster.
Didn’t you just describe Agile?
Sorry couldn’t resist. Agile’s point was getting feedback during the process rather than after something is complete enough to be shipped thus minimizing risk and avoiding wasted effort.
Instead people are splitting up major projects into tiny shippable features and calling that agile while missing the point.
"Splitting up major projects into tiny shippable features and calling that agile" feels like a much more accurate description of what I've experienced.
I wish I'd gotten to see the real thing(s) so I could at least have an informed opinion.
The manager for the only team I think actually checked all the agile boxes had a UI background so she thought in terms of mock-ups, backend, and polishing as different tasks and was constantly getting client feedback between each stage. That specific approach isn’t universal, the feedback as part of the process definitely should be though.
What was a little surreal is the pace felt slow day to day but we were getting a lot done and it looked extremely polished while being essentially bug free at the end. An experienced team avoiding heavy processes, technical debt, and wasted effort goes a long way.
The failure modes I've personally seen is an organization that isn't interested in cooperating or the person running the show is more interested in process than people. But I'd say those teams would struggle no matter what.
Ultimately, I think it's up to the engineering side to do its best to leverage the process for better results, and I've seen very little of that (and it's of course always been the PM side's fault).
And you're right: use what works for you. I just haven't seen anything that felt like it actually worked. Maybe one problem is people iterating so fast/often they don't actually know why it's not working.
Either way, the ability to produce "working software" (as the manifesto puts it) in "frequent" iterations (often just seconds with an LLM!) and iterate on feedback is core to Agile.
I'm not highlighting this to gloat or to prove a point. If anything in the past I have underestimated how big LLMs were going to be. Anyone so inclined can take the chance to point and laugh at how stupid and wrong that was. Done? Great.
I don't think I've been intentionally avoiding coding assistants and as a matter of fact I have been using Claude Code since the literal day it first previewed, and yet it doesn't feel, not even one bit, that you can take your hands off the wheel. Many are acting as if writing any code manually means "you're holding it wrong", which I feel it's just not true.
A year ago I was using GitHub Copilot autocomplete in VS Code and occasionally asking ChatGPT or Claude to help write me a short function or two.
Today I have Claude Code and Codex CLI and Codex Web running, often in parallel, hunting down and resolving bugs and proposing system designs and collaborating with me on detailed specs and then turning those specs into working code with passing tests.
The cognitive overhead today is far higher than it was a year ago.
You can go further and faster, but you can get to a point where you're out of juice miles from home, and getting back is a chuffing nightmare.
Also, you discover that you're putting on weight and not getting that same buzz you got on your old pushbike.
When using it for code or architecture or design, I’m always watching for signs that it is going off the rails. Then I usually write code myself for a while, to keep the structure and key details of whatever I’m doing correct.
- incessantly duplicating already existing functionality: utility functions, UI components etc.
- skipping required parameters like passing current user/actor to DB-related functions
- completely ignoring large and small chunks of existing UI and UI-related functionality like layouts or existing styles
- using ad-hoc DB queries or even iterating over full datasets in memory instead of setting up proper DB queries
And so on and so forth.
YYMV of course depending on language and project
... which is exactly the kind of thing this new skills mechanism is designed to solve.
That they routinely ignore.
> which means documenting the functionality they should be aware of
Which means spending inordinate amounts of time writing down about every single function and component and css and style which can otherwise be easily discovered by just searching. Or by looking at adjacent files.
> which is exactly the kind of thing this new skills mechanism is designed to solve.
I tried it yesterday. It immediately duplicated functionality, ignored existing styles and components, and created ad-hoc queries. It did feel like there were fewer times when it did that, but it's hard to quantify.
I also have to remember all of the new code that’s coming together, and keep it from re-inventing other parts of the codebase, etc.
More productive, but hard work.
P.s. always thought you were one of those irrational AI bros. Later, found that you were super reasonable. That's the way it should be. And thank you!
For context, I mainly do game development so I'm viewing it through that lens - but I find it easier to debug something bad than to write it from scratch. It's more intensive than doing it yourself but probably more productive too.
It's funny because not far below this comment there is someone doing literally this.
C'mon, such self-congratulatory "Look at My Potency: How I'm using Nicknack.exe" fluffies always were and always will be a staple of the IT industry.
However for complex projects IMO one must read what was written by the llm … every actual word.
When it ‘got away’ from me, in each case I left something in the llm written markdown that I should have removed.
99% “I can ask for that later” and 1% “that’s a good idea i hadn’t considered” might be the right ratio when reading an llm generated plan/spec/workunit.
Breaking work into single context passes … 50-60k tokens in sonnet 4.5 has had typically fantastic results for me.
My side project is using lean 4 and a carelessly left in ‘validate’ rather than ‘verify’ lead down a hilariously complicated path equivalent to matching an output against a known string.
I recovered, but it wasn’t obvious to me that was happening. I however would not be able to write lean proofs myself, so diagnosing the problem and fixing it is a small price to be able to mechanically verify part of my software is correct.
maybe explicit support from providers would make it feasible?
The costs of self hosting some reasonable size models for a development group of various sizes is what I would want to know before investing in the skills to do a high usage style that might be being mostly bankrolled by investors for now.
I don’t like the looks of that. If I used this, how soon before those instructions would be in conflict with my actual priorities?
Not everything can be the first law.
Spend some time digging around in his https://github.com/obra/Superpowers repo.
I wrote some notes on this last night: https://simonwillison.net/2025/Oct/10/superpowers/
The packaged collection is very cool and so is the idea of automatically adding new abilities, but I’m not fully convinced that this concept of skills is that much better than having custom commands+sub-agents. I’ll have to play around with it these next few days and compare.
Some of these skills are probably better as programmed workflows that the LLM is forced to go through to improve reliability/consistency, that's what I've found in my own agents, rather than using English to guide the LLM and trusting it to follow the prescribed set of steps needed. Some mix of LLMs (choosing skills, executing the fuzzy parts of them) and just plain code (orchestration of skills) seems like the best bet to me and what I'm pursuing.
The ability to isolate context-noisy subtasks (like agentically searching through a large codebase by grepping through dozens of irrelevant files to find the one you actually need) unlocks much longer-running loops, and therefore much more complex tasks.
And you don't need a system this complicated to take advantage of it. Literally just a simple "codebase-searcher" agent (and Claude can vibe the agent definition for you) is enough to see the benefit first-hand. Once you see it, if you're like me, you will see opportunities for subagents everywhere.
Using them in a way that doesn't waste tokens is something I haven't fully figured out out yet!
What am I missing?
Also, memory itself can be a tool the subagent calls to retrieve only the stuff it needs.
Not only do I have know everything about the code, data and domain, but now I need to understand this whole AI system which is a meta skill of its own.
I fear I may never be able catch up till someone comes along and simplifies it for pleb consumption.
Admittedly that stance is easiest to take if you were old enough, experienced enough already by the time this era hit.
I don't see the coding as the hard or critical part of my work, so I don't put effort into accelerating or delegating that part.
I’ve found you have to use Claude Code to do something small. And as you do it iterate on the CLAUDE.md input prompt to refine what it does by default. As it doesn't do it your way, change it to see if you can fix how it works. The agent is then equivalent to calling chatgpt / sonnet 1000 times a hour. So these refinements (skills in the post are a meta approach) are all about how to tune the workflow to be more accurate for your project and fit your mental model. So as you tune the md file you’ll start to feel what is possible and understand agent capabilities much better.
So short story you have to try it, but long story its the iteration of the meta prompt approach that teaches you whats possible.
I've found that a single CLAUDE.md does really well at guiding it how I want it to behave. For me that's making it take small steps and stop to ask me questions frequently, so it's more like we're pairing than I'm sending it off solo to work on a task. I'm sure that's not to everyone's taste but it works for me (and I say this as someone who was an agent-sceptic until quite recently).
Fwiw my ~/.claude/CLAUDE.md is 2.2K / 49 lines.
No matter what you are told, there is no silver bullet. Precisely defining the problem is always the hard part. And the best way to precisely define a problem and its solution is code.
I’ll let other people fight swarms of bots building… well who knows what. Maybe someday it will deliver useful stuff, but I’m highly skeptical.
just remember that it works the same for everyone: you input text, magic happens, text comes out.
if you can properly explain a software engineering problem in plain language, you're an expert in using LLMs. everything on top of that people experimenting or trying to build the next big thing.
Just give it a few months. If some technics really work, it’ll get streamlined.
Jesse on Bluesky: https://bsky.app/profile/s.ly/post/3m2srmkergc2p
> The core of it is VERY token light. It pulls in one doc of fewer than 2k tokens. As it needs bits of the process, it runs a shell script to search for them. The long end to end chat for the planning and implementation process for that todo list app was 100k tokens.
> It uses subagents to manage token-heavy stuff, including all the actual implementation.
Save yourself the experience of having to write and maintain prompts like this.
That paper is about using persuasion prompts to overcome trained in "safety" refusals, not to improve prompt conformance.
It is the thing I find most irritating about working with llms and agents. They seem forever a generation behind in capabilities that are self referential.
"Phase 2 will take about one week"
No, Claude, it won't, because you you and I will bang this thing out in a few hours.
(The latest Claude has a `/context` command that’s great at measuring this stuff btw)
The whole world is changing around us and nothing is secure. I would not gamble that the market for our engineering careers is safe with so much disruption happening.
Tools like Lovable are going to put lots of pressure on technical web designers.
Business processes may conform to the new shape and channels for information delivery, causing more consolidation and less duplication.
Or perhaps the barrier to entry for new engineers, in a worldwide marketplace, lowers dramatically. We have accessible new tools to teach, new tools to translate, new tools to coordinate...
And that's just the bear case where nothing improves from what we have today.
Is this just someone who has tingly feelings about Claude reiterating stuff back to them? cuz that's what an LLM does/can do
This is voodoo.
It likely works - but knowing that YAGNI is a thing, means at some level you are invoking a cultural touchstone for a very specific group of humans.
Edit -
I dug into the superpowers and skills for a bit. Definitely learned from it.
There’s stuff that doesn’t make sense to me on a conceptual basis. For example in the skill to preserve productive tensions. There’s a part that goes :
> The trade-off is real and won't disappear with clever engineering
There’s no dimension for “valid” or prediction for tradeoff.
I can guess that if the preceding context already outlines tradeoffs clearly, or somehow encodes that there is no clever solution that threads the needle - then this section can work.
Just imagining what dimensions must be encoding some of this suggests that it’s … it won’t work for situations where the example wasn’t already encoded in the training. (Not sure how to phrase it)
I was struggling to find the exact reason this type of article bugs me so much, and I think "voodoo" is precisely the correct phrase to sum up my feelings.
I don't mean that as a judgement on the utility of LLMs or that reading about what different users have tried out to increase that utility isn't valuable. But if someone asked me how to most effectively get started with coding agents, my instinct is to answer (a) carefully and (b) probably every approach works somewhat.
This is actually a really cool idea. I think a lot of the good scaffolding right now is things like “use TDD” bit if you link citations to the book, then it can perhaps extract more relevant wisdom and context (just like I would by reading the book), weather than using the generic averaged interpretation of TDD derived from the internet.
I do like the idea of giving your Claude a reading list and some spare tokens on the weekend where you’re not working, and having it explore new ideas and techniques to bring back to your common CLAUDE.md.
... So, we're refactoring the process of prompting?
> As Claude and I build new skills, one of the things I ask it to do is to "test" the skills on a set of subagents to ensure that the skills were comprehensible, complete, and that the subagents would comply with them. (Claude now thinks of this as TDD for skills and uses its RED/GREEN TDD skill as part of the skill creation skill.)
> The first time we played this game, Claude told me that the subagents had gotten a perfect score. After a bit of prodding, I discovered that Claude was quizzing the subagents like they were on a gameshow. This was less than useful. I asked to switch to realistic scenarios that put pressure on the agents, to better simulate what they might actually do.
... and debugging it?
... How many other basic techniques of SWEng will be rediscovered for the English programming language?
idk, but if you already assume that the LLM knows what TDD is (it probably ingested ~100 whole books about it), why are we feeding a short (and imo confusing) version of that back to it before the actual prompt?
i feel like a lot of projects like this that are supposed to give LLMs "superpowers" or whatever by prompt engineering are operating on the wrong assumption that LLMs are self-learning and can be made 10x smarter just by adding a bit of magic text that the LLM itself produced before the actual prompt.
ofc context matters and if i have a repetitive tasks, i write down my constraints and requirements and paste that in before every prompt that fits this task. but that's just part of the specific context of what i'm trying to do. it's not giving the LLM superpowers, it's just providing context.
i've read a few posts like this now, but what i am always missing is actual examples of how it produces objectively better results compared to just prompting without the whole "you have skill X" thing.
I’ve found the most helpful things for me is just voice to Whisper to LLMs, managing token usage effectively and restarting chats when necessary, and giving it quantified ways to check when its work is done (say, AI-Unit-Tests with apis or playwright tests.) Also, every file I own is markdown haha.
And obviously having different AI chats for specialized tasks (the way the math works on these models makes this have much better results!)
All of this has allowed me to still be in the PM role like he said, but without burning down a needless forest on having it reevaluate things in its training set lol. But why would we go back to vendor lock in with Claude? Not to mention how much more powerful 5o-codex-high is, it’s not even close
The good thing about what he said is getting AI to work with AI, I have found this to be incredibly useful in promoting, and segmenting out roles
I would say that systems like this are about getting the agent to correctly choose the precisely correct context snippet for the exact subtask it's doing at a given point within a larger workflow. Obviously you could also do that manually, but that doesn't scale to running many agents in parallel, or running automomously for longer durations.
I've been exploring AI for two years now. It's certainly upgraded itself from the toy classification to a basic utility. However, I increasingly run into its limitations and find reverting to pre-LLM ways of working more robust, faster, and more mentally sustainable.
Does someone have concrete examples of integrating LLM in a workflow that pushes state-of-the-art development practices & value creation further?
@/Users/jesse/.claude/plugins/cache/Superpowers/...
I agree with this, it’s frustrating that in 2025 apps are still polluting my home dir.
XDG may not be perfect but wiping related data for apps following it is straightforward. There are a few directories to delete instead of 1, but still consistently structured at least.
No, no. Stop.
What is this? What're we doing here?
This goes past developping with AI into something completely different.
Just because AI coding is a radical shift doesn't mean everything has changed. There needs to be some semblance of structure and design. Instead what we're getting is straight up vodoo nonsense.
It always has been. Starting with the term "AI" itself.
Articles like these read the same way to me as any OpenAI announcement from the past 5 years. A bunch of technical mumbo jumbo laced with hyperbole, grand promises of how the technology is changing the world, and similar platitudes. I've learned to filter most of it out.
Occasionally I'll stumble upon an actually useful and practical tidbit of information which I can apply in my own workflow, which does involve LLMs, but most of the time it's just noise.
Maybe not in this case.
For the AI to create a solution, it has to come up with a vector for your intention and goals. It makes some sense for an AI trained on human persuasion materials (basically, everything has a rhetorical aspect) to also track human persuasion features for intentions.
However, results will vary. Just as people trying to deploy rhetorical techniques (and ridiculous power stances) often come off as foolish, I believe trying to hack your intention vector with all-caps and super-superlatives won't always work as intended (pun intended).
Still, if you find yourself not getting what you want, and you check your prompt and find some persuasion feature missing (e.g., authority), I think it's worth trying to add something on point.
Why?
> However, results will vary.
Like in voodoo?
I'm sorry to be dismissive, but your comment is entirely dismissing the point it's replying to, without any explanation as to why it's wrong. "You are holding it wrong" is not a cogent (or respectful) response to "we need to understand how our tools work to do engineering".
I have no interest in trying to understand the thought process of people who write and work like this. They're more interested in chasing the latest overhyped trends produced by tech companies and influencers, than actually producing quality software that solves real-world problems. It's some weird product of the tech and social media echo chambers they perpetually live in, which I find difficult to describe.
But apparently I have to learn about "skills" and "superpowers" now... Give me a break.
> The first time we played this game, Claude told me that the subagents had gotten a perfect score. After a bit of prodding, I discovered that Claude was quizzing the subagents like they were on a gameshow. This was less than useful. I asked to switch to realistic scenarios that put pressure on the agents, to better simulate what they might actually do.
Also his Claude says shit a lot
At least it's an MIT license, but since AI output isn't copyrightable, I'm unsure what the point is since people can legally ignore the license.
The copyright office has been quite clear (rightly so imo) that AI output is not protected by copyright without substantial human creative expression in the final product and purely prompt-created works simply don't qualify.
Indeed, I expect people muddling their codebases with AI output are going to find themselves in an interesting position of having to prove how much code humans actually wrote to enforce copyright claims if their code ever gets leaked.
And while yes, the US is just one country, but it does have a bit of an outsized software development industry. I also haven't hear of any other countries lining up to give machine-generated works copyright protection.
I've tried agents every now and then, recently for something very simple- add an option to request csb format in a data api.
The results were, well, not good. . . I ended up undoing literally all changes because writing from scratch was a lot easier than trying to refactor the total mess it has made from what I'd have thought was a trivial feature.
I haven't done loads of prompt engineering etc, in all honesty it seems a lot of work when I haven't seen promise yet in the tool.
I see articles like this, and I always wonder, am I the outlier or is the writer? My experience of agentic AI is so hugely different to what some people are finding.
I've had success with eg spitting out templated html; sometimes with css; sometimes with writing tests where I'm very specific about what I want (set up these structures, test this condition), etc. It's mediocre (good start, very far from production) with writing screens in react native. It does slightly better on rails, but far from production ready.
After that, it kinda works, but my effort level to turn the output into working code is higher than just writing it myself.
what's the likelihood that the solution exists in the github repo in a way that the machine can recognize as relevant to your prompt?
If many versions of the solution exist, due to the problem's common occurrence, and if you can evaluate the LLM's output, then you're good to go.
For example, I have found Claude Code and Codex to be tremendously helpful for my web development work. But my results for writing Zig are much worse. The gap in usefulness of agents between tasks is very big.
The skill ceiling for using agents is also surprisingly high. Planning before coding, learning agent capabilities, environment setup, and context engineering can make a pretty massive difference to results. This can all be a big time sink though, and I'm not sure if it's really worth it if agents don't already work decently well for the work you do.
But with the performance gaps between domains, and the skill curve, I can definitely understand why there is such a divide between people claiming agents are ridiculously overhyped, and people who claim coding is fundamentally changing.
When I see a pro-AI person insisting that they are fully automated, I often scour their recent comments to find code or git repos they have shared. You find something every now and again.
My thinking is that I want to use this stuff, but don't find the agentic AI at all effective. I must be doing something wrong! So I should learn from the real world success of others.
A regular pattern is they say they're using vibe coding for complex problems. You check, and they're trivial features.
One egregious example was a basic randomizer to pick a string from a predetermined set, and save that value into an existing table to re-use later.
To me that's a trivial feature, a 15-30 minute task in a codebase I'm familiar with.
For this extremely AI bullish developer it was described as a major feature. The prompts were timestamped and it took them 1/2 day using coding agents.
They were sharing their .claude folder. It had 50 odd md files in it. I sampled a bunch of them and most of them boiled down to:
'You are an expert [dev/QA/architect/PM/tester]. Ultrathink. Be good'.
Worse, I looked at their linkedin, and on paper they looked experienced. Seeing their code, they were not.
There's a subset of the "fully automated" coders who are just bad. They are incapable of judging how bad AI code is. But vocally, and often aggressively, advocate for it.
Some are good, but I just can't replicate their success. And they're clearly also still hand-writing a lot of the code.
In web development, where I get the most out of agents, I am still only using them for implementing basic things. I will write anything even moderately complex, as agents often make the wrong assumptions somewhere. And then there's also manual work required to review and tidy up agent output. But there's just so much grunt work in web development from adding to a DB schema, writing a migration, adding the data to your model, exposing it in an API endpoint, and finally showing it on a page. None of that is complicated, so agents are pretty good at it.
The best of them are rediscovering basic software project management and post about it on every social media site and their substack like they discovered something brand new :)
"Turns out if you plan first, then iterate on the plan and split the plan into manageable chunks, development is a lot smoother!!!11 (subscribe to my AI podcast)"
No shit, Sherlock. I wish they read a book once or twice.
If you just tell it some vague feature to make, it's gonna do whatever it's gonna do and maybe it will be good, maybe it won't. It probably won't. The more specific you are the better it will do.
Instead of trying to 100x or 1000x your effort, try to just 2x or 3x it. Give it small specific tasks and check the work thoroughly, use it as an extension of yourself rather than a separate "agent".
I can tell it to write a function and it'll do pretty well. I can ask it to fix things if it doesn't do it the way I want. This is all easy. Maybe I can even get it to write a whole class at once or maybe I can get it to write a class in a few iterations.
The key here is I'm in control, I'm doing the design, I'm making the decisions. I can ask it how I should approach a problem and often it'll have great suggestions. I can ask it to improve a function I've written and it'll do pretty well. Some times really well.
The point is I'm using it as a tool I'm not using it to do my job for me. I use it to help me think I don't use it to think for me. I don't let it run away from me and edit a whole bunch of files etc, I keep it on a tight leash.
I'm sold now. I am, indisputably, a better software developer with LLMs in my toolbelt. They help me write better code, faster, while learning things faster and easier, it's really good. Reliability isn't a problem when I keep a close eye on it. It's only a problem if you try to get it to do a whole big task on it's own.
- Case 1 was troubleshooting what turned out to be a complex and messy dependency injection issue. I got pulled in to unblock a team member, who was struggling with the issue. My efforts were a dead-end, but Claude (Code) managed to spot a very odd configuration issue. The codebase is a large, legacy one.
- Case 2 was the same codebase, I again got pulled in to unblock a team mate, investigating why some integration tests were running individually, but not when run as a group. Clearly there was a pretty obvious smoking gun, and I managed to isolate the issue after about 15-30 minutes of debugging. I had set Claude on the goose chase as well, and as I closed the call with my teammate, I noticed it had found the same exact two lines that were causing the issue.
Clearly, it occasionally does insane stuff, or lies its little pants off. The number of times where it "got me" are fairly low, however, and its usefulness to me is extreme. In the cases above, it out-did a teammate who has at least 10 years of experience, and equalled me in the one case and outdid me in the other, with over 25 years now. I have a similar wonderment to your situation, but the opposite: "how are people NOT finding value in this?".
My experience is coding agents work best for either absolute beginners, or for lead engineers who have experience building and training teams. Getting good results out of coding agents is a lot like getting good results out of interns: You need to explain clearly what you want, ask them to explain what they plan to do, give feedback on the plan, and then very carefully review the results. You need to write up your preferred coding style, you need a document that explains "how to work on this project", you need to establish rigorous automated quality checks, etc. Using a coding agent heavily is a lot like being promoted to "technical lead", with all the tradeoffs that entails.
Here's a recent discussion of a good blog post on the subject: https://news.ycombinator.com/item?id=45503867
I have gotten some very nice results out of Sonnet 4.5 this past week. But it required using my "technical management" skills very heavily. And it required lots extremely careful code review. Clear documentation, robust QA, and code review are the main bottlenecks.
I mean, the time I spent writing AGENTS.md wasn't wasted. I'm writing down a lot of stuff I used to teach in pairing sessions.
But only on micro tasks, coming with explicit instructions, inside a very well documented architecture.
Give AI freedom of expression and they will never find first principals in their training data. You will receive code that is not performant and when analyzing the output, AI will try to convince you that it is. If the task goes beyond your domain, you may believe the wrong principals are ok.
Meaning, the previous work may have involved multiple CLI sessions, summaries dumped to various markdown files like documentation files, plan files, issue files, PR-descriptions etc. Then when starting new work with a code agent you have to hunt down all of this scattered context from various md files and session logs to fill in background for the code-agent about what was recently done.
I see many workflows that help with working on a fresh feature or fix, but nothing that addresses context-recall. But maybe the OP workflow or others do that, I haven’t dug too deep into them.
``` Claude Code v2.0.13 Sonnet 4.5 (with 1M token context) Claude Max /Users/jesse/tmp/new-tool/.worktrees/todo-cli ```
How does this person have access to Sonnet 4.5 with 1m token context? I don't see this referenced anywhere when I search or when I ask Claude about it.
I haven’t gotten access yet.
One of the nice things about Codex (GPT-5) is the supposed 400k token context (although performance starts to deteriorate when you get to 80% context usage).
Having to beg and emotionally manipulate an agent into doing what you want goes so far beyond black-box that I find it difficult to believe these people actually get useful work done using these tools.
I generally consider myself pro-ai in the workplace, but this nonsense is starting to change my mind.
gjm11•3mo ago
(There probably are some. Most likely I notice the bad ones more than the good ones. But it does seem like I notice a lot of bad ones, and never any good ones.)
[EDITED to add:] For context, the actual article title begins "Superpowers: How I'm using ..." and it has been auto-rewritten to "Superpowers: I'm using ...", which completely changes what "Superpowers" is understood as applying to. (The actual intention: superpowers for LLM coding agents. The meaning after the change: LLM coding agents as superpowers for humans.)
add-sub-mul-div•3mo ago
dvfjsdhgfv•3mo ago
bryanrasmussen•3mo ago