According to this, we'll all be reading the feelings journals of our LLM children and scolding them for cheating on our carefully crafted exams instead of, you know, making things. We'll read psychology books, apparently.
I like reading and tinkering directly. If this is real, the field is going to leave that behind.
It's just that we designed this iteration of technology foundationally on people's fickle and emotional reddit posts among other things.
It's a designed-in limitation, and kind of a happy accident it's capable of writing code at all. And clearly carries forward a lot of baggage...
That's fine.
Perhaps we can RL away some of this or perhaps there's something else we need. Idk, but this is the problem when engineers are the customer, designer, and target audience.
either your business isn't successful, so you're coding when you shouldn't be, or cosplaying coding with Claude, or you're lying, or you're telling us about your expensive and unproductive hobby.
How much do you spend on AI? What's your annual profit?
edit: oh cosplaying as a CEO. I see. Nice WPEngine landing page Mr AppBind.com CEO. Better have Claude fix your website! I guess that agent needs therapy...
Is Claude really "learning new skills" when you feed it a book, or does it present it like that because you're prompting encourages that sort of response-behavior. I feel like it has to demo Claude with the new skills and Claude without.
Maybe I'm a curmudgeon but most of these types of blogs feel like marketing pieces with the important bit is that so much is left unsaid and not shown, that it comes off like a kid trying to hype up their own work without the benefit of nuance or depth.
The most challenging part when working with coding agents is that they seem to do well initially on a small code base with low complexity. Once the codebase gets bigger with lots of non-trivial connections and patterns, they almost always experience tunnel vision when asked to do anything non-trivial, leading to increased tech debt.
I made a crude first stab at an approach that at least uses similar steps and structure to compare the effectiveness of AI agents. My approach was used on a small toy problem, but one that was complex enough the agents couldn't one-shot and required error correction.
It was enough to show significant differences, but scaling this to larger projects and multiple runs would be pretty difficult.
https://mattwigdahl.substack.com/p/claude-code-vs-codex-cli-...
"We should have rigorous evaluations of whether or not [thing] works." seems like an incredibly obvious thought.
But in the realm of LLM-enabled use cases they're also expensive. You'd need to recruit dozens, perhaps even hundreds of developers to do this, with extensive observation and rating of the results.
So rather than actually try to measure the efficacy, we just get blog posts with cherry-picked example of "LLM does something cool". Everything is just anecdata.
This is also the biggest barrier to actual LLM adoption for many, many applications. The gap between "it does something REALLY IMPRESSIVE 40% of the time and shits the bed otherwise" and "production system" is a yawning chasm.
It applies to using LLMs too. I guess the one largest difference here is that LLM has few enough companies with abundant enough money pushing it to make it trivial for them to run a test like this. So the fact that they aren't doing that also says a lot.
> "We should have rigorous evaluations of whether or not [thing] works." seems like an incredibly obvious thought.
Heh, I'd rephrase the first part to:
> What you're getting at is the heart of the problem with software development though, isn't it?
If that's what we need to do, don't we already have the answer to the question?
I’ve similarly been using spec.md and running to-do.md files that capture detailed descriptions of the problems and their scoped history. I mark each of my to-do’s with informational tags: [BUG], [FEAT], etc.
I point the LLM to the exact to-do (or section of to-do’s) with the spec.md in memory and let it work.
This has been working very well for me.
On top of that, it doesn't sound enjoyable. Anti slop sessions? Seriously?
Lastly, the largest problem I have with LLMs is that they are seemingly incapable of stopping to ask clarifying questions. This is because they do not have a true model of what is going on. Instead they truly are next token generators. A software engineer would never just slop out an entire feature based on the first discussion with a stakeholder and then expect the stakeholder to continuously refine their statement until the right thing is slopped out. That's just not how it works and it makes very little sense.
If asking clarifying questions is plausible output text for LLMs, this may work effectively.
Didn’t you just describe Agile?
I'm not highlighting this to gloat or to prove a point. If anything in the past I have underestimated how big LLMs were going to be. Anyone so inclined can take the chance to point and laugh at how stupid and wrong that was. Done? Great.
I don't think I've been intentionally avoiding coding assistants and as a matter of fact I have been using Claude Code since the literal day it first previewed, and yet it doesn't feel, not even one bit, that you can take your hands off the wheel. Many are acting as if writing any code manually means "you're holding it wrong", which I feel it's just not true.
When using it for code or architecture or design, I’m always watching for signs that it is going off the rails. Then I usually write code myself for a while, to keep the structure and key details of whatever I’m doing correct.
C'mon, such self-congratulatory "Look at My Potency: How I'm using Nicknack.exe" fluffies always were and always will be a staple of the IT industry.
However for complex projects IMO one must read what was written by the llm … every actual word.
When it ‘got away’ from me, in each case I left something in the llm written markdown that I should have removed.
99% “I can ask for that later” and 1% “that’s a good idea i hadn’t considered” might be the right ratio when reading an llm generated plan/spec/workunit.
Breaking work into single context passes … 50-60k tokens in sonnet 4.5 has had typically fantastic results for me.
My side project is using lean 4 and a carelessly left in ‘validate’ rather than ‘verify’ lead down a hilariously complicated path equivalent to matching an output against a known string.
I recovered, but it wasn’t obvious to me that was happening. I however would not be able to write lean proofs myself, so diagnosing the problem and fixing it is a small price to be able to mechanically verify part of my software is correct.
maybe explicit support from providers would make it feasible?
I don’t like the looks of that. If I used this, how soon before those instructions would be in conflict with my actual priorities?
Not everything can be the first law.
Spend some time digging around in his https://github.com/obra/Superpowers repo.
I wrote some notes on this last night: https://simonwillison.net/2025/Oct/10/superpowers/
The packaged collection is very cool and so is the idea of automatically adding new abilities, but I’m not fully convinced that this concept of skills is that much better than having custom commands+sub-agents. I’ll have to play around with it these next few days and compare.
also whats with overuse of these words in ai space. every team is 'cracked' , everything is 'wild'. real car salesman vibes.
> This isn’t necessarily surprising, but it’s worth noting anyway. Claude Sonnet 4.5 is capable of building a full Datasette plugin now.
I do worry a bit about how often I use positive adjectives. If something isn't notable I won't write about it though. In this particle case Jesse's prompting / skills stuff really does deserve the superlatives IMO.
what have u built with to come to those conclusions ? is this too much to ask.
It's "wild" because, among many other experiments, Jesse has experimented with giving Claude a "feelings journal" and prompting it using Graphviz DOT diagrams.
For my previous writing and work on this you can consult my blog - here's the AI-assisted programming tag: https://simonwillison.net/tags/ai-assisted-programming/
great.
Not only do I have know everything about the code, data and domain, but now I need to understand this whole AI system which is a meta skill of its own.
I fear I may never be able catch up till someone comes along and simplifies it for pleb consumption.
Admittedly that stance is easiest to take if you were old enough, experienced enough already by the time this era hit.
I’ve found you have to use Claude Code to do something small. And as you do it iterate on the CLAUDE.md input prompt to refine what it does by default. As it doesn't do it your way, change it to see if you can fix how it works. The agent is then equivalent to calling chatgpt / sonnet 1000 times a hour. So these refinements (skills in the post are a meta approach) are all about how to tune the workflow to be more accurate for your project and fit your mental model. So as you tune the md file you’ll start to feel what is possible and understand agent capabilities much better.
So short story you have to try it, but long story its the iteration of the meta prompt approach that teaches you whats possible.
I've found that a single CLAUDE.md does really well at guiding it how I want it to behave. For me that's making it take small steps and stop to ask me questions frequently, so it's more like we're pairing than I'm sending it off solo to work on a task. I'm sure that's not to everyone's taste but it works for me (and I say this as someone who was an agent-sceptic until quite recently).
Fwiw my ~/.claude/CLAUDE.md is 2.2K / 49 lines.
No matter what you are told, there is no silver bullet. Precisely defining the problem is always the hard part. And the best way to precisely define a problem and its solution is code.
I’ll let other people fight swarms of bots building… well who knows what. Maybe someday it will deliver useful stuff, but I’m highly skeptical.
just remember that it works the same for everyone: you input text, magic happens, text comes out.
if you can properly explain a software engineering problem in plain language, you're an expert in using LLMs. everything on top of that people experimenting or trying to build the next big thing.
Jesse on Bluesky: https://bsky.app/profile/s.ly/post/3m2srmkergc2p
> The core of it is VERY token light. It pulls in one doc of fewer than 2k tokens. As it needs bits of the process, it runs a shell script to search for them. The long end to end chat for the planning and implementation process for that todo list app was 100k tokens.
> It uses subagents to manage token-heavy stuff, including all the actual implementation.
Save yourself the experience of having to write and maintain prompts like this.
That paper is about using persuasion prompts to overcome trained in "safety" refusals, not to improve prompt conformance.
The whole world is changing around us and nothing is secure. I would not gamble that the market for our engineering careers is safe with so much disruption happening.
Tools like Lovable are going to put lots of pressure on technical web designers.
Business processes may conform to the new shape and channels for information delivery, causing more consolidation and less duplication.
Or perhaps the barrier to entry for new engineers, in a worldwide marketplace, lowers dramatically. We have accessible new tools to teach, new tools to translate, new tools to coordinate...
And that's just the bear case where nothing improves from what we have today.
Is this just someone who has tingly feelings about Claude reiterating stuff back to them? cuz that's what an LLM does/can do
This is voodoo.
It likely works - but knowing that YAGNI is a thing, means at some level you are invoking a cultural touchstone for a very specific group of humans.
Edit -
I dug into the superpowers and skills for a bit. Definitely learned from it.
There’s stuff that doesn’t make sense to me on a conceptual basis. For example in the skill to preserve productive tensions. There’s a part that goes :
> The trade-off is real and won't disappear with clever engineering
There’s no dimension for “valid” or prediction for tradeoff.
I can guess that if the preceding context already outlines tradeoffs clearly, or somehow encodes that there is no clever solution that threads the needle - then this section can work.
Just imagining what dimensions must be encoding some of this suggests that it’s … it won’t work for situations where the example wasn’t already encoded in the training. (Not sure how to phrase it)
I was struggling to find the exact reason this type of article bugs me so much, and I think "voodoo" is precisely the correct phrase to sum up my feelings.
I don't mean that as a judgement on the utility of LLMs or that reading about what different users have tried out to increase that utility isn't valuable. But if someone asked me how to most effectively get started with coding agents, my instinct is to answer (a) carefully and (b) probably every approach works somewhat.
This is actually a really cool idea. I think a lot of the good scaffolding right now is things like “use TDD” bit if you link citations to the book, then it can perhaps extract more relevant wisdom and context (just like I would by reading the book), weather than using the generic averaged interpretation of TDD derived from the internet.
I do like the idea of giving your Claude a reading list and some spare tokens on the weekend where you’re not working, and having it explore new ideas and techniques to bring back to your common CLAUDE.md.
... So, we're refactoring the process of prompting?
> As Claude and I build new skills, one of the things I ask it to do is to "test" the skills on a set of subagents to ensure that the skills were comprehensible, complete, and that the subagents would comply with them. (Claude now thinks of this as TDD for skills and uses its RED/GREEN TDD skill as part of the skill creation skill.)
> The first time we played this game, Claude told me that the subagents had gotten a perfect score. After a bit of prodding, I discovered that Claude was quizzing the subagents like they were on a gameshow. This was less than useful. I asked to switch to realistic scenarios that put pressure on the agents, to better simulate what they might actually do.
... and debugging it?
... How many other basic techniques of SWEng will be rediscovered for the English programming language?
idk, but if you already assume that the LLM knows what TDD is (it probably ingested ~100 whole books about it), why are we feeding a short (and imo confusing) version of that back to it before the actual prompt?
i feel like a lot of projects like this that are supposed to give LLMs "superpowers" or whatever by prompt engineering are operating on the wrong assumption that LLMs are self-learning and can be made 10x smarter just by adding a bit of magic text that the LLM itself produced before the actual prompt.
ofc context matters and if i have a repetitive tasks, i write down my constraints and requirements and paste that in before every prompt that fits this task. but that's just part of the specific context of what i'm trying to do. it's not giving the LLM superpowers, it's just providing context.
i've read a few posts like this now, but what i am always missing is actual examples of how it produces objectively better results compared to just prompting without the whole "you have skill X" thing.
I've been exploring AI for two years now. It's certainly upgraded itself from the toy classification to a basic utility. However, I increasingly run into its limitations and find reverting to pre-LLM ways of working more robust, faster, and more mentally sustainable.
Does someone have concrete examples of integrating LLM in a workflow that pushes state-of-the-art development practices & value creation further?
@/Users/jesse/.claude/plugins/cache/Superpowers/...
gjm11•5h ago
(There probably are some. Most likely I notice the bad ones more than the good ones. But it does seem like I notice a lot of bad ones, and never any good ones.)
[EDITED to add:] For context, the actual article title begins "Superpowers: How I'm using ..." and it has been auto-rewritten to "Superpowers: I'm using ...", which completely changes what "Superpowers" is understood as applying to. (The actual intention: superpowers for LLM coding agents. The meaning after the change: LLM coding agents as superpowers for humans.)
add-sub-mul-div•4h ago
dvfjsdhgfv•3h ago
bryanrasmussen•3h ago