> I’ve tried it on one of my pet projects and it produced an entire wiki full of dev docs
Did it? No, it didn't. "Wiki" is not a synonym for "project documentation". (You could set up a wiki to manage the documentation for your project. But that's not what any of these things are about.)
These aren't wikis.
brickLock: The lock of the brick.
brickDrink: The drink of the brick.
brickWink: The wink of the brick.
My role as a writer is then to (1) extract net-new information out of the team, (2) figure out how all of that new info fits together, (3) figure out the implications of that info for readers/users, and then (4) assemble it in an attractive manner.
An autogenerated code wiki (or a lazy human) can presumably do the fourth step, but it can't do the first three steps preceding it, and without those preceding steps you're just rearranging known data. There are times where that can be helpful, but it's more often just gloss
It doesn't seem impossible for an LLM to go "hmmm, the way this repo passes configurations around isn't standard. I should focus more on that." But that's a level of understanding I don't think they currently have
I think they do, at least in some of the cases, especially if it's something well represented in the dataset. I've been surprised sometimes by the insights it provides, and other times it's completely useless. That's one of the problems, it's unreliable, so you have to treat all info it gives you with doubt. But, anyways, at times it makes very surprising and seeming intelligent observations. It's worth at least considering it and thinking it through.
It's so odd and random it seems like there must be more to it.
The biggest problem we face right now is that the large majority of people are terrible writers and can't recognize why this is awful. It really felt like the moment before chatgpt arrived we were coming into a new world where the craft of writing was surging in popularity and making a difference. That all feels lost.
This kind of post makes me have hope.
PowerPoint created a very similar Eternal September.
After PowerPoint got done applying lipstick to the pig, your incoherent garbage was presented on screen with neat fonts and bullet points, so I couldn't immediately judge your lack of preparedness. I had to wade through 10+ minutes of your slop before realizing you were wholly unprepared.
AI is just extending the amount of time I have to waste before realizing that what I'm looking at is slop. I'm not happy about this.
I have used AI to ask specific questions about a codebase and it has been helpful in narrowing the search base. Think of AI as probable cause, not evidence. It speeds up getting to the truth but cant be trusted as the truth.
And here's one of the primary issues. It's possible to create good docs with an agentic workflow, but it takes time (could be up to several hours depending on the repo size) and a ton of tokens, processing every line of code multiple times with multiple levels of summarization and different lenses of analysis and synthesis. This is very valuable for legacy systems no one understands, for instance.
The problem is that that doesn't scale to a DeepWiki-style product available for literally every repo on GitHub. It's way less flashy of a demo. And this is a demo.
Probably slightly off topic but I’d be curious to hear what other people want out of automated systems in this space. I have so many half baked ideas and would love to hear what’s others think/want.
Google Releases CodeWiki
rurban•2mo ago