As I develop it, I run into regressions where previously working features become broken. I'd like to keep iterating on it this way, since I have built perfectly working applications with AI. Do you have any tips for me? How did you successfully scale developing with AI?
janpio•3mo ago
logicallee•3mo ago
Did you have success having AI iterate on code fully covered by tests?
I began to add tests, however, currently I am manually testing after each change. This is because I asked ChatGPT for a research study of best practices for AI development, which it produced here [1]. It suggested:
>Notably, some found that Claude’s first attempt often includes excess or "over-engineered" code. A candid blog post mentioned Claude as a "real master at shitting in the code" if not guided properly – it can "generate a ton of unnecessary code… even when you ask for minimalism, it will slap on a pile of code with useless tests that outsmart themselves and don’t work."
and:
>a developer noted they initially tried having Claude maintain extensive docs and tests for everything, but realized this added too many points of failure (the AI would waste effort updating documentation instead of focusing on code). Over-engineering the process can backfire.
Due to these reasons, I have been testing in a manual way between iterations. (Though I develop using ChatGPT 5 as well as Claude, depending on the task.)
[1] https://chatgpt.com/share/68fbaeea-f528-800b-b090-1bb6b3b2ca...
janpio•3mo ago
Aside: I often remove some of the tests that seem superfluous to me, or explicitly ask for the minimal set of tests that still cover the functionality in the first place. Some models definitely can go "all in" on tests like a very eager intern that just learned about testing. For your cases where after a prompt you end up with broken functionality, just having an integration test that fails when the functionality breaks, might be enough.