I don't know why you would run an agent loop without TDD? Should you write code without test coverage? So something must run the tests to be sufficient anyway?
> TLDR; Based on Opus's judgment of the quality of the outcomes, there was no clearly discernable difference based on TDD workflow versus no TDD workflow. On the contrary, more than once Opus ranked the non-TDD workflow solutions slightly higher in design and test quality. There was also no meaningful difference in mutation scores across the solutions.
That's really surprising.
Was there a difference in cost?
Does it matter whether you ask the agent to write the tests first or last? Is one way TDD and the other not?
Without property testing or fuzzing or formal methods, could the code metrics in this experiment have been sufficient?
I don't know why you would run an agent loop without TDD? Should you write code without test coverage? So something must run the tests to be sufficient anyway?
You can have the agent write them after. The article is specifically about TDD, meaning writing a small test that will fail, making it pass, (refactoring), and repeating until the feature is finished.
Was there a difference in cost?
The author claims TDD used 3x tokens, but points out it might not be 3x the cost due to caching. It's near the end of the article.
westurner•22m ago
> TLDR; Based on Opus's judgment of the quality of the outcomes, there was no clearly discernable difference based on TDD workflow versus no TDD workflow. On the contrary, more than once Opus ranked the non-TDD workflow solutions slightly higher in design and test quality. There was also no meaningful difference in mutation scores across the solutions.
That's really surprising. Was there a difference in cost?
Does it matter whether you ask the agent to write the tests first or last? Is one way TDD and the other not?
Without property testing or fuzzing or formal methods, could the code metrics in this experiment have been sufficient?
Perhaps there's a TDD gauntlet loop that's as justified as tests for QC; https://news.ycombinator.com/item?id=49261545
groomlake•55s ago