It seems a combination of changes in model behavior and increasing code base size—emphasis on the latter. The larger the code base and documented history/progress (e.g. through updates to completed work and instructions, as in .md files), the more drift starts to set in.
At a certain point, you get collapse. This is where it can't maintain consistency within a single unit of work. It can manifest, for example, as adding a similar element to a view, but styling it in a completely different manner than existing elements. It literally feels random.
It can also manifest as incoherent or poorly implemented logic and behavior. I've followed different advice, restructured projects, entered plan mode, added deterministic checks, styles, rules, etc. But, it feels like whack-a-mole and it's unclear that it actually adheres to these rules in the first place.
Then, there's the question of it moving the problem. In response to my complaints of inconsistent UI generation, it recommended a style check to run after its UI changes. Of course, that begs the question of why it doesn't just apply those rules in the initial generation.
But, I am at a place now where it spins up AI agents to do a full UI/UX review—Playwright and all. Yet, it's still drifting and not catching it. This is consistent with its unit tests—on my latest project it has now generated a suite of hundreds of somewhat time consuming tests that it constantly runs pre-commit, and not a single test has ever failed. Meanwhile, there have been a number of bugs. When questioned, it said it had been testing the "wrong layer" and had essentially generated self-confirming tests.
My question is who is really winning here? When querying Google, it points to so many claims of success and approaches that supposedly contribute to that success. But, when you break down the underlying source for those claims, it appears to be largely Reddit.
Yet, pressing it further reveals that virtually every actual study shows either little to no productivity gains or actual regression—particularly where experienced engineers are concerned.
So, who here has developed a quality production application of any complexity using AI coding? If anyone, which harness/model, and what techniques did you use? Also, can you point us to your repo?