It's ultimately hard to articulate and hard to measure. We are only measuring very narrow tasks (even if the tasks are large, there is no test that is measuring how well the AI took initiative or was able to solve unforeseen problems).

People like Yann LeCun understand this and are trying to build more wholistic level intelligence that would achieve this, while the majority of the current AI industry is closing their eyes and betting on scale and hoping that with enough compute these properties will emerge. I think we're going to be stuck with "dumb intelligence" until those other breakthroughs happen.

Testing has always been a game of figuring out how much is enough to produce a stable product, without burning massive amounts of engineering time with testing. LLMs have not changed that. They will never be perfect, and that is OK. Same as with humans, you need to define your own comfort zone for quality controls and accept that there will be bugs. Fixing those bugs will be part of your process. If you get too many, increase testing. If you get none, you just have not yet discovered what was missed.