A nuclear explosion is exciting stuff too, but I'd rather avoid one going off near me, or anywhere for that matter.

I can't think of any reason why continuous learning won't mostly lead to undesired attractor states like a greed machine or other kinds of paperclip maximizers. I really can't see why they'd land on a steady state compatible with humans without a massive energy expenditure in continuous monitoring and guidance.

However 95% is still half the error rate of 90%, and 97.5% is half the error rate of that.

And when test time compute like reasoning and looping harnesses stack many inference acts with many tokens each, those seemingly small accuracy gains stack tremendously.