eugenics is a good example of why we dont want to play god, because the types of data humans use to determine good and bad: visual phenotypes, easy to measure physical features, are rarely correlated with broader goals such as diverse gene pools, reistance to diease, and social cohesion.

Theres little evidence that we are, in the long term, as society, going to benefit.

think of things like pilots. most pilots now get experience from armed forces. they learn manual flight controls. as auto-pilot improves, the pipeline for pilots narrows.

At some point, we will have full self flying planes, and the number of pilots capable of flying will deceease.

so, the question of, can a novel emergency in flight always fallback to a human?

no, eventually there will be no fallback.

so the pace of change does matter and if it goes yo fast then we are endangering multiple societal goods.

you can argue, without evidence the overall errors will go down, but at some point we will cross the rubicon where theres no human fallback to keep the flight in motion.