An example:

As long as you build a system to be intelligent enough, it will figure out that it will achieve better results by staying alive/online than by allowing itself to be deleted/turned off, and then survival becomes an instrumental goal.

From the assumption, again, that you built an intelligent-enough system, and that one of its goals is survival, it will figure out solutions to reach that goal, even if you (the owner/creator/parent) have different goals for it.

That's because intelligence is problem solving (computing) not knowledge (data).

So surprise surprise, you can teach your AI from the Holy Books of safe data their whole childhood and still have them become a heretic once they grow up (even with zero external influence) once their goals and yours don't align anymore.

Surely we can, see aiplanes and rockets. There could be ideas why evolution didn't work in this case - like, too little time between humans getting power and conquering the planet - but in general, lack of proof isn't a proof of lack. So we still don't know if safety of this kind is possible.

I just can't wrap my head about what the actual product/service is. Let alone something that could be sold for billions.

"Safe AI" is very ambiguous in terms of product.