How do people validate that a change to video data (encoding, preprocessing, annonimization etc.) won’t break models later?

We’ve seen cases where encoding was changed, psnr/ssim barely moved, video looked great - yet model behavior changed dramatically.

How do you validate video data changes before training? Or is the only real answer 'retrain and find out'?