I managed the system administrators for a high performance computing center. We took a lot of blame for the applications when in reality, often times it was poor programming on the developer's part. So, I got really tired of taking the blame and implemented statistical process control to track the mean time between failures of the jobs. I was really just shining the flashlight on production jobs and was hoping it could change the culture. It was not my job to fix their code, and the applications were developed by a different group of people with a very different culture. I thought the process control worked really well, and it did allow me to take the heat off me for random blaming of my team, when I could respond with "your job is failing XX times per year" and from there, push for a root cause analysis. But pushing against that culture was really hard, and there was a lot of "set the job to complete and I will look at it on Monday". If they do not want to conduct a root cause analysis on the failure modes for their code, I can't do much. So, even implementing some type of monitoring can have little effect if the ones who need to fix something do not support the culture. And, as I read your post, I'd think people would be looking at these business metrics a little closer or develop more sensitive metrics to catch these issues.

We had this very issue where a bad configuration change (human error) caused a large & sudden revenue drop and a drop in our streaming ad event metrics. This is a realtime adtech system where a delay in detecting sudden changes in business metrics can have monetary impact and visible customer experience drops. In this case, the major drop in revenue was immediately found and addressed but not all of our expected alerts went off. Our streaming ad events metric threshold was statically set too low. This threshold was appropriate at earlier stages of our business but as our business has grown, this threshold happens to be too low to set off the alert I would have expected to go off as the very first one. We do have sophisticated metrics instrumentation and alerting but an effective anomaly detection around sudden upticks/downticks in business metrics while being conscious of underlying metric trends evolving organically with the business would be a game changer.

Larger, incident-worthy changes in metrics are also easier to set static thresholds around and ring more than one bell when they occur. I'd be more concerned about smaller to mid deviations from the trend, say, sudden -/+10% change in my business metrics over X minutes. Can I reliably set a static threshold that will universally be appropriate here? A good anomaly detector would ideally bring something like this to attention without hard coded alert configs here