The fundamental lesson of at least half my information systems undergraduate courses was you adapt the system to observed user behavior, do not expect the user to adapt their behavior to the system.

I have done it lots of times! With machines where you just dip the tip, you're bound to put the side with the chip in, but most machines want it facing up, and some want it the other way. The iconography is only illustrative once you've messed it up at those machines enough times (around me, Walgreens has difficult machines). Readers where you insert the whole card are easier to mess up, too.

> If the reader was able to read the card both way, there wouldn't have been a problem and no training required. The next best thing would be for the card to not fit upside down. Or have a clear message "try flipping the card". It is not something you should train people for, it should be obvious.

I suspect the HSM was an off the shelf component. The real issue with training is that a system with a complex startup procedure hadn't been restarted in 5 years. You should rehearse complex procedures at least once a year, otherwise there's a good chance nobody with experience has done it. Also, maybe someone would have flagged the issue of needing the cards to start the system than grants access to the cards. (Although drill + 1 hour is a reasonable recovery procedure that was obvious and didn't need training, apparently)

You're underestimating the amount of goodwill-run, unstaffed projects that any big corporation accrues over time, which accidentally become load bearing without anyone realizing until something goes wrong. Such unstaffed projects are usually very stable (from not having pressure to add features or earn profit) and therefore "just work" for years until something unusual, like an accidental DDoS, happens. In that time, the original author(s) and everyone with context have left the company. This is a very hard process/human problem to solve at FAANG scale.