Everything would be working, sometimes for hours, but once in a while the MCU would HardFault. The exact instruction where it would fault was pretty consistent but not perfectly. We started adding more and more instrumentation. We had logic analyzers and at least two oscilloscopes hooked up to the system at one point. Sometimes it would crash a bunch in rapid succession. When it did, the team that was primarily responsible for it would call me over hoping that it would happen again.
Eventually, late at night, we were sitting around scratching our heads trying to figure it out. I was frustrated. “OK I’m going to go have a smoke. Back in a few.” I stand up and the scope connected to the power rail, right then, shows a spike up to about 10V (on a 3V3 rail). I sit back down and try to figure out what caused it. Can’t figure out what could have possibly triggered it. “Fuck, ok, I’m going to go try to have that smoke again…” stand up. Blip. Crash. Sure enough, it was that specific wheeled chair that caused it. The two other chairs there didn’t, only that one. Ultimately it was a ground loop between two of the boards, but we would have never found it without that chair giving us a way to reproduce the problem.
jsiepkes•2mo ago
pavel_lishin•2mo ago
yetihehe•2mo ago