Short version: "It turned out our chosen base address for the FMC (FPGA Mezzanine Card) bus had a default memory type of Normal Cached."

They accidentally put an external non-memory device behind the cache. That's never going to work right, but might work some of the time.

For the uninitiated, Service Processor (SP) is just another kind of Baseboard Management Controller.