I realize NVIDIA is just cranking them out as fast as they can, but the quality on them is terrible. They overheat, disappear after you reboot, they fall off the bus, memory failures, and then mix in all the software crashes your users generate...
Our current server vendor is actually good at replacing them, unlike our previous vendor, but the failure rates are just insane. If any other component failed this much we'd have the vendor buy the servers back.
A deep dive on why these beastly cards fail so frequently compared to all other common current day hardware would be fascinating!
Component Type MTBF (yrs) AFR
─────────────────────────────────────────────────────────
SSD Hardware ~100 ~1%
RAM uncorrectable error Hardware ~75 ~1-4%
NVIDIA A100 critical error† Hardware 0.18 (65d) -
NVIDIA H100 critical error† Hardware 0.15 (50d) -
Only a minority of GPU 'failures' appear to be permanent hardware problems, such as row remapping errors. A lot seem to be, like another comment says, a consequence of operating too close to the operational limit, tipping over it, and then requiring a power cycle.
Does this mean even for a model that fits on a single server that trains for a few weeks will absolutely need a recovery process? Interested in peoples experiences around this.
> We’ve chosen not to refer to cloud providers directly, but instead give them anonymized A, B, C, D identifiers. If you want know who’s who, track the clues or buy us a beer sometime.
Come on, either name names or admit it is pure PR.
Edit: or will someone who can decode the clues weigh in?
bflesch•2h ago
Ed Zitron also called out the business model of GPU-as-a-service middleman companies like modal deeply unsustainable, and I also don't see how they can make a profit if they are only reselling public clouds. Assuming they are VC funded the VCs need returns for their funds.
Unlike fiber cable during the dot com boom the currently used GPUs eventually end up in the trash bin. These GPUs are treated like toilet paper, you use them and throw them away, nothing you will give to the next generation.
Who will be the one who marks down these "assets"? Who is providing money to buy the next batch of GPUs, now that billions are already spent?
Maybe we'll see a wave of retirements soon.
> It’s underappreciated how unreliable GPUs are. NVIDIA’s hardware is a marvel, the FLOPs are absurd. But the reliability is a drag. A memorable illustration of how AI/ML development is hampered by reliability comes from Meta’s paper detailing the training process for the LLaMA 3 models: “GPU issues are the largest category, accounting for 58.7% of all unexpected issues.” > Imagine the future we’ll enjoy when GPUs are as reliable as CPUs. The Llama3 team’s CPUs were the problem only 0.5% of the time. In my time at Modal we can’t remember finding a single degraded CPU core. > For our Enterprise customers we use a shared private Slack channel with tight SLAs. Slack is connected to Pylon, tracking issues from creation to resolution. Because Modal is built on top of the cloud giants and designed for dynamic compute autoscaling, we can replace bad GPUs pretty fast!
ares623•1h ago
touisteur•3m ago
Just wondering, if reliability increases if you slow down your use of GPUs a bit. Like pausing more often and stopping chasing every bubble and nvlink-all-reduce optimization.
pixl97•1h ago
I'm guessing this may be highly dependant on what the bathtub curve looks like, and how much the provider wants to spend on cooling.
Of course with Nvidia being a near monopoly here, they might just not give a fuck and will pump out cards/servers with shitty reliability rates simply because people keep buying them and they don't suffer any economic loss or have to sit in front of a judge.
Be interesting to see what the error rate per TFLOP (no /s, we're looking at operations not time) is compared to older generation cards.
topaz0•25m ago
Presumably this can't last that much longer, because the people that are buying/running these are already taking on loads of debt/venture capital to buy the past/current round of hardware without seeing much revenue from it. It's much harder to ask investors for multiples of your annual revenue just to maintain your current capabilities than it was a couple years ago to ask for many multiples of your revenue to expand your capabilities dramatically.
charles_irl•24m ago
You got a link for that? I work on Modal and would be interested in seeing the argument!
We think building a proper software layer for multitenant demand aggregation on top of the public clouds is sufficient value-add to be a sustainable business (cf DBRX and Snowflake).