a pet peeve of mine, (along with people brigading on issues/threads e.g. posting them to unrelated news sites... op....) is woefully incorrect language.
> at day 66 all our jobs started randomly failing
if there's a definable pattern, you can call it unpredictabily, but you can't call it randomly.
paulddraper•13m ago
Unexpectedly is probably what they meant
wincy•26m ago
Crazy, so if I understand correctly, something with B200s and nvlink is causing issues where after 66 days and 12 hours of uptime, nvidia-smi and other jobs start failing, timing out, then once you restart the cluster it starts working again.
They suspect jobs will work if you only use 1 B200, but one person power cycled so wasn’t able to test it. Hopefully they won’t have to wait another 66 days for further troubleshooting.
layla5alive•18m ago
Some 32-bit counter somewhere used when in NVLINK overflows?
mook•4m ago
Isn't 32bit counter 49 days? Assuming that one was counting milliseconds, at least.
Only remember that because that's the limit for Windows 95…
blackoil•10m ago
*China specific code leaked into mainline.
zeehio•2m ago
66 days 14 hours and 24 minutes (66.6 days) would have been a far more diabolical hang...
nulone•2m ago
NVLink postRxDetLinkMask errors show up right before the hang. Has anyone captured a bug report or stack trace while nvidia-smi is stuck to see what it's blocking on?
grayhatter•27m ago
> at day 66 all our jobs started randomly failing
if there's a definable pattern, you can call it unpredictabily, but you can't call it randomly.
paulddraper•13m ago