Best of luck to the team at Matrix/Element for restoring from a nasty outage.

> We are in the process of restoring the matrix.org database from a backup. The matrix.org homeserver will be offline until this has been completed

whoops

Hope this gets done sooner, horrible outage

> So: the matrix.org database secondary lost its FS due to a RAID failure earlier today (11:17 UTC). Then, we lost the primary at 17:26. We're trying to restore the primary DB FS (which could be fastish), while also doing a point-in-time backup restore from last night (which takes >10h). We believe the incremental DB traffic since last night is intact however. Apologies for the downtime; folks on their own homeserver are of course not impacted.

The stuff of absolute nightmares...

https://mastodon.matrix.org/@matrix/115136245785561439

Love y'all and love matrix. Thanks for the free matrix.org server. But maybe now is the time to research setting up a home server for mission critical stuff?

I don't immediately see an official doc on this; is it right under my nose?

Is this doc good? https://www.redpill-linpro.com/techblog/2025/04/08/matrix-ba...

Looks like it's going to take a while to come back up.

> Sorry, but it's bad news: we haven't been able to restore the DB primary filesystem to a state we're confident in running as a primary (especially given our experiences with slow-burning postgres db corruption). So we're having to do a full 55TB DB snapshot restore from last night, which will take >10h to recover the data, and then >4h to actually restore, and then >3h to catch up on missing traffic.

https://mastodon.matrix.org/@matrix/115136866878237078