Google really need to improve their support team. It's strange such a big corp can't even afford to have proper support team.
This seems to be by design.
Railway say they are in touch with that support team.
Obviously a fiasco but I’m not prepared to call them liars when it could be an honest mistake.
1. We depend on X but could gracefully migrate to an alternate in a week if we really needed to.
2. All data is mirrored instantly so that we can do seamless fail-over in case X has its own outage.
AWS may have data centers[0] go[1] down[2], but that's within expected bounds of standard ops.
[0] https://hooks.slack.com/services/TJ7HQS7FC/B0B5S7UTBJ4/PUHIC...
[1] https://www.aljazeera.com/news/2025/10/21/what-caused-amazon...
[2] https://netflixtechblog.com/lessons-netflix-learned-from-the...
Thank God I'm not dealing with any public-facing sites! Would have been an expensive lesson for a newbie coder if my job depended on this.
Or did they just mean that they’re not renting VPSs but only metal from the cloud provider?
In my mind I was so excited that there was another provider not just paying one of the hyperscalars but at a minimum colocating and owning more of their stack. https://blog.railway.com/p/heroku-walked-railway-run
I can provide an explanation about the GCP dependency. Yes, we have host workloads off GCP, and we have been able to build a good business by performing a cloud exit. However, we were worried that we would have a circular dependency on our own cloud. I don't think we expected to get auto-modded out of our own account, hence we left our DB on CloudSQL.
It was never our intent to deceive people that we didn't own our own destiny with our business. The last GCP issue, we were assured that this scenario wouldn't happen (when we got auto-ratelimited, which was bad, but survivable) - but it seems like we have further work to do. Apologies.
I mean, the pain we have caused our customer ultimately proves you correct. That said, we made our decisions with the information and constraints that we knew in that moment in time. Railway has hosts in AWS/GCP/and co-los, so coordinating those workloads in a fully distributed manner would be ideal but end of the day, we didn't forsee that would just have our project get deleted just like that.
(Even if we did get assurances from them in 2024, that it wouldn't happen again, although we just got auto-rate limited the last time.)
At $2m/mo spend, this kind of thing is insane. GCP has never been the most reliable of clouds but this is pretty awful. I would never have expected this.
The other notion that we have intuited is that you can’t build a cloud on another cloud. We have devoted years of practice running our own metal (and playing well with other clouds) to make sure that Railway’s business, which invariably becomes your customer’s business, is as rock solid as possible."
I’m aware of some companies hosting their own metal and infra, but I’m not aware of large companies mitigating risk by hosting on separate cloud providers as a fallback mechanism. We might disagree with cloud provider choice, or think they should have been hosting their own metal, but that’s still an “all your eggs in one basket” choice, right?
Heck, they might even have multi-region fallback with GCP, but if GCP bans your account, that doesn’t matter.
Are there good examples of running a company of railway’s size so redundantly that their host could nuke one of their accounts and they’d just keep on trucking?
iloveplants•2h ago