Everytime I read something like this , I get nervous about the cloud providers and Google. Since this is a relatively high profile customer standards, shouldn't they explain what caused them to suspend the account ?
Everytime I read something like this , I get nervous about the cloud providers and Google. Since this is a relatively high profile customer standards, shouldn't they explain what caused them to suspend the account ?
And vice versa. Company accounts have been deleted because an employee's personal account tripped Google's random number generator. Again, only Google does this.
it's not like they're the only provider, or even the #1
Their security track record is pretty exceptional compared to the other two.
If Railway isn't satisfied with the explanation, they're able to say so publicly, yes?
They're a web host; it could be any number of plausible mundane things that triggered automated action. This is a big recurring problem for any shared hosting provider.
an entire gcp project deleted along with its persistent disks.
how does that make any sense? nobody thought to call them or anything
No matter how damaging the behavior?
> an entire gcp project deleted along with its persistent disks.
Railway doesn't say that - "persistent disks inaccessible", followed by "persistent disks restored to ready state". It was a suspension, not a wipe.
I'd also expect a story around how it is this happened w/o a human spending at least an hour working his/her way through a call list to reach someone at Railway. Starting with ops and escalating to the ceo if necessary.
If you are saying they should be required to by law, then no I disagree.
But, given that this incident unjustly caused real damages to another company, I am pretty certain that Google will be required to make some sort of response to this, and if it ends up in the courts, it will be public.
That statement is the last 15 or so years.
Maybe AWS is the only player in town now? I don't know. Google doesn't instill confidence with these incidents, same with those cases of insurmountable bills caused by simple mistakes where there should be a way for smaller customers to cap usage.
These sorts of things have happened before with Google and the other expensive providers.
Are cheaper providers known for doing this? I would have thought they would be less lively to, as they are smaller and therefore every customer is relatively more important to them, and they are therefore more likely to check before turning services off.
In my experience, the reason they're cheaper is because they offer fewer features (cut down on ongoing expenses) and because they aggressively enforce TOS (the margins are thinner, so you're less able to afford people using more resources than allocated).
Thanks.
You're already placing blame here.
We don't know the details other than what Railway has said.
If Railway did something wrong, then letting that be known may help other customers avoid the same ~mistake.
You can't rely your business on GCP. Honestly, this is the most silly way to kill your own business.
For context, copied from my post 3 years ago.
March 10, 2023 | hide | past | favorite As a 4 years customer, our production severs have been suspended by Google Cloud because we didn't fill up some information on-time. Contacted support but they expect us to wait for 24-48 hours to get it resolved while all our servers are down. Anyone linked with someone powerful in google cloud can help?
======
- Running production on google cloud for 4 yrs with my startup. 100% legit SaaS business.
- Always pay bills on-time no issue. Good customer never open tickets, ask for help or what just quietly pay my bills each month.
- Our servers was abruptly suspended yesterday midnight and my whole business is now down for > 10hrs.
- We run a SaaS business that other ecommerce stores rely on and have hundreds of paying merchants.
- My customers have been grilling me and I don't feel gcloud's trust and safety team understand/care how urgent the issue is.
======
Why were our servers suspended? Because we didn't fill up information in time?
- See https://imgur.com/a/x0Y3RJl
- Apparently they dropped us an email 10 days back that I missed out
- Titled "Important Information Regarding Your Google Account" with no indication of suspension or what in title.
- Given the number of subprocessor "Important" emails they send it's too easy to miss out the email.
- 10 days gone by and our servers were abruptly shutdown with zero suspension notification or what.
- We've been paying $400-$700/mo for the past 4 yrs consistently and they shut us down because we didn't fill up some information?
When I tried to ask them to at least temporarily get our servers back while the verification is ongoing, I didn't get any answers.
Google Cloud have zero empathy for customers.
It's not like my account got suspended for fradulent issue or what. It's suspended because I didn't fill up some information on-time and they don't even allow me to temporarily reactivate my services or what. Especially when I had to wait for hours to get their team to verify my details before I can get my servers back.
You can't trust them with your business. Don't run any production stuffs with Google Cloud, ever.
Google/GCP can only make very general statements and in this case we want more than that.
They need to tell Railway and Railway needs to tell us, or Railway can tell us that Google is refusing to tell them.
Either way, we need to hear about this from Railway.
The bigger point though is Google really need to flag any business account as not subject to these suspensions until checked into by several humans. Back when I had a team that used a lot of App Engine they would even call us when we caused all their pagers to go off, and then conspire to keep the lights on while things got fixed. It's sad they have ended up like this.
That case is called:
> or Railway can tell us that Google is refusing to tell them.
I'm not paying you (which let's face it, that's what an NDA is) just to find out why you messed up about as severely as one can imagine. In theory, there was a contract here: $ for cloud services, and the rug got pulled. One should get a very clear, and very apologetic explanation as to why, with no strings attached, or one should be voting with their wallet.
Now, whether Railway will do any of that, who knows.
"We take full responsibility for the architectural decisions that allowed a single upstream provider action to cascade into a platform-wide outage, and detail below what happened, how we recovered, and the changes we are making to prevent this from happening again."
My guess is they will be switching away from Google Cloud. Because anything else would be nuts.
The calibration of the alarms might be off, and that's acceptable, but in the end if something can be held wrong, somebody will hold it wrong.
I did the same thing in the past, albeit in a much smaller scale. There's no shame in being wrong and admitting as long as it results in progress, so this stance of "we do nothing wrong" from both parties is getting a bit old now.
To me, what it sounds like is that a Google Cloud system identified Railway as a misbehaving customer. Spam, hackers, that sort of thing. Often this happens for "platform as a service" companies, because Railway themselves probably do host some spammers and hackers, and they have their own systems for dealing with it.
So, it's quite possible that according to the Google team, Railway violated the terms of something or other, and according to the Railway team, they did not, and now everyone has to argue about it.
But who knows, this is just me guessing based on some experience running a PaaS that itself was running on top of AWS.
I don't know what happened in this case, there's a chance it wasn't Google's fault but it doesn't matter, Google already lost all benefit of the doubt long ago.
Nobody doubts Google's engineering ability. It's their soft skills that are lacking. Google is culturally unable to understand the concept of customers.
I'm interpreting that bit from Railway's blog to mean it wasn't just them that was impacted.
Every regulated firm running on GCP is going to spend Monday explaining to their board how their resilience plan accounts for a hyperscaler that operates this opaquely. The compliance paperwork is the easy bit - the honest answer is we trusted that a hyperscaler would behave like a utility and they didn't.
So yes - they owe a statement. The whole point of paying hyperscaler prices is the assumption you won't wake up suspended with no explanation.
#1. you will most likely never be as big a customer as railway. if they did it them, you're fucked.
#2. if shit hits the fan, even a company as large as railway can get no human being on the phone. that's insane.
how is gcp still a thing after this event? if you're in charge of tech at your company, how do you stomach choosing this? how do you defend your choice to your CEO?
This is a small unknown startup.
Incident Report: May 19, 2026 – GCP Account Suspension - https://news.ycombinator.com/item?id=48204770 - May 2026 (144 comments)
Previously:
Incident Report: Railway Blocked by Google Cloud [resolved] - https://news.ycombinator.com/item?id=48201484 - May 2026 (344 comments)
This would have been an amazing conversation to witness.
farwaabbas•2h ago
esseph•1h ago
Are they really "high profile"?
I've never heard of them until this incident or maybe the other one with the database.
They just seem... Loud, publicly. And always about failure.
OJFord•39m ago
https://hn.algolia.com/?dateEnd=1779148800&dateRange=custom&...