The fun is really gonna start if the root cause of this somehow implicates an AI as a primary cause.

Well at least you don't have to figure out how to test your setup locally.

They might have an implicit dependency on AWS, even if they're not primarily hosted there.

Since I'm 5+ years out from my NDA around this stuff, I'll give some high level details here.

Snapchat heavily used Google AppEngine to scale. This was basically a magical Java runtime that would 'hot path split' the monolithic service into lambda-like worker pools. Pretty crazy, but it worked well.

Snapchat leaned very heavily on this though and basically let Google build the tech that allowed them to scale up instead of dealing with that problem internally. At one point, Snap was >70% of all GCP usage. And this was almost all concentrated on ONE Java service. Nuts stuff.

Anyway, eventually Google was no longer happy with supporting this and the corporate way of breaking up is "hey we're gonna charge you 10x what did last year for this, kay?" (I don't know if it was actually 10x. It was just a LOT more)

So began the migration towards Kubernetes and AWS EKS. Snap was one of the pilot customers for EKS before it was generally available, iirc. (I helped work on this migration in 2018/2019)

Now, 6+ years later, I don't think Snap heavily uses GCP for traffic unless they migrated back. And this outage basically confirms that :P

Yep, was just thinking the same when my Kubernetes failed a HelmRelease due to a pull error…

Just yesterday I saw another Hetzner thread where someone claimed AWS beats them in uptime and someone else blasted AWS for huge incidents. I bet his coffee tastes better this morning.

The Register calls it Microsoft 364, 363, ...

Reported uptimes are little more than fabricated bullshit.

They measure uptime using averages of "if any part of a chain is even marginally working".

People experience downtime however as "if any part of a chain is degraded".

I'd say that this is true for the average admin who considers PaaS, Kubernetes and microservices one giant joke. Vendor-neutral monolithic deployments keep on winning.

Feel bad for the Amazon SDR randomly pitching me AWS services today. Although apparently our former head of marketing got that pitch from four different LinkedIn accounts. Maybe there's a cloud service to rein them in that broke ;)

quay.io is down

It should be! When I was a complete newbie at AWS my first question was why do you have to pick a region, I thought the whole point was you didn't have to worry about that stuff

> another data centre

Yes, within the same region. Doing stuff cross-region takes a little bit more effort and cost, so many skip it.

Assuming that the service actually bothered to have multiple regions as fallbacks configured.

Happened to lots of commercial routers too (free wifi with sign-in pages in stores for example) and that's way outside us-east-1

WiFi login portal (Icomera) on the train I'm on doesn't work either.

exactly why it won't happen :)

That depends on a lot of factors, but for me personally, yes it is. Much worse.

Assuming we’re talking about hosting things for Internet users. My fiber internet connection has gone down multiple times, though relatively quickly restored. My power has gone out several times in the last year, with one storm having it out for nearly 24 hrs. I was sleep when it went out and I didn’t start the generator until it was out for 3-4 hours already, far longer than my UPSes could hold up. I’ve had to do maintenance and updates both physical and software.

All of those things contribute to a downtime significantly higher than I see with my stuff running on Linode, Fly.io or AWS.

I run Proxmox and K3s at home and it makes things far more reliable, but it’s also extra overhead for me to maintain.

Most or all of those things could be mitigated at home, but at what cost?

Unanswerable question. Better to perform a failure mode analysis. That rack in your basement would need redundant power (two power companies or one power company and a diesel generator which typically won't be legal to have at your home), then redundant internet service (actually redundant - not the cable company vs the phone company that underneath use the same backhaul fiber).

so, funny story, my fiber got cut (backhoe) and it took then 12 hours to restore it.

If you had /two/ houses, in separate towns, you'd have better luck. Or, if you had cell as a backup.

Or: if you don't care about it being down for 12 hours.

So for me, extremely anecdotally, I host a few fairly low-importance things on a home server (which is just an old desktop computer left sitting under a desk with Ubuntu slapped on it): A VPN (WireGuard), a few Discord bots, a Twitch bot + some auth stuff, and a few other services that I personally use.

These are the issues I've ran into that have caused downtime in the last few years:

- 1x power outage: if I had set up restart on power, probably would have been down for 30-60 minutes, ended up being a few hours (as I had to manually press the power button lol). Probably the longest non-self-inflicted issue.

- Twitch bot library issues: Just typical library bugs. Unrelated to self-hosting.

- IP changes: My IP actually barely ever changes, but I should set up DDNS. Fixable with self-hosting (but requires some amount of effort).

- Running out of disk space: Would be nice to be able to just increase it.

- Prooooooobably an internet outage or two, now that I think about it? Not enough that it's been a serious concern, though, as I can't think of a time that's actually happened. (Or I have a bad memory!)

I think that's actually about it. I rely fairly heavily on my VPN+personal cloud as all my notes, todos, etc are synced through it (Joplin + Nextcloud), so I do notice and pay a lot of attention to any downtime, but this is pretty much all that's ever happened. It's remarkable how stable software/hardware can be. I'm sure I'll eventually have some hardware failure (actually, I upgraded my CPU 1-2 years ago because it turns out the Ryzen 1700 I was using before has some kind of extremely-infrequent issue with Linux that was causing crashes a couple times a month), but it's really nice.

To be clear, though, for an actual business project, I don't think this would be a good idea, mainly due to concerns around residential vs commercial IPs, arbitrary IPs connecting to your local network, etc that I don't fully pay attention to.