The future of SRE will be the company putting some amount of money on a prediction market against the site going down and you get to take home the winnings as long as the site stays up.

Yeah but you thought about it, didn’t you?

Was about to send my bill to you.

... You're off the hook this time./s

Uh oh. That means there's at least one more like you out there that we don't know about.

Hah, I know the feeling. I installed Ubuntu on a PC recently, it obviously happened to be one of the days they got DDOSed and apt repos were unreachable. I had other things to take care of, so I put it aside for the next week or so. It didn't help very much, cause after picking it back up, halfway through, Snapcraft went down.

Next thing you're gonna tell us you're SRE at GitHub.

Same here. You’d think they could at least separate out the GitHub-hosted and self-hosted runners, so you’re still able to dispatch jobs if the self-hosted runners are down.

Depending on how many thousands of $ per year, it would probably be cheaper and more reliable to self-host GitLab. It's better in terms of organisational structure (you can have one, including access and secret inheritance), and (personal view) Gitlab-CI is better than GitHub Actions because it doesn't push you towards a JavaScript/NPM style dependency hell. And it's actually fairly easy to self-hosted, with options from a single machine with an omnibus package that handles everything to a full blown autoscaling Kubernetes deployment.

Same thoughts - we use an action to ship to production, its builds an image, pushes it to ECS which triggers a deployment.

We can't be blocked here. Seems silly what we settled on this, but for a long time GitHub had been reliable enough for many years, but things are sliding down the pan as of late.

It is a control pain

> For a service we're paying thousands of $ per year for even though we host runners ourselves...

Wait until you charge you for self-hosting runners.

Oh wait. They already tried.

[delayed]

Ugh, same. 30 mins with 2 devs trying to figure it out before they posted an update.

It's (a) they're under massively increased load because everyone's vibing up new projects these days, (b) they've been in a weird frankenstein "on azure but also we have our own control plane" state for years and they're pushing to no longer have that be the case.

I don't think vibecoding at Github has much to do with it.

GitHub had a blog post about this recently. They reported a significant uptick in volume (repos created, PRs, etc.), which they attribute to AI usage and tooling.

Microsoft has boasted 30% of their code written by AI.[1] However we could only guess if AI generated code is the issue or something else, or a combination of things.

That being said there was a noticeable trend starting around 2022.[2] That being said they’ve also been doing a big migration to Azure. It’s likely a combination of things.

1: https://www.cnbc.com/2025/04/29/satya-nadella-says-as-much-a...

2: https://www.reddit.com/r/sysadmin/s/LOMPaSv3wY

The instability started well before vibecoding, in around 2018-2019, shortly after the Microsoft acquisition.

https://damrnelson.github.io/github-historical-uptime/

https://news.ycombinator.com/item?id=47591928

I personally trigger github actions approximately 50x more than I did prior to AI-driven developer coding and I'm not alone.

A brownout redefined.

Same. It's weird how I always find out that GitHub is down before GitHub does. Took 15 minutes before it appeared on githubstatus.com

Github is more likely to be up before noon in UTC timezone. i.e. before the majority of US users are online and causing load.

Or maybe it's before the GitHub internal devs are online and deploying changes.

I have fun somebody imaging somebody internally explaining that this is a heavy traffic page and we should use it to increase reach.

Straight-up, "degraded" should strictly mean "may be slower, or so slow it randomly fails" on these kinds of status pages.

How would you call "available, but only sometimes"?

The whales are dying. But we don't know why. Well, some are still alive though...

github actions themselves can be self hosted, its quite nice actually to be able to keep your same patterns as cloud hosted actions and with one line change to the yaml have it running on your own hardware. I do this for actions that take 6-7 hours so I am not burning through the 3000 minutes that come free with my account.

Founder of Depot here. To my knowledge, we are the first engine to support different syntaxes in this compatible way via Depot CI [0]. Great time to try it out and let us know your thoughts! We’ve built a lot of cool stuff into it like parallel steps, custom images, and a full CLI/API interface so you can literally everything without going into the web app.

[0] https://depot.dev

there are a couple and have very good reputation - though I've never used them

https://www.blacksmith.sh/ and https://runs-on.com/

They also say that they're much cheaper than github

We currently use external runners (Blacksmith.sh), but that didn't shield us from this as GitHub actions is still the control plane for triggering and monitoring them.

We're now considering Buildkite (apparently they have a GH actions migration tool) or self hosting something (GitLab CI, maybe even Jenkins), as it looks like that would've kept ticking over since we're still seeing webhooks being triggered today during the downtime.

It's big enough that every time it goes down, it surely stops somebody from pushing fix for what they currently have broken, so I wonder if status page services see some kind of ripple from github outages.

I agree, but that's not at all related to this outage.

Hmm... 88.8888888%?

Jesus, that's both horrible and seems within reach.

"Its not like act, because we can add AI"

Is what it boils down to.

> codex "Fix this pipeline, use `act` to verify your changes"

What I don’t get about this is how you run OS specific tasks (Windows, macOS, Linux)..

I started playing with proxmox VMs and containers in them (docker and tart) to see if I can build some local infrastructure to properly solve this…

this has happened to me too. i am guessing then it is not a real reason?

We use TeamCity for CI builds, before that Jenkins. Only accessible from the inside of the network.

Even though it's selfhosted and we don't have a dedicated infrastructure team, I don't remember it ever being down in the last 12 years I have been working here.