GitHub Incident with Git Operations, Pull Requests and Actions

https://www.githubstatus.com/incidents/djlmxz2zd0j7

106•gagan2020•46m ago

Comments

geooff_•45m ago

Actions being down is one thing - I can't even land a PR.

vitosartori•44m ago

PR? I can't even push haha

laurent123456•44m ago

I can't even create issues (error 500)

gagan2020•44m ago

Same here. I wanted to push something on Github and its was giving 500. This is most severe issue in recent times.

balanceiskey15•43m ago

GitHub added new webhook IP ranges this morning, curious if that's related.

gagan2020•41m ago

That might be the cause but let's see.

There 4 updates contains webhook mention in 2 of the updates.

"Webhooks is experiencing degraded performance"

iercan•41m ago

first time seeing "git push" on github throwing an "Internal Server Error"

cowlevel•40m ago

gitea, forgejo, gitlab CE, and ssh still working fine

hermannj314•16m ago

Exactly. I'm running forgejo in a container, pushing backups to a private S3 storage, and sent the forgejo team a small liberapay payment.

Maybe 30 minutes to get it all setup and then I moved on with my life.

user43928•14m ago

I never noticed downtime with public GitLab. It's also generally better than GitHub and Actions.

A self-hosted enterprise GitLab instance in Azure, on the other hand, has had outages and performance degradation quite a few times.

cowlevel•5m ago

Of course. Azure is the reason GitHub is down so often, too.

alexellisuk•39m ago

I thought it was odd that we'd not had one of these for a few weeks.

A few hours ago GH pages just would not publish at all.. and the status page showed as fine, I guess it's now cascaded into a proper event.

cowlevel•6m ago

There was one yesterday.

malfist•3m ago

Not had one for a few weeks? There was an outage on Monday (as usual)

caaqil•39m ago

Who even runs GitHub at this point?

dam_jackalopes•36m ago

I just work here man.

Kudos•36m ago

Copilot.

herpdyderp•32m ago

According to themselves, 225 million people.

https://github.com/about

esafak•32m ago

Who's the commanding officer here, soldier?

freakynit•38m ago

Spent 5 minutes debugging why my builds were failing... then recalled github is not google when it comes to availability... checked status.github.com ... and voila.. there it is.

embedding-shape•19m ago

I'm surprised people still rely so much on GitHub. Anything I want to (have to*) be able to deploy on a moments notice, for hotfixes and what not, was moved away from Github like a year ago, because of the horrible uptime. Did others not get the memo yet?

It's not getting better, if anything it's getting worse. Best time to get off GitHub was yesterday, the second-best day to get off is today. Tangled, Codeberg or self-hosted Forgejo (my approach) or Gitea are all good alternative solutions here.

pixelesque•37m ago

One of the PRs I eventually managed to open 8 mins ago had a page flash up a generic error page containing a badge error of: "Unexpected token '<', "<!DOCTYPE "... is not valid JSON"

aarestad•35m ago

99% of the time, this is a server returning a generic 500 HTML response to an HTTP call that was expecting a JSON response.

aleksandrm•34m ago

"Rewrite GitHub in Rust. Make no mistakes."

jeremyjh•32m ago

codefloe.com is still up

freakynit•30m ago

So, "git push" is down ... how are github engineers gonna push the fix now?

whalesalad•28m ago

sudo git push

pkthunder•27m ago

Make no mistakes.

tripleee•20m ago

in vim over ssh of course.. agents can use vim right?

dodslaser•9m ago

Yeah, but they keep trying to use Emacs commands...

C-x M-c M-unfuck-git

bearjaws•20m ago

rsync and sftp, like god intended

redanddead•12m ago

Like the olden days

ChuckMcM•

oslem•29m ago

Color me shocked

nstott•17m ago

This is so frustrating. I wish it were easier to migrate away from this tool

Brian_K_White•28m ago

holy crap push is a new low

acclay•27m ago

I'm tired, boss.

rapind•25m ago

Based on the comments in this thread, you'd think that Reddit was down.

rvz•25m ago

The last time GitHub went down was...yesterday.

GitHub is never going to improve and at this point. You might as well say that the AI chatbots, Tay and Copilot are maintaining the platform and are running it into the ground. No CEO of GitHub to go to as well.

It is time to self host instead of using something as broken as GitHub as I predicted 6 years ago.

time0ut•25m ago

Well it is a Wednesday.

I noticed because SSO was down. They don't even report on that as far as I can tell.

coliveira•24m ago

That's what you get when the bulk of open source is now depending on Microsoft.

knowknow•23m ago

Embarrassing that a multi-billion dollar corporation that other billion dollar corporations depend on seems to be fine with breaking their clients workflows. Was github always this shoddy or has something changed to cause this many outages?

TavsiE9s•22m ago

I mean... they got acquired by Microsoft.

cowlevel•8m ago

Almost 10 years ago

embedding-shape•17m ago

They used to run their own data centers, and while the unicorn (their error page back in the day) was visible sometimes, it wasn't nearly as bad as things are today.

And of course, Microsoft is saying that none of this downtime is at all related to them moving everything to Azure, and also at the same time they'll fix all this downtime by finishing moving everything to Azure.

I wouldn't hold my breath here.

mjr00•15m ago

> has something changed to cause this many outages?

AI coding. Which can be interpreted one of two ways:

* The generous way, which is that Github is so overloaded with massive volumes of AI codebases and AI-driven automation that they're hitting a scale they never anticipated; or

* The not-so-generous way, which is that Github itself was one of the first companies to push everyone to AI code as much as possible, which has lead to an eventual breakdown of the stability of the system, as the people responsible for it no longer understand how it actually works, leading to production outages once every few days.

MadrasTh0rn•20m ago

Love it

gwbas1c•20m ago

What's frustrating is that:

1: The status site says that pull requests are working,

2: A PR I'm working on right now is missing commits,

3: The contact support page errors out.

bossyTeacher•19m ago

This is becoming way more frequent than a monthly event. Does anyone know why GitHub keeps having incidents?

christophilus•17m ago

It’s mentioned on most threads, but their traffic and usage is way up due to AI / agentic programming.

cowlevel•6m ago

Because it's vibecoded and because it's running on Azure

tripleee•17m ago

Three days in a row now right?

HenkieTenkie•17m ago

Another case of tactical downtime.

sleepybrett•16m ago

i see they started letting the github engineers ship code again after that shitshow last month.

user-•11m ago

I can attest to the outage starting around 11:05, as that’s when my GitHub auth for Tailscale SSH broke. Of course, the next 10 minutes of troubleshooting and checking GitHub’s status page showed “all systems operational,” so I assumed it was something on my end. Then, it started working again.

And while I was typing this comment it just blipped again. Atleast this time I know why.

I feel dumb for setting up tailscale like 7 years ago using the github auth, time to figure out how to just move to email on that account.

honeycrispy•6m ago

Funny, yesterday git operations had an unreported outage too. I couldn't pull changes for ~10 minutes.

odo1242•40s ago

To be fair to GitHub, outages are pretty short (~40 minutes), and they rarely impact critical parts of the service, nor do they affect one’s ability to work (git offline)

Ultra Low Latency Solution with U4 enables high-velocity trading

A sense of control over workplace technology linked to wellbeing, productivity

Show HN: CrowDB – an iceberg table storage single Docker container

A Cop's Case for Flock

The Dawn of the Age of the Exoskeleton

Christa Pike's lawyers hold press conference after she regains consciousness

FortiBleed still a bleeding nuisance as FBI confirms ongoing attacks

Trump Mobile customers' data dumped – and some never even received their device

Trade group crunches numbers on Trump's impossible push for 100% US-made tech

Biohub leads $1.8B initiative to build AI-ready biological datasets

Tecotype: macOS mail app for professionals who live in their mailbox

Show HN: Configstream – Live config for Spring Boot via MongoDB change streams

An OpenAI Insider Explains Why He Quit [video]

IBM's Selective Sequence Electronic Calculator (2003)

Ukrainian drones swarmed Moscow last night

DatCal

LC-3 Assembly Tutor-AI tutor that explains what the machine is doing

Open source 160 sound visualization experiments

My static site was serving my internal docs

Show HN: Trigora – durable execution without history replay

Show HN: Bigwords.page – Turn any screen into a sign. The URL is the app

Show HN: Luke – Voice planning for coding agents

AsyncAPI Initiative

Nokia-style Android launcher – Simple Dumbphone

Show HN: Srl – Angular-style web components with no build step

Redundancy Is the Engine of Progress

Boredpolymath/Truth-Beacon

Morning Coffee Simulator

Thank You Indonesia for the Clean Air

Show HN: The Last SaaS – An open source headless SaaS for agents and assistants