Days since last GitHub incident

https://github-incidents.pages.dev/

211•AquiGorka•1d ago

Comments

_def•1d ago

woah this time i even caught it before the status page reported something - i thought they were rate-limiting me.

laurmaedje•1d ago

It seems to have started slowly. For me, Github releases have failed to serve requests for hours already.

rienbdj•1d ago

If GitHub actions break I now assume it’s them and not me. GitHub needs to work on stability ahead of AI features.

udev4096•1d ago

[flagged]

GaryBluto•1d ago

Friendly reminder to stop saying friendly reminder when what you're saying isn't a reminder.

echelon•1d ago

For any ESL folks here -

"Friendly reminder" is typically used for reminding people of common knowledge. Especially for beneficial but inconvenient things that some or most people neglect to do, either because they're annoying, inconvenient, or time consuming. Things for which busy people might need a "wink wink, nudge nudge".

Friendly reminder to floss. Friendly reminder to have your cancer screening. Friendly reminder to check your tires. Friendly reminder to file your taxes early. Friendly reminder to drink more water, eat fiber, etc.

loloquwowndueo•1d ago

You’re assuming that no native speaker is unfamiliar with the idiom and also that no ESL speaker is familiar with it.

udev4096•1d ago

[flagged]

tomhow•1h ago

We’ve banned this account for repeatedly abusing other community members and refusing our requests to stop. We've given several warnings and been more than patient but if we are to have any standards at all, eventually enough is just enough.

udev4096•1d ago

[flagged]

DetroitThrow•19h ago

This quickly became much less friendly

GaryBluto•17h ago

> Keep getting fucked

By what?

> The future is decentralized and p2p

I wish it was but that isn't how things are going to turn out, especially if it's only people like you pushing it.

ZeroConcerns•1d ago

This is a bit... low-effort, isn't it? I'd at least expect a video of an exasperated Github user walking up to the '# days since the last GitHub incident' board, sliding out the '1' or '2' card, and replacing it with a '0'.

I mean, that joke is as old as the universe (heck, in the brief period that I worked in an office, decades ago, I had a "# days since the last person asked a stupid question" sign to enact the exact same gag)...

rob74•1d ago

Or an octocat standing in front of the board, holding cards from 0 to 7 in its tentacles (with the rest lying on the ground) and looking at them quizzically?

kermatt•1d ago

An octocat with zero tentacles?

queuebert•1d ago

Base 8 numbering system?

old_bayes•1d ago

How can we add AI to this perfectly functional product?

venturecruelty•1d ago

Tie it to quarterly performance results!

llbbdd•1d ago

I don't use Github Pages so I might be wrong but IMO I think at least part of the joke is that its URL betrays that it's a completely static site.

Tade0•1d ago

I was half-expecting the "days since last" meme - the one with a person smiling awkwardly while clapping with a large four-digit counter in the background showing only zeroes.

venturecruelty•1d ago

I used to have a magic 8-ball that people could use when they wanted me to debug their code for them. I think it was broken, though; it kept saying "Outlook good". Must've been a Microsoft magic 8-ball.

eleumik•1d ago

you found the bad-magic-ball, looks like a standard magic-ball with genius inside but is not, the one that finds it works, not the genius..hmmm...but hey glass balls with LLM inside...christmas...wonderful idea

asplake•1d ago

Very low effort. Couldn’t read the text on my iPhone without zooming in. I nearly mistook it for a blank page!

geophph•1d ago

Is GitHub deployed using GitHub Actions?

Oakwhisper•1d ago

I was talking with some GH sales/marketing engineers last month and they said it deploys with actions, but they have a custom deploy queue

0cf8612b2e1e•1d ago

It is a fun bootstrapping problem. How do you firewall enough dedicated resources to stand up your infrastructure if you dogfood your own product. Probably insidiously easy to have a dependency on the production service.

shakna•1d ago

An Azure outage took out Office365 the day before CrowdStrike happened. I would not trust Microsoft to get this balance right.

Lapalux•1d ago

Oh nice - I'm literally playing around with a site to detect outages for major provides (AWS/cloudflare/github) based on social media/HN posts

Lapalux•1d ago

hehe - thank you for helping me Github - https://imgur.com/a/0KqmKpU

llbbdd•1d ago

I've gotten accustomed lately to spending a lot of time in the Github Copilot / agent management page. In particular I've been having a lot of fun using agents to browse some of my decade-old throwaway projects; telling it to "setup playwright, write some tests, record screenshots/videos and commit them to the repo" works every time and it's a great way to browse memory lane without spending my own time getting some of these projects building and running again.

However this means I'm now using the Github website and services 1000x more than I was previously, and they're trending towards having coin-flip uptime stats.

If Github sold a $5000 box I could plug into a corner in my house and just use that entire experience locally I'd seriously consider it. I'm guessing maybe I could get partway there by spending twice that on a Mac Pro but I have no idea what the software stack would look like today.

Is there a fully local equivalent out-of-the-box experience that anyone can vouch for? I've used local agents primarily through VSCode, but AFAIK that's limited to running a single active agent over your repo, and obviously limited by the constraints of running on a single M1 laptop I currently use. I know at least some people are managing local fleets of agents in some manner, but I really like how immensely easy Github has made it.

bastardoperator•1d ago

They do, it's called GHES.

https://docs.github.com/en/enterprise-server@3.19/admin/over...

"GitHub Enterprise Server is a self-hosted version of the GitHub platform"

ModernMech•1d ago

I've tried getting this set up at my University, it was hell dealing with them. We ended up going with Gitlab.

verst•1d ago

That does not include the Copilot related APIs though.

AceJohnny2•1d ago

you're not getting copilot on the self-hosted version, which is what the parent was focusing on.

bastardoperator•15h ago

Incorrect, you can use GitHub connect to sync licenses, this allows you to license users under GHEC and GHES at the cost of a single seat. You will need an entitlement for Copilot, but the fact is you can absolutely get access while storing none of your code on .com.

Aurornis•1d ago

None of the open weights models you can run locally will perform at the same level as the hosted frontier models. Some of them are becoming better, but the step-down in output quality is very noticeable for me.

> If Github sold a $5000 box I could plug into a corner in my house and just use that entire experience locally I'd seriously consider it. I'm guessing maybe I could get partway there by spending twice that on a Mac Pro but I have no idea what the software stack would look like today.

Right now, the only reasons to host LLMs locally are if you want to do it as a hobby or you are sensitive about data leaving your local network. If you only want a substitute for Copilot when GitHub is down, any of the hosted LLMs will work right away with no up front investment and lower overall cost. Most IDEs and text editors have built-in support for connecting to other hosted models or installing plugins for it.

> I know at least some people are managing local fleets of agents in some manner,

If your goal is to run fleets of agents in parallel, local LLM hosting is going to be a bottleneck. Familiarize yourself with some of the different tool options out their (Claude Code, Cline, even the new Mistral Vibe) and sign up for their cloud API. You can also check OpenRouter for some more options. The cloud hosted LLMs will absorb parallel requests without problem.

llbbdd•1d ago

Thank you, a bit sad to hear that local inference isn't really at this level of performance yet. I was previously using the VSCode agent chat and playing with both OpenAI and Github hosted models but I switched to using the Github web UI directly a lot since my workflow became a lot more issue/PR-focused. Sounds like I should probably tighten up the more generic IDE-centric workflow and make it a keyboard shortcut to switch around when a given provider is down. I haven't actually used Claude directly yet but I think Github agents often use it under the hood anyway.

colechristensen•1d ago

An NVIDIA DGX Spark is $4000, pair that with a relatively cheap second box to run GitLab in the corner and you would have pretty good local AI inference setup. (you'd probably have to write a nontrivial amount of software to get your setup where you want)

The local models are just right on the edge of being really useful, there's a tipping point to where accuracy is high enough so that getting things done is easy vs models getting continuously stuck. We're in the neighborhood.

Alternatively, just have local GitLab and use one of the many APIs, those are much more stable than github. Honestly just get yourself a Claude subscription.

llbbdd•1d ago

I can't say I'm not tempted looking at the Spark, I could probably save some cash on heating my house with that thing. Though yeah unless there's some good software already built around a similar LLM workflow I could use it'd probably be wasted on me, or spend its time desperately trying to pay for itself with crypto mining.

Adding Claude to my rotation is starting to look like the option with the least amount of building the universe from scratch. I have to imagine it can be used in a similar or identical workflow to the Copilot one where it can create PRs and make adjustments in response to feedback etc.

colechristensen•1d ago

>Though yeah unless there's some good software already built around a similar LLM workflow I could use it'd probably be wasted on me, or spend its time desperately trying to pay for itself with crypto mining.

A big part of my success using LLMs to build software is building the tools to use LLMs and the LLMs making that tool building easy (and possible).

llbbdd•1d ago

I tried this for a little while and couldn't really get passionate about it; I have too many other backlogged projects that I was eager to tear into with LLMs and I got impatient. That was a while ago though and the ROI for building my own tools has probably gotten a lot more attractive.

colechristensen•1d ago

I started building my own tool set because I was doing too many projects with LLMs and getting frustrated by a very real need for organization and tooling to get repetitive meaningless tasks out of the way and to get all of my projects organized so I could see what was going on.

llbbdd•1d ago

I'm convinced. :) I've got some time to kill in transit later today, maybe time to think about my setup a bit.

smcleod•1d ago

The DGX Spark is not good for inference though it's very bandwidth limited - around the same as a lower end MacBook Pro. You're much better off with a Apple silicon for performance and memory size at the moment but I'd recommend holding off until the M5 Max comes out early in the early as the M5 has vastly superior performance to any other Apple silicon chip thanks to its matmul instruction set.

llbbdd•1d ago

Oof, I was already considering an upgrade from the M1 but was hoping I couldn't be convinced to go for the top of the line. Is the performance jump from the M# -> M# Max chips that substantial?

baby_souffle•1d ago

> Is the performance jump from the M# -> M# Max chips that substantial

From m1? Yes, absolutely. M3 is marginal now but m5 will probably make it definite.

smcleod•1d ago

The main jump is from anything to M5; not because it's simply the latest but because it has matmul instructions similar to a CUDA GPU which fixes the slow prompt processing on all previous generation Apple Silicon chips.

doublerabbit•1d ago

I've not been able to browse any repo sources without the Unicorn for the past few hours.

tonymet•1d ago

This is pretty dishonest because some trivial service no one cares about will reset this counter .

guywithabike•1d ago

Your definition of "trivial" is not everyone's definition of trivial.

bdcravens•1d ago

True, but the point remains that defining the whole as "down" when a subset is dilutes the value.

tonymet•1d ago

github haters (who still use the platform, for free) are the worst

toastal•1d ago

This is why I keep encouraging folks to a) have a mirror & b) make sure their tools automatically pick up the mirrors.

I recently got mirror support upstreamed into Nixpkgs for fetchdarcs & fetchpijul which actually work on my just-alpha-released pinning tool, Nixtamal <https://darcs.toastal.in.th/nixtamal/trunk/README.rst>, for just this sort of thing.

barbazoo•1d ago

I envy you. Most of us struggle to get the resources to make our actual customer facing applications resilient, let alone our build pipeline.

toastal•1d ago

Building your software usually involves getting dependencies, & those dependencies are, hopefully, in more than one location—which includes a cronjob to a bare repo, or Alice’s fork on another repo that at least has the latest tags. It should be trivial to point to these as mirrors for the cases where any forge/repository, even the ones held by megacorporations, inevitably go down. Even Nixpkgs itself, while not maintaining their own official mirrors, are mirrored by TUNA. Backups are an important strategy, & the source code should also be a part of that.

maccard•1d ago

That's great for the repository, but what about if you're using ghcr, actions, issues, or copilot?

toastal•1d ago

These are different concerns. There are a lot of use cases, where folks are just getting dependencies & not interacting with bug tracker or continuous integration use which are less critical & can be accessed later or ran locally.

ferguess_k•1d ago

I guess none of us really needs those 9s, and even two 9s are just good enough. I even doubt whether *SOME* of the banking transactions really really really need those 9s too -- like, I don't really mind if 1 out of 100 credit payment doesn't go through so I have to do it again -- it does happen once for a while and I just swiped it again.

0xdeafbeef•1d ago

Retry is fine, but imagine being unable to pay for something within 10 minutes in month. And 10m in 1M is 99.98% sla. So it depends

dec0dedab0de•1d ago

I had an ATM glitch out on me a few months ago, I tried again and it confiscated my card. I called, and they explained that it is the failure mode to prevent people modifying them while they're offline.

tormeh•1d ago

GitHub has a container registry. That going down can cause pod start failure. I agree the source code probably doesn't need infinite nines, but the container registry is different.

mystifyingpoi•1d ago

Which should not even be that hard, because read-only replicas of artifact repos are trivial to create and easy to loadbalance.

cedws•1d ago

I found out when Actions started failing again for the Nth time this month.

The internal conversation about moving away from Actions or possibly GitHub has been triggered. I didn't like Zig's post about leaving GitHub because it felt immature, but they weren't wrong. It's decaying.

DetroitThrow•1d ago

Combined with security concerns, this made us reconsider even our self-hosted GH Actions last month.

GH Packages is something we're extricating ourselves from after today too. One more outage in the next year and maybe we get the ammunition to move away from GH entirely.

It's still hard to believe that they couldn't even keep the lights on on this thing.

IgorPartola•1d ago

The Zig post has since been updated and the objectionable parts have been removed. I think we can put that part to rest.

landr0id•1d ago

I have no problem with their opinions but I don’t think it should have been said in a Foundation post.

It may have been updated, but nobody is reading the update.

000ooo000•1d ago

OK, so we keep banging on about it forever? Move on

esseph•1d ago

I just did and wouldn't have know about it if it hadn't been talked about here.

YetAnotherNick•1d ago

Idk, if being bad is the reason for leaving Github Actions, I think people would have left it ages ago. It stuck not because it is better than competitors but because it is included in the Github plans. It's decaying implies that it has somehow became worse, in fact it was one of the worst implementation to start with.

hinkley•1d ago

If you consider that an American maintainer was cheesed off enough to move an entire project off GitHub two days before Thanksgiving then the tone of the original post was completely in line with the energy involved.

Anger is a communication tool. It should absolutely be used when boundaries are being violated. Otherwise you’ll get walked all over.

bilkow•1d ago

I mostly agree, but a generalized attack at the remaining GitHub workers by calling them "losers" and then "rookies" is unwarranted and leaves a bad taste IMO.

See the edit history here: https://news.ycombinator.com/item?id=46133179

Edit: 1. just to be clear, it's very good that they have accepted the feedback and removed that part, but there's no apology (as far as I know) and it still makes you wonder about the culture. On the other side, people make mistakes under stress. 2. /s/not warranted/unwarranted/

Nextgrid•1d ago

A structural engineer will not sign off on bad designs no matter how much pressure the company applies to them. They will resign and/or report the incident to their local regulator as a safety issue.

We don't have that for developers. Maybe shame/offense is our next best bet. You are free to work for a terrible company accepting and/or encouraging terrible design decisions, but you need to take into account the potential of being laughed at for said decisions.

stefan_•1d ago

GitHub has seem to come under the same management as VSCode, everything has to be made AI and that is the only priority. It's like the Google+ of old but stupider.

chrisandchris•1d ago

Hopefully with that much AI they can finally make the Explore page more useful than "most stars" and "most recent updated". There seems to be no way to discover stuff on GitHub except knowing where it is (hence not discovering but knowing).

samcheng•1d ago

Missed a chance to put this in meme format e.g. https://imgflip.com/memetemplate/439302803/Days-without-acci...

lawlessone•1d ago

The amazing part about this is the page even works when i'm offline.

queuebert•1d ago

At this point, is there any downside to switching to GitLab?

loloquwowndueo•1d ago

What’s gitlab?

(Snarky way of saying: GitHub still has huge mindshare and networking effects, dealing with another forge is probably too much friction for a lot of projects)

Not that GitHub doesn’t suck…

burningChrome•1d ago

When GitHub was bought by Microsoft, Gitlab made moving your repos to them super easy. Apparently not enough people have moved and it would seem even with sustained attacks from all kinds of different vectors, it would seem people continue to stick with them.

I use both Gitlab and Github and have yet to experience any downtime on any of my stuff. I do however, work at a large corporation and the latest NPM bug that hit Github caused enough of a stir where it basically shut down development in all of our lower environments for about two weeks so there's that.

But I do agree, and it seems like their market share increased after the Microsoft acquisition which is contrary to what I heard in all my dev circles because of how uncool MSFT is to many of my friends.

richardwhiuk•1d ago

Is the uptime any better?

burningChrome•1d ago

Not really:

GitHub - Historically, GitHub reports uptime around 99.95% or higher, which translates to roughly 20–25 minutes of downtime per month. They have a large infrastructure and redundancy, so outages are rare but can happen during major incidents.

GitLab - GitLab also targets 99.95% uptime for its SaaS offering (GitLab.com). However, GitLab has had slightly more frequent service disruptions compared to GitHub in the past, especially during scaling events or major upgrades. For self-hosted GitLab instances, uptime depends heavily on your own infrastructure.

bdcravens•1d ago

If escaping downtime is your goal, then you should aim for a service with less downtime than Github. (they're roughly the same, with Gitlab having a slightly higher percentage of "major" outages)

JackSlateur•1d ago

Is it any better ?

We had that last year, with the full premium stuff ("pay as much as we can" mindset)

Please see this: a basic feature, much needed by lots of people (those who are stuck on azure ..): https://gitlab.com/gitlab-org/gitlab/-/issues/360592

Please read the entire thread with a particular attention to the timeline

matheusmoreira•1d ago

I've been getting some weird cryptocurrency spam notifications on GitHub and they can't be cleared for some reason. Blue dot is gonna be there forever apparently. Some users made an issue out of it but nobody cared to fix it.

fastball•1d ago

Same happened to me. You can clear it via the CLI, hilariously.

```

gh api notifications\?all=true | jq -r 'map(select(.unread) | .id)[]' | xargs -L1 sh -c 'gh api -X PATCH notifications/threads/$0'

```

bdcravens•1d ago

HN doesn't support markdown, but you can "code" format it with 2+ spaces.

https://news.ycombinator.com/formatdoc

dennis-tra•1d ago

Had the same issue that the blue dot won’t disappear. I was able to clear the dot with:

gh api notifications -X PUT -F last_read_at=2025-10-06T00:00:00Z

Just change the date to today. I also got that line from a gh issue somewhere - maybe it was the same issue that you’re referring to.

cassidoo•1d ago

This should have been fixed here: https://github.blog/changelog/2025-12-04-notifications-trigg...

Are you still seeing it, would you mind checking? Our team will get on it if so.

shakna•1d ago

I had the same. The dot is cleared on the mobile site. It is present on the desktop site.

matheusmoreira•1d ago

Somewhat similar situation here. Cleared cache and logged back in on mobile, dot was fixed. Haven't tested on my laptop yet.

cassidoo•1d ago

Thank you all, sharing internally now to get that fixed! Super appreciate the feedback.

matheusmoreira•1d ago

Thanks for fixing it!

ashton314•1d ago

Here’s how you can clear it:

https://github.com/orgs/community/discussions/174310#discuss...

I had the same issue too, and this was the only thing that fixed it for me.

Aperocky•1d ago

Once GitHub fully migrates to Azure, it should be known as GitHub 11.

ragall•1d ago

Github 3.11 for Workgroups.

aranw•1d ago

GitHub Vista Cloud Edition

doubled112•1d ago

After a couple of service packs will it be OK?

Gazoche•1d ago

I think you mean "365 Code Copilot".

Aperocky•19h ago

AgenticHub!

TuxPowered•1d ago

It should always be at 0, because GitHub is unreachable over IPv6, which in 2025 should be considered an incident.

tonymet•1d ago

mobile adoption is high, desktop (residential and corporate) is still quite low.

I'm a big advocate for github to add ipv6 support , but let's not pretend it's critical for their business.

pas•1d ago

just a few hours ago we found a pretty nice residential desktop use case for proper v6 (with prefix delegation), due to no need for NAT the old router (2013) became less of a bottleneck!

tonymet•1d ago

Double check (I mean by remote port scan) that your firewall is working. I’ve seen routers with no IPv6 firewall . And it actually matters

kalleboo•1d ago

Aren't there serveral hosts now where IPv6 access is included but you have to pay for each attached IPv4? E.g. AWS and Hetzner

tonymet•1d ago

Yep

fragmede•1d ago

"yeah but when I turn on ipv6 everything breaks"

doubled112•1d ago

I haven’t had a residential ISP that provided IPv6 yet.

locusofself•1d ago

The Primagen video about the bash scripts underpinning github actions runner was crazy. I'm a half-assed programmer at best and I don't even think I would make some of those mistakes.

OptionOfT•1d ago

GitHub Actions is a good example of systems thrown together that at face value have something to offer until they get put under stress.

Just now I found:

    * a job that's > 1 month old, still running
    * another job that started 2 hours ago that had 0 output
    * a job that was marked as pending, yet I could rerun it
    * auto-merges that don't happen
    * pull requests show (1), click it, no pull requests visible

Makes me wonder in how many places state is stored, because there is some serious disconnect between them.

Nextgrid•1d ago

That's just post-Windows 8 Microsoft quality for you. Every product has been like that - looks "ok" on the outside (in reality it looks shit, but at least that's intentional), but the second you dig deeper and start using it you get all kinds of paper cuts like that.

Ozymandias-9•1d ago

When I opened the link, I just laughed for 5 mins straight

The secretive world of North Korean science fiction (2023)

Windows 3.1 in the Browser

Show HN: Hands on tutorial for open source contribution

New Solitaire Gaming Website

Show HN: League of Legends AI Assistant (OpenSource)

Gemini with Thinking 3 Pro can't script multi-line string replacement

Can We Really Claim That Civilization is on the Steady Path of Progress?

Commonplace Book

Sperm donor with cancer-causing gene fathers nearly 200 children

Surgical Masks and Viral Transmission

Ask HN: Is there a local dev tool you wish existed because of a repeating issue?

Revolutionary gene therapy brings hope of leukaemia cure [video]

Flow depression treatment now FDA approved

Oilwell is a wellness app to help you embrace climate chaos

Show HN: This week we shipped 'Surfaces' on rynk.io

Breaking Down Trump's 2025 National Security Strategy

Ask HN: is Archive.is a Kremlin Asset?

Kpython – A MicroPython Sidecar for the Linux Kernel (Experimental)

What is "involution", China's race-to-the-bottom competition trend

50 Years of Proof Assistants

Show HN: Clai – Unixlike CLI context feeder for LLMs. Now with recursive tooling

Technology Radar

Public Prompt License (PPL) – prompt-native licensing for LLM prompts

Hetz Demo – build HTML tables online, copy code instantly (no signup)

The military's new AI says boat strike 'unambiguously illegal'

Android PSA: have at least 2 ways of taking your files (especially pictures) out

Computer Animator and Amiga fanatic Dick Van Dyke turns 100

Where Disney Is Heading in Its Next Century [video]

What's the point of lightweight code with modern computers?

Show HN: Zunorm – BYOM Spreadsheet Editor

The secretive world of North Korean science fiction (2023)

Windows 3.1 in the Browser

Show HN: Hands on tutorial for open source contribution

New Solitaire Gaming Website

Show HN: League of Legends AI Assistant (OpenSource)

Gemini with Thinking 3 Pro can't script multi-line string replacement

Can We Really Claim That Civilization is on the Steady Path of Progress?

Commonplace Book

Sperm donor with cancer-causing gene fathers nearly 200 children

Surgical Masks and Viral Transmission

Ask HN: Is there a local dev tool you wish existed because of a repeating issue?

Revolutionary gene therapy brings hope of leukaemia cure [video]

Flow depression treatment now FDA approved

Oilwell is a wellness app to help you embrace climate chaos

Show HN: This week we shipped 'Surfaces' on rynk.io

Breaking Down Trump's 2025 National Security Strategy

Ask HN: is Archive.is a Kremlin Asset?

Kpython – A MicroPython Sidecar for the Linux Kernel (Experimental)

What is "involution", China's race-to-the-bottom competition trend

50 Years of Proof Assistants

Show HN: Clai – Unixlike CLI context feeder for LLMs. Now with recursive tooling

Technology Radar

Public Prompt License (PPL) – prompt-native licensing for LLM prompts

Hetz Demo – build HTML tables online, copy code instantly (no signup)

The military's new AI says boat strike 'unambiguously illegal'

Android PSA: have at least 2 ways of taking your files (especially pictures) out

Computer Animator and Amiga fanatic Dick Van Dyke turns 100

Where Disney Is Heading in Its Next Century [video]

What's the point of lightweight code with modern computers?

Show HN: Zunorm – BYOM Spreadsheet Editor

Days since last GitHub incident

Comments