You know how you measure eternity?
When you finish learning German.
Perfect.
The hilarious part of the whole story is that the same PMs and product managers were (and I cannot overemphasize this enough) absolutely militant orthodox agile practitioners with jira.
I'd love to know more about what those specific circumstances were!
But on a personal level, this is like ordering something at a restaurant and the cook burning the kitchen because they forgot to take out your pizza out of the oven or something.
I would be telling it to everyone over beers (but not my boss).
I wonder if it has anything to do with the replicate.com purchase? Probably not.
In many ways it's still true, but it doesn't feel like a given anymore.
Feels like half the internet is down.
cannot login to get to workers to check - auth errors
I thought this was the point of a cached CDN!
Your browser: Working
Host: Working
Cloudflare: Error
What can I do?
Please try again in a few minutes.
I have Cloudflare running in production and it is affecting us right now. But at least I know what is going on and how I can mitigate (e.g. disable Cloudflare as a proxy if it keeps affecting our services at skeeled).
> If the problem isn’t resolved in the next few minutes, it’s most likely an issue with the web server you were trying to reach.
/s
Even just the basic question of "are we down or is our monitoring system just having issues" requires a human. And it's never "are we down", because these are distributed systems we're talking about.
If service X goes down entirely, does that warrant a status page update? Yes? Turns out system X is just running ML jobs in the background and has no customer impact.
If service Z's p95 response latency jumps from 10ms to 1500ms for 5 minutes, 500s spike at the same time, but overall 200s rate is around 98%, are we down? is that a status page update? Is that 1 bad actor trying to cause issues? Is that indicative of 2,000 customers experiencing an outage and the other 98,000 operating normally? Is that a bad rack switch that's causing a few random 500s across the whole customer base and the service will reject that node and auto-recover in a moment?
At the same time I'm worried about how the internet is becoming even more centralized, which goes against how it was originally designed.
So which large service we have left that could take chunk of internet out?
HAHA!
Our servers are still down, though
Maybe I'll do both
Globally meaningful outages of either are quite rare.
https://www.cloudflarestatus.com/history?page=8
https://www.cloudflarestatus.com/history?page=7
https://www.cloudflarestatus.com/history?page=6
https://www.cloudflarestatus.com/history?page=5
https://www.cloudflarestatus.com/history?page=4
https://www.cloudflarestatus.com/history?page=3
All the major cloud providers have regular incidents. Most go unnoticed, because they’re small or short.
When Cloudflare goes down: Oh no
Lister: What's the damage Hol?
Holly: I don't know. The damage report machine has been damaged.https://www.penguinrandomhouse.ca/books/661/mostly-harmless-...
Happily, the small ones that I also use are still going without anyone apparently even noticing. At least, the subject has yet to reach their local timelines at the time that I write this.
2 of the other major U.K. nodes are still up, too.
Everyone laughs when Azure collapses too
I rushed to Hacker News, but it was too early. Clicking on “new” did the job to find this post before making it to the Homepage:)
The web is still alive!
edit: It now says "Cloudflare Global Network experiencing issues" but it took a while.
Not downplaying the immense work of infra / engineering at this scale but my neighborhood local grocery market shouldn’t be down
I run a small video game forum with posts going back to 2008. We got absolutely smashed by bots scraping for training data for LLMs.
So I put it behind Cloudflare and now it's down. Ho hum.
As a side note, what does your site do that it's possible to use up all server resources? Computers are stupid fast these days. I find it's really difficult to build something that doesn't scale to at least multiple hundreds of requests per second.
The other part is just how convenient it is with CF. Easy to configure, plenty of power and cheap compared to the other big ones. If they made their dashboard and permission-system better (no easy way to tell what a token can do last I checked), I'd be even more of a fan.
If Germany's Telekom was forced to peer on DE-CIX, I'd always use CF. Since they aren't and CF doesn't pay for peering, it's a hard choice for Germany but an easy one everywhere else.
Hetzner has the WEAKEST DDoS protection out of ANYTHING out there - Arbor sucks.
Send me your website url and I'll keep it down for DAYS and whenever you cry to hetzner I'll just fry it again, it's that easy and that's why they're the cheapest - because everyone ran away from them back then.
The first time we switched to Cloudflare which saved us. Even with Cloudflare, the DDoS attempts are still damaging (the site goes down, we use Cloudflare to block the endpoints they're targeting, they change endpoints, etc.) but manageable. Without Cloudflare or something like it, I think it's possible that we'd be out of business.
How?
The VPS I use will nuke your instance if you run a game server. Not due to resource usage, but because it attracts DDoS like nothing else. Ban a teen for being an asshole and expect your service to be down for a week. And there isn't really Cloudflare for independent game servers. There's Steam Networking but it requires the developer to support it and of course Steam.
Valve's GDC talk about DDoS mitigation for games: https://youtu.be/2CQ1sxPppV4
So why be on Cloudflare to start with? Well, if you have a more reliable way then there's no reason. If you have a less reliable way, then you're on average better off with Cloudflare.
As for websites which don't need Cloudflare, in my experience almost every website will be DdoS attacked from time to time.
And why I should overthink my architecture now? If I had to manage redundant systems and keep track of circular dependencies I just could keep managing my infra the old way, no?
I'm being sarcastic here, obviously, but really one of the selling point for cloud back in the day it was "you don't have to care about those details". You just need to care about other details, now.
The place I work at has been online since 1996, not even a DoS yet, let alone a DDoS. Though we now use CF to filter all that bot traffic.
That is true. it is also the problem. It means the biggest providers do not even need to bother to be reliable because everyone will use them anyway.
But this is not really the case. When Azure/AWS were down, same as this one with Cloudflare: significant amount of web was down but most of it was not. It just makes more obvious which provider you use.
The issue is DNS since DNS propagation takes time. Does anyone have any ideas here?
Only if you're doing very basic proxy stuff. If you stack multiple features and maybe even start using workers, there may be no 1:1 alternatives to switch to. And definitely not trivially.
You think we have a say in this?
If the internet was always a nice place we wouldn't need Cloudflare and similar :(
However the https://www.cloudflarestatus.com/ does not really mention anything relevant. What's the point of having a status page if it lies ?
Update Ah I just checked the status and now I get a big red warning (however the problem existed for like 15 minutes before 11:48 UTC):
> Investigating - Cloudflare is aware of, and investigating an issue which potentially impacts multiple customers. Further detail will be provided as more information becomes available. Nov 18, 2025 - 11:48 UTC
Status pages are basically marketing crap right now. The same thing happened with Azure where it took at least 45 minutes to show any change. They can't be trusted.
RIP to the engineers fixing this without any AI help.
Concerning though how much the web relies on one (great) service.
> Investigating - Cloudflare is aware of, and investigating an issue which potentially impacts multiple customers. Further detail will be provided as more information becomes available.
Things are back up (a second time) for me.
Cloudflare have updated their status page now to reflect the problems now. It doesn’t sound like they are confident the problem is fully fixed yet.
Edit: and down again a third time!
Yeah, those multiple customers is like 70% of the internet.
The internet is officially down.
How are others doing this? How is Hacker News hosted/protected?
Your physical servers should have similar issues if you put a CDN in front unless the physical server is able to achieve a 100% uptime (100% * 3 9s = 3 9s). Or you don’t have a CDN but can be trivially knocked offline by the tiniest botnet (or even hitting hacker news front page)
And yes, I know that there's sites that need the scale of an operation like Cloudflare or AWS. But 99.9(...)% of pages don't, and people should start realizing that.
[0] (German) https://www.bsi.bund.de/SharedDocs/Downloads/DE/BSI/Cyber-Si...
DDoS mitigation is one of those areas that an on-prem solution just isn’t well suited to solve.
The system isn't designed for technical, rational decision making.
When even Cloudflare goes down, nobody can blame the little guys.
Then I was like... "when did I last time fly for 10+ hours and wanted to do programming, etc, so that I need offline docs?" So I gave up.
Today I can't browse the libs' docs quickly, so I'm resuming the work on my local mirroring :-)
A global upstream provider :)
Not-so-funny thing is that the Betterstack dashboard is down but our status page hosted by Betterstack is up, and we can't access the dashboard to create an incident and let our customers know what's going on.
Edit: wording.
If there is a slight positive note to all this, then it is that these outages are so large that customers usually seem to be quite understanding.
It's monthly by now
I think at the very least, one should plan the ability to switch to an alternative when your main choice fails… which together with AWS and GitHub is a weekly event now.
Most time I get ddosed now it's either Facebook directly, Something something Azure or any random AI.
If I choose something else, we're down, and our competitors aren't, then my overlords will start asking a lot of questions.
CF can be just as difficult if not more to migrate off of especially when using things like durable objects
Tbh though this is sort of all the other companies fault, "everyone" uses aws and cf and so others follow. now not only are all your chicks in one basket, so is everyone elses. When the basket inevitably falls into a lake....
Providers need to be more aware of their global impact in outages, and customers need to be more diverse in their spread.
Not really sure how our community is supposed to deal with this.
I have always felt so, but my opinion is definitely in the minority.
In fact, I find that folks have extremely negative responses to ant discussion of improving software Quality.
A large proportion of “developers” enjoy build vs buy arguments far too much.
Which only shows that chasing five 9s is worthless for almost all web products. The idea is that by relying on AWS or Cloudflare you can push your uptime numbers up to that standard, but these companies themselves are having such frequent outages that customers themselves don't expect that kind reliability from web products.
"Yes but what if they go down" - it doesnt matter, having it hosted by someone who can be down for the same reason as your main product/service is a recipe for disaster.
It took a few minutes but I got https://hcker.news off of it.
But then, that’s what Cloudflare signed up to be.
/s
BetterStack, InStatus and HetrixTools seemingly all use Cloudflare on their dashboards, which means I can't login but I keep getting "your website/API is down" emails.
Update: I also can't login to UptimeRobot and Pulsetic. Now, I am getting seriously concerned about the sheer degree of centralization we have for CDNs/login turnstiles on Cloudflare.
Name Server: NS-225.AWSDNS-28.COM Name Server: NS-1411.AWSDNS-48.ORG Name Server: NS-1914.AWSDNS-47.CO.UK Name Server: NS-556.AWSDNS-05.NET
At least for DNS. Data center appears to be Lightedge.
- [Sorry I broke the server](https://news.ycombinator.com/item?id=9052128) - [New attempt at mobile markup](https://news.ycombinator.com/item?id=10489499) - [Clickable domains and QoL](https://news.ycombinator.com/item?id=10223645) - [New features and a moderator](https://news.ycombinator.com/item?id=12073675) - [Thanks to thehodge and littlewarden, this site is up today](https://news.ycombinator.com/item?id=28472350)
Maybe one day. Seeing all of these big providers stumbling, an article about HN staying on top of everything would surely resonate.
The outages are the Roomba.
> If the problem isn’t resolved in the next few minutes, it’s most likely an issue with the web server you were trying to reach.
[1] https://www.cloudflare.com/5xx-error-landing/?utm_source=err...
Yeah I don't think you are using this phrase correctly
Shouting will not prevent errors, and you are only creating a hostile work environment where not acting is better than the risk of making a mistake and triggering an aggressive response from your part.
Looking forward to the post-mortem.
edit: it's up!
edit: it's down!
I'm going to take the metro now and thinking how long do we have until the entire transit network goes down because of a similar incident.
I'm leaving the redaction because I couldn't work atm...
Time for a beer , greetings from germany!
Edit: beautiful, this decentralised design of the internet.
I don't like it.
I get why, but it would give me more confidence if they would tell me about everything.
Im going home. Time for a beer .
Greetings from germany
Almost no one gets mad if your site and half the internet were down.
They'd only have to take down a few services to completely cripple the West - the exact case ARPANET was designed to prevent.
I am a self-host enthousiast. So I use Hetzner, Kamal and other tools for self-managing our servers, but we still have Cloudflare in front of them because we didn't want to handle the parts I mentioned (yet, we might sometime).
Calling it a mistake is a very narrow look at it. Just because it goes down every now and then, it isn't a mistake. Going for cloud or not has its trade-offs and I agree that paying 200 dollars a month for a 1GB Heroku Redis instance is complete madness when you can get a 4GB VPS on Hetzner for 3,8 a month. Then again, some people are willing to make that trade-off for not having to manage the servers.
Cloud servers have taught me so much about working with servers because they are so easy and cheap to spin up, experiment with and then get rid of again. If I had had to buy racks and host them each time I wanted to try something, I would've never done it.
But in the face of adversity, it's a huge liability. Imagine Chinese Hackers taking down AWS, Cloudflare, Azure and GCP simultaneously in some future conflict. Imagine what that would do to the West.
I don't believe in Fukuyamas End of History. History is still happening, and the choices we make will determine how it plays out.
and my alarms are going off my and support line is ringing...
I cant even login to my CF dashboard to disable the CDN!
Edit... and it's back. Hopefully it will stay up!
I guess claude is more important than your average site :)
The dashboard's API server runs on Cloudflare and is currently blocking all logins, will fix.
Y'know, along with most other SAAS services.
or search a new job for yourself. Maybe digging to the earth core. Why? Idk. Because then you can say: I did it, or so.
(index):64 Uncaught ReferenceError: $ is not defined at (index):64:3
I hope it gets resolved in the next hour or two, or it could be a serious problem for me.
resource "cloudflare_dns_record"
- proxied = true
+ proxied = false
https://www.pcgamer.com/gaming-industry/legendary-game-desig...
Browser Working
San Jose
Cloudflare Error
mysite.com
Host Working
Lol! Like a solar eclipse!
Our support portal provider is currently experiencing issues
Same goes for my personal projects: I've never been worried about being targeted by a botnet so much that I introduce a single point of failure like this.
What, they have Cloudflare and we don't? We also must have cloudflare. Don't ask why.
Now that you have it, you are at least level 15 and not a peasant.
Same applies to every braindead framework on the web. The gadget mind of the bois is the cause for all this.
I finally work around this by change the tcp options sent by vpp tcp stack.
But the whole thing made me worry there must be something deployed which cause this issue.
But I do not think that related with this network issue, it just reminds me the above, I feel there are frequently new articles about cloudflare networking, maybe new method or new deployment sort of related high probability of issues
It seems 20% of the Internet is down every two weeks now.
Unfortunately, that means they can also break 75% of the internet.
Coincidence? I think not.
So everyone who's wrapped their host with Cloudflare is stuck with it.
imdsm•1h ago
https://news.ycombinator.com/item?id=45955900
alexdns•1h ago
Jordan-117•1h ago
hirako2000•52m ago
polisaez•1h ago
chedabob•1h ago
pstation•1h ago
watermelon0•1h ago