frontpage.
newsnewestaskshowjobs

Made with ♥ by @iamnishanth

Open Source @Github

Open in hackernews

Tell HN: The Hetzner Experience - Invisible Outages

21•AmazingTurtle•8h ago
I'm a DevOps engineer at Schäfer Shop GmbH, and we've been running into recurring quirks with Hetzner's cloud infrastructure. Occasionally, volumes randomly disappear or become unavailable, with no indication on Hetzner's status page. Support ticket response times vary wildly - from hours to an entire business day.

Yesterday, we had a particularly stressful incident involving Hetzner load balancers in Falkenstein. Our Kubernetes control planes were unreachable due to load balancer targets showing as unhealthy. We quickly worked around the issue by deploying an identical load balancer configuration in another region. Despite explicitly instructing Hetzner support not to recreate our resources (since they're managed via Terraform), they manually recreated the load balancer anyway, causing momentary panic - though thankfully our Terraform state wasn't impacted.

We pay nearly €20,000 per month for Hetzner's services, yet they refuse to offer a direct support hotline, even if we were willing to pay extra for it. What's especially troubling is their persistent silence on these outages. Hetzner's status page showed no signs of this incident, neither during nor after. This pattern makes us question the transparency and purpose of the status page itself.

Have any of you experienced similar invisible outages with Hetzner?

Comments

herbst•7h ago
I know Hetzner just from a small customer perspective, I think I never had anything else than their discount servers.

However, I don't run anything mission critical with them as they don't really have a reliable support. Just using their cheap dedis for background tasks.

The server I have right now is stable, but I had different experiences before as well. And their network is unreliable either way, timeouts etc...

preisschild•6h ago
We have 100s of HCloud machines and never encountered similar issues, fortunately.

Just the typical server outages and a "Fault report" notification Email from Hetzner

jeduardo•6h ago
6 years ago when I used Hetzner, it was widely known as unreliable, a provider where you could get hosting that delivered good performance for a cheap price. The tradeoff always was that you needed to treat each machine almost as EC2 Spot: it could go down at any time.

You also needed to consider that when this happened, the data inside the machine was mostly lost. Finally, you also needed to plan to graduate out of it as soon as you had enough money to go either to a colocated data center or the "real cloud".

I kept Hetzner as a backup provider in more than one company, mainly to have real machines for take home tests, back when hiring was plentiful. Even so, we often faced problems with the machines going down due to hardware or networking issues, and the need to rebuild them from the ground up. Those mirrored all tales of woe everyone in the department had from years of working with Hetzner, sometimes losing production data because the rules of the game were not followed.

So it seems that 6 years later their scale has increased but the experience remains the same. On the bright side, kudos to Hetzner for teaching waves of engineers about reliability and disaster recovery during all these years.

jesterson•6h ago
> What's especially troubling is their persistent silence on these outages.

Based on many years of experience, all providers are guilty of that. Only large scale outages or ones that just couldnt be ignored are reflected on status page. This doesn't necessarily mean malevolence on provider side - their sensors just may not be good enough to spot the issue.

On larger scale - why would you choose hetzner and then complain about uptime? Its a well know provider with low prices and low reliability. There are tons of businesses who find this model suitable for them. If yours is not one of them - just switch to something more reliable. Granted, your bill will likely be 2x+ of 20k eur, but you get what you whine about.

As old adage says, we can make this project fast, cheap and with amazing quality, but you can choose only 2 options.

johcard•5h ago
We've had a very similar experience over the past year. We've been using Hetzner for over a decade and, until recently, we were really satisfied with their services. But in the last 12 months, the reliability has noticeably dropped.

Most of the issues we've faced are related to their Storage Boxes, multiple incidents where they were completely unavailable, sometimes for hours or even days. What’s frustrating is that these outages are never reflected on their status page, so you're left in the dark unless you open a ticket yourself. Even then, the only explanation we usually get is that the specific Storage Box is under "heavy load", and the suggested fix is always to migrate to another box. That might be fine for infrequent use, but it's not acceptable when you're relying on it.

To be fair, Hetzner has been a solid provider for many years, and we’re still hoping this is just a temporary rough patch. We really hope they get things back on track soon.

palata•1h ago
Could it be related to people in Europe suddenly looking for European alternatives? Like suddenly they got a lot of new customers in the last few months?
danielops•3h ago
Crazy to see this now as we also had a very similar issue yesterday! All backend nodes of our kubernetes cluster were suddenly and inexplicably showing as unhealthy despite being all green on the cluster side, no signs of issues whatsoever.
adamcharnock•3h ago
We’ve used Hetzner dedicated servers for many years now, but not cloud.

Our experience has been excellent, but we also design for the platform. Redundant dedicated networking, multi-AZ, networking failover, RAID, k8s, Mayastor, etc.

The worst issues we see are occasional scheduled outages of an upstream router. This will take out an AZ for external traffic for about 20 mins, but the internal dedicated network will ensure internal services all stay up and quorate.

It’s not cheap, but it’s still cheaper than AWS.

I think their dedicated offering is probably more stable as it has been around longer, and it is also much much simpler. They need to provide networking, power, and finance the hardware. All of which is very much solved problems.

(We’re https://lithus.eu, if anyone is interested. You can contact me at adam@…. I’m on holiday this week, but back next week)

oulipo•1h ago
I've had a lot of issues with resources located at Falkenstein, there seems to be issue at that particular location. I've moved to other locations, and so far it's running fine
rgavuliak•56m ago
In my first job we've used Hetzner for Data Science work. The we lost servers twice in a year back then.

37Signals' HEY is finally for sale on the iPhone

https://world.hey.com/dhh/hey-is-finally-for-sale-on-the-iphone-a08cb218
1•skadamat•29s ago•0 comments

Show HN: An Introduction to Event Storming and DDD

https://leanpub.com/understand-your-domain-first
1•meaboutsoftware•1m ago•0 comments

Show HN: Colab script for low-cost transcription and summarization of meetings

https://github.com/tommasodesantis/meeting_summarization
1•tomdesantis•4m ago•0 comments

What would a walkable city look like? (2018)

https://www.theguardian.com/cities/2018/sep/19/what-would-a-truly-walkable-city-look-like
1•Tomte•4m ago•0 comments

Despite strict regulations, California still has the nation’s dirtiest air

https://www.thenewlede.org/2025/05/postcard-from-california-despite-strict-regulations-the-state-still-has-the-nations-dirtiest-air/
1•PaulHoule•6m ago•0 comments

For AI agents, what's the bigger problem: context sharing or prompting?

1•exclusivewombat•6m ago•0 comments

Coherence Information Theory and the Future of Communication

https://archive.org/details/info_20250204
1•xavaki•6m ago•0 comments

European editors oppose Hungary's move against foreign-funded groups

https://www.reuters.com/world/european-editors-sign-petition-opposing-hungarys-move-against-foreign-funded-2025-05-20/
1•pabo•6m ago•0 comments

SpiderMonkey Embedding Resources

https://github.com/mozilla-spidermonkey/spidermonkey-embedding-examples
1•mliezun•7m ago•0 comments

America Makes AI Chip Diffusion Deal with UAE and KSA

https://thezvi.substack.com/p/america-makes-ai-chip-diffusion-deal
1•jsnider3•8m ago•1 comments

The data center boom in the desert

https://www.technologyreview.com/2025/05/20/1116287/ai-data-centers-nevada-water-reno-computing-environmental-impact/
1•Brajeshwar•9m ago•0 comments

For AI agents, what's the bigger problem: context sharing or prompting?

1•exclusivewombat•9m ago•1 comments

Memos – An open-source, lightweight note-taking solution

https://github.com/usememos/memos
1•Brajeshwar•9m ago•0 comments

Show HN: 90s.dev - game maker that runs on the web

https://90s.dev/blog/finally-releasing-90s-dev.html
6•90s_dev•10m ago•2 comments

Evidence suggests a single hoaxer created 'Piltdown man' (2016)

https://royalsocietypublishing.org/doi/10.1098/rsos.160328
2•onychomys•10m ago•0 comments

Cartoon Network's Last Gasp

https://www.bloomberg.com/news/features/2025-05-13/cartoon-network-and-adult-swim-struggle-to-survive-zaslav-and-streaming
2•CharlesW•11m ago•0 comments

US hasn't seen a human bird flu case in three months. Experts are wondering why

https://www.bostonglobe.com/2025/05/19/nation/us-hasnt-seen-human-bird-flu-case-three-months-experts-are-wondering-why/
3•bookofjoe•12m ago•1 comments

Show HN: Astra – a new js2exe compiler

https://github.com/astracompiler/cli
8•qwertycodepl•13m ago•0 comments

If an AI agent can't figure out how your API works, neither can your users

https://stytch.com/blog/if-an-ai-agent-cant-figure-out-how-your-api-works-neither-can-your-users/
1•mattmarcus•16m ago•0 comments

Huawei widens lead in global telecom race, Western giants retreat under pressure

https://www.digitimes.com/news/a20250516PD220/telecom-huawei-policy-europe-5g.html
2•ksec•16m ago•0 comments

Nvidia part of plans for mega 1.4 GW AI datacenter near Paris

https://www.theregister.com/2025/05/20/paris_dc_hub_1_point_4_gw/
2•rntn•16m ago•0 comments

History of Diffusion – Sander Dieleman [video]

https://www.youtube.com/watch?v=ktPGNhe11cQ
1•jhoho•17m ago•0 comments

The tech billionaires are missing the point of their favorite sci-fi series

https://www.vox.com/culture/413502/iain-banks-culture-series-elon-musk-jeff-bezos-mark-zuckerberg
1•howard941•17m ago•0 comments

Mindflow

https://apps.apple.com/ru/app/mind-flow/id6740059119?l=en-GB
1•kaidendev•18m ago•2 comments

Ask HN: Did you build or buy your customer analytics?

2•lane_g•19m ago•0 comments

Without philosophy Einstein said he would have "contributed nothing"

https://iai.tv/articles/without-philosophy-einstein-said-he-would-have-contributed-nothing-auid-3172
1•delichon•19m ago•0 comments

China launches orbital data center satellites

https://spacenews.com/china-launches-first-of-2800-satellites-for-ai-space-computing-constellation/
2•seanwatson•22m ago•1 comments

Chicago Sun Times used AI for its (fake) summer reading list

https://old.reddit.com/r/chicago/comments/1kr4eg7/chicago_sun_times_used_ai_for_its_summer_reading/
4•tahoupt•22m ago•0 comments

Greenspun's 10th rule and the sad state of software quality

https://gist.github.com/MIvanchev/1b0af1de6324626bf9cd6358e7154177
6•eadmund•22m ago•1 comments

Sag-Aftra Responds to Fortnite's AI Darth Vader with Unfair Labor Complaint

https://deadline.com/2025/05/sag-aftra-fortnites-ai-darth-vader-unfair-labor-complaint-1236405218/
1•ChrisArchitect•24m ago•0 comments