In the end, Hetzner is a provider of "cheap but not 100% uptime" infrastructure, probably why it's so cheap in the first place.
As every other provider, if you want 100% uptime (or getting close to it), you really need at least N+1 instances of everything, as every hosting provider end up fucking something up, sooner or later.
Sure they’ll throw you some service credits. But it’ll always be magnitudes less than the cost of their disruption to you.
We kept most smaller-scale, stateless services in AWS but migrated databases and high-scale / high-performance services to bare metal servers.
Backups are stored in S3 so we still benefit from their availability.
Performance is much higher thanks to physically attached SSDs and DDR5 on-die RAM.
Costs are drastically lower and for much larger server sizes which means we are no getting stressed about eventually needing to scale up our RDS / EC2 costs.
It's literally a agency doing professional development for others, among other services. Clearly not "toys".
HN dismissals are going down in quality, at least they used to be well researched some years ago. Now people just spew out the first thing that comes up in their mind, and zero validation before hitting that "reply" button.
It's a rotten attitude, and judging a projects worth by an AWS bill is a very poor comparator. I could spin up a massive aws bill doing some pointless machine learning workloads, is that suddenly a valid project in your eyes?
When I've needed dedicated servers in the US I've used Vultr in the past, relatively nice pricing, only missing unmetered bandwidth for it to be my go-to. But all those US-specific cases been others paying for it, so hasn't bothered me, compared to personal/community stuff I host at Hetzner and pay for myself.
For example, I got a dedicated server from Hetzner earlier this year with a consumer Ryzen CPU that had unstable SIMD (ZFS checksums would randomly fail, and mprime also reported errors). Opened a ticket about it and they basically told me it wasn't an issue because their diagnostics couldn't detect it.
And based on our different experiences, the quality of care you receive could differ too :)
To be fair, they probably would've done the same for me if I'd pushed the issue further, but after over a week of trying to diagnose the issue and convince them that it wasn't an problem with the hard drives (they said one of the drives was likely faulty and insisted on replacing it and having me resilver the zpool to see if it fixed the issue. spoiler: it didn't) I just gave up, disabled SIMD in ZFS and moved on.
That sucks big time :( In the most recent case I can recall, I successfully got access, noticed weirdness, gathered data and sent an email, and had a new instance within 2-3 hours.
Overall, based on comments here on HN and otherwhere, the quality and speed of support is really uneven.
Too cool to not share, most of the providers listed there have dedicated servers too.
Edit: Ironically, that website doesn't have Hetzner in their index.
excellent website, thanks.
FWIW, Hetzner has two data centers in the US, in case you're just looking for "Hetzner quality but in the US", not for "American/Canadian companies similar to Hetzner".
Hetzner, OVH, Leaseweb, and Scaleway (EU locations only).
I've used other providers as well, but I won't mention them because they were either too small or had issues.
Years ago Broadberry has a similar thing with Supermicro, but not any more. You have to talk to a sales person about how they can rip you off. Then they don't give you what you specced anyway -- I spec 8x8G sticks of ram, they provide 2x32G etc.
In the best case scenario. In the worst, some cluster f-up will eat 10x that in engineering time.
The only benefit you get is reliability, temporary network issues on AWS are not a thing.
On DigitalOcean they are fairly bad (I lose thousands of requests almost every month and I get pennies in credit back when I complain - while my users churning cost way more), on Hetzner I've heard mixed reviews.
Some people complains, some say it's extremely reliable.
I'm looking forward to try Hetzner out!
Yeah, I remember when AWS first appeared, and the value proposition was basically "It's expensive but you can press a button and a minute later you have a new instance, so we can scale really quickly". For the companies that know more or less the workload they have during a week don't really get any benefits, just more expensive monthly bills.
But somewhere along the line, people started thinking it was easier to use AWS than the alternatives, and I even heard people saying it's cheaper...
But in general if you don't need to scale crazy Hetzner is amazing, we still have a lot of stuff running on Hetzner but fan out to other services when we need to scale.
My point of people moving to Hetzner for the dedicated instances rather than the cloud still remains though, at least in my bubble.
I'm not sure if this is a difference between other clouds, at least a few years ago this was a weekly or even daily problem in GCP; my experience is if you request hundreds of VMs rapidly during peak hours, all the clouds struggle.
He's also just released a book on hosting scale production Python apps [3]. Haven't read yet though would assume it'll get covered there in more detail too.
--
[1] https://talkpython.fm/blog/posts/we-have-moved-to-hetzner/
[2] https://talkpython.fm/blog/posts/update-on-hetzner-changes-p...
Yeah, even when you move to "EC2 Dedicated Instances" you end up sharing the hardware with other instances, unless you go for "EC2 Dedicated Hosts", and even then the performance seems worse than other providers.
Not sure how they managed to do so for even the dedicated stuff, would require some dedicated effort.
A good example is a the big lichess outage from last year [1]. Lichess is a non-profit, and also must serve a huge user base. Given their financials, they have to go the cheap dedicated server route (they host on OVH). They publish an Excel sheet somewhere with every resources they use to run the services and last year, I had fun calculating how much it would cost them if they were using an hyperscaler cloud offering instead. I don't remember exactly but it was 5 or 6x the price they currently pay OVH.
The downside, is that when you have an outage, your stuff is tied to physical servers and they can't easily be migrated, when cloud provider on the opposite can easily move around your workload. In the case of Lichess outage, it was some network device they had no control of that went bad, and lichess was down until OVH could fix it, that is many hours.
So, yes you get a great deal, but for a lot of businesses, uptime is more important than cost optimization and the physicality of dedicated servers is actually a serious liability.
[1]: https://lichess.org/@/Lichess/blog/post-mortem-of-our-longes...
Even hosting double of everything when you're doing dedicated servers will let you have cheaper monthly bills, compared to the same performance/$ you could get with AWS or whatever.
But Hetzner does seem a bit worse than other providers in that they have random failures in their own infrastructure, so you do need to take care if you wanna avoid downtime. I'm guessing that's how they can keep the prices so low.
> is that when you have an outage, your stuff is tied to physical servers and they can't easily be migrated
I think that's a problem in your design/architecture, if you don't have backups that live outside the actual servers you wanna migrate away from, or at least replicate the data to some network drive you can easily attach to a new instance in an instant.
When you pay 1/4 for 3X the performance you can duplicate your servers and then be paying 1/2 for 3X the performance.
I find baffling that people forget about how things were done before the cloud.
So they could have had 100% redundant systems at OVH and still be under half the cost of a traditional "cloud" provider?
I would look at architecture and operations first. Their "main" node went down, and they did not have a way they could just bring another instance of it online fast on a fresh OVH machine (typically provisioned in a few minutes, assuming they had no hot standby). If the same happened to their "main" VM at a "hyperscaler" , I would guess they also would have been up the same creek. It is not the difference between 120 and 600 seconds to provision a new machine that caused their 10 hrs downtime.
But I think "redundancy" is more like a spectrum, rather than a binary thing. You can be more or less redundant, even within the same VPS if you'd like, but that of course be less redundant than hosting things across multiple data centers.
I don't see how that follows? Could you please explain?
I run my stuff on Hetzner physical servers. It's deployed/managed through ansible. I can deploy the same configuration on another Hetzner cluster (say, in a different country, which I actually do use for my staging cluster). I can also terraform a fully virtual cloud configuration and run the same ansible setup on that. Given that user data gets backed up regularly across locations, I don't see the problem you are describing?
This is a myth, created so cloud providers can sell more, and so those who overpay can feel better. I've been using dedicated servers since 2005, so for 20 years across different providers. I have machines at these providers with 1000-1300 days of uptime.
Looking at Hetzner or Vultr as alternatives. A few folks mentioned me Infomaniak has great service and uptime, but I haven't heard much about them otherwise.
Anyone used Infomaniak in production? How do they compare to Hetzner/Vultr?
Both Vultr and Hetzner are solid options, I'd go for Hetzner if I know the users are around Europe or close to it, and I want to run tiny CDN-like nodes myself across the globe. Also, Hetzner if you don't wanna worry about bandwidth costs. Otherwise go for Vultr, they have a lot more locations.
The lightsail instance sometimes just hangs and we have to reboot it when people performing simple action like login or queryng API (we have a simple express / nextjs app)
We ended up building a managed Postgres that runs directly on Hetzner. Same setup, but with HA, backups, and PITR handled for you. It’s open-source, runs close to the metal, and avoids the egress/I/O gotchas you get on AWS.
If anyone’s curious, I added here are some notes about our take [1], [2]. Always happy to talk about it if you have any questions.
[1] https://www.ubicloud.com/blog/difference-between-running-pos... [2] https://www.ubicloud.com/use-cases/postgresql
Not having an ops background I am nervous about:
* database backup+restore * applying security patches on time (at OS and runtime levels) * other security issues like making sure access to prod machines is restricted correctly, access is logged, ports are locked down, abnormal access patterns are detected * DoS and similar protections are not my responsibility
It feels like picking a popular cloud provider gives a lot of cover for these things - sometimes technically, and otherwise at least politically...
How come? The baseline for that comparison will also stay static, regardless of how many TPS or whatever is going on, meanwhile the AWS price they're comparing to would only increase the more people use whatever they deploy.
My hosting bill is a fraction of what people pay at AWS or other similar providers, and my servers are much faster. This lets me use a simpler architecture and fewer servers.
When I need to scale, I can always add servers. The only difference is that with physical servers you don't scale up/down on demand within minutes, you have to plan for hours/days. But that's perfectly fine.
I use a distributed database (RethinkDB, switching to FoundationDB) for fault tolerance.
In terms of networking many offer no-headache solutions with some kind of transit blend.
<rant>I recently had to switch away from hetzner due to random dhclient failures causing connectivity loss once ip's expired, complete failure of the loadbalancer - stopped forwarding traffic for around 6 hours and the worst part is that there was no acknoledgement from hetzner about any of these issues so at some point I was going insane over trying to find what is the issue when in the end it was hetzner. (US VA region)
A dedicated server or VPS from OVH, Hetzner, Scaleway, etc., or even Docker containers on Koyeb, will give you way more bang for your buck.
Call me a dinosaur, but I’ve never used any of the big cloud providers like AWS. They’re super expensive, and it’s hard to know what you’ll actually end up paying at the end of the month.
I'd love to hear more about how you use terraform and helm together.
Currently our major friction in ops is using tofu (terraform) to manage K8s resources. Avoiding yaml is great - but both terraform and K8s maintaining state makes the deployment of helm from terraform feel fragile; and vice-versa depending on helm directly in a mostly terraform setup also feels fragile.
It was a wake up moment for me about keeping billing in shape, but also made me understand that a cloud provider is as good as their support and communications when things go south. Like an automated SMS would be great before you destroy my entire work. But because they are so cheap, they probably can't do that for every 100$/month account.
I've had similar issues with AWS, but they will have much friendlier grace periods.
But if you do not pay and you do not check your e-mails, it's basically your fault. Who is using SMS these days even?
1. How many nodes do you have? 2. Did you install anything to monitor your node(s) and the app deployed on these nodes? If so, which software?
geenat•1h ago
Amazon gets far too greedy- particularly bad when you need egress.
Also an "amazon core" is like 1/8th of a physical cpu core.
CaptainOfCoit•1h ago
vidarh•1h ago
Clearly when Amazon realised the enormous potential in AWS, they scrapped that principle. But the idea behind it - that an organisation used to fat margins will not be able to adapt in the face of a competitor built from the ground to live of razor thing margins - still applies.
AWS is ripe for the picking. They "can't" drop prices much, because their big competitors have similar margins, and a price war with them would devastate the earnings of all of them no matter how much extra market share they were to win.
The challenge is the enormous mindshare they have, and how many people are emotionally invested even in believing AWS is actually cost effective.
master_crab•45m ago
Yup, that phrase was running through my head as I skimmed the comments.
To that, an interesting observation I’ve made is that their frequency for service price cuts have dropped in the past several years. And the instances of price increases have started to trickle in (like the public IP cost).
If core compute and network keep getting cheaper faster than inflation, and they never drop their prices (or drop them by less relatively) the margins are growing.