Tell HN: DigitalOcean's managed services broke each other after update

27•neilfrndes•1h ago

Yesterday my production app went down. The cause? DigitalOcean's managed PostgreSQL update broke private VPC connectivity to their managed Kubernetes.

Public endpoint worked. Private endpoint timed out. Root cause: a Cilium bug (#34503) where ARP entries go stale after infrastructure changes.

DO support responded relatively quickly (<12hrs). Their fix? Deploy a DaemonSet from a random GitHub user to ping stale ARP entries every 10 seconds. The upstream Cilium fix is merged but not yet deployed to DOKS. No ETA.

I chose managed services specifically to avoid ops emergencies. We're a tiny startup paying the premium so someone else handles this. Instead, I spent late night hours debugging VPC routing issues in a networking layer I don't control.

HN's usual advice is "just use managed services, focus on the business." Generally good advice. But managed doesn't mean worry-free, it means trading your failure modes for the vendor's failure modes. You're not choosing between problems and no problems. You're choosing between problems you control and (fewer?) problems you don't.

Still using DO. Still using managed services. Just with fewer illusions about what "managed" means.

Comments

cosmin800•1h ago

Lower prices come with a cost. I am not a fan of AWS but they higher reliability.

delish•15m ago

The font color implies this comment is downvoted, but I earnestly encourage readers to take very seriously the difference in SLOs and SLAs between high-cost vendors like AWS and GCP and low-cost vendors like DigitalOcean. Read their docs; do not assume DO is "the same, but lower cost."

sethops1•1h ago

Obligatory, do you actually need kubernetes? I struggle to imagine any tiny startup that does.

osigurdson•35m ago

Running Kubernetes in a managed environment like DO is no harder than using docker compose.

cadamsdotcom•44m ago

100% uptime is impossible of course, a 100% reliable service would survive the next ice age.

But reliability at the holy grails of 4 and 5 nines (99.99%, 99.999% uptime) means ever greater investment - geographically dispersing your service, distributed systems, dealing with clock drift, multi master, eventual consistency, replication, sharding.. it’s a long list.

Questions to ask: could you do better yourself - with the resources you have? Is it worth the investment of a migration to get there? Whats the payoff period for that extra sliver of uptime? Will it cost you in focus over the longer term? Is the extra uptime worth all those costs?

kevin_nisbet•24m ago

> I chose managed services specifically to avoid ops emergencies. We're a tiny startup paying the premium so someone else handles this. Instead, I spent late night hours debugging VPC routing issues in a networking layer I don't control.

This happens with managed services and I understand the frustration, but vendors are just as fallible as the rest of us and are going to have wonky behaviour and outages, regardless of the stability they advertise. This is always part of build vs buy, buy doesn't always guarentee a friction free result.

It happens with the big cloud providers as well, I've spent hours with AWS chasing why some VMs are missing routing table entries inside the VPC, or on GCP we had to just ban a class of VMs because the packet processing was so bad we couldn't even get a file copy to complete between VMs.

The secretive powerbroker with a tight grip on corporate Spain

Timelang: Natural Language Time Parser

Young Men Are Lost. A New Book Can Guide the Way

Trump Touts New Microsoft Data-Center Pledges After Local Backlash

Show HN: ProofLoop – Autonomous long-running agents with verifiable completion

Show HN: I made a $25 lifetime Screen Studio alternative

Are Atomic Operations Better Than a Mutex? It Depends

They Write the Right Stuff

A rare interview with the elusive Agatha Christie

Be Wary of Digital Deskilling

Show HN: Yoth-yoth – your all-in-one workspace

Data is not available upon request

I Graduated from Survival Mode

The Pentagon used a secret aircraft painted to look like a civilian plane

Verizon to stop automatic unlocking of phones as FCC ends 60-day unlock rule

A 'Holy Grail' Sleep Apnea Pill Could Be on the Market Next Year

Great code doesn't matter if you can't sell it

FCC revises Verizon phone unlocking rules after significant fraud issues

Canada's Scaling Problem Isn't Compute, It's Coastlines

Show HN: Minimal type-safe language for software architecture

Bug 55867 – Doesn't know how to tag XI_TRACKBALL

From Starbase: Pete Hegseth on Defense Innovation Reform [video]

Meta shakes up its review system with 'stronger rewards for top performers'

Paramount Wants Warner to Show Its Work

Turning Agents into Learning Machines

DJT Says Microsoft to Make Changes to Curb Data Center Power Costs for Americans

Living with LLMs Everywhere – How Ambient LLMs Negate Security Policy

Who Companies Call When They Want to Become a Bank

Apple: You (Still) Don't Understand the Vision Pro

Show HN: Idlen.io ($IDL), the first privacy-first AI ad network is launched