frontpage.
newsnewestaskshowjobs

Made with ♥ by @iamnishanth

Open Source @Github

fp.

Open in hackernews

A logging loop in GKE cost me $1,300 in 3 days – 9.2x my actual infrastructure

5•nthypes•4h ago
Last month, a single container in my GKE cluster (Sao Paulo region) entered an error loop, outputting to stdout at ~2k logs/second. I discovered the hard way that GKE's default behavior is to ingest 100% of this into Cloud Logging with no rate limiting. My bill jumped nearly 1000% before alerts caught it.

Infrastructure (Compute): ~$140 (R$821 BRL) Cloud Logging: ~$1,300 (R$7,554 BRL)

Ratio: Logging cost 9.2x the actual servers.

https://imgur.com/jGrxnkh

I fixed the loop and paused the `_Default` sink immediately.

I opened a billing ticket requesting a "one-time courtesy adjustment" for a runaway resource—standard practice for first-time anomalies on AWS/Azure.

I have been rejected twice.

The latest response: "The team has declined the adjustment request due to our internal policies."

If you run GKE, the `_Default` sink in Log Router captures all container stdout/stderr.

There is NO DEFAULT CAP on ingestion volume which is an absurd!

A simple while(true); do echo "error"; done can bankrupt a small project.

Go to Logging -> Log Router. Edit _Default sink.

Add an exclusion filter: resource.type="k8s_container" severity=INFO (or exclude specific namespaces).

Has anyone successfully escalated a billing dispute past Tier 1 support recently?

It seems their policy is now to enforce full payment even on obvious runaway/accidental usage which is absurd since its LOGS! TEXT!

Comments

pants2•3h ago
Congrats on finding it in within three days. I recently discovered something similar on our infra that had been going on for months (just an INFO log in a tight loop). This is the type of thing that major cloud providers should absolutely have sensible defaults and alerts on, but don't because it's how they make all their money.

It's one thing when it's company cash but I would never ever use a big cloud provider for a personal project (with my credit card). There are way too many ways to accidentally run up an infinite bill. DigitalOcean has plenty of services and more predictable costs, or your local data center will be happy to make a deal on some bare metal servers if you need more horsepower.

I hope you're able to get it reversed - try finding your GCP rep. Either way let this be a valuable lesson on using cloud.

nthypes•1h ago
Yes, it's my 4th email to Billing Support and getting "No" as answers. Moving to Azure..

Ask HN: Scheduling stateful nodes when MMAP makes memory accounting a lie

6•leo_e•2h ago•3 comments

Ask HN: Hearing aid wearers, what's hot?

320•pugworthy•17h ago•187 comments

GhostBin A lightweight pastebin, built with Go and Redis

2•sanaf•2h ago•0 comments

A logging loop in GKE cost me $1,300 in 3 days – 9.2x my actual infrastructure

5•nthypes•4h ago•2 comments

Ask HN: Good resources to learn financial systems engineering?

127•_1tan•1d ago•24 comments

Tell HN: Cursor charged 19 subscriptions, won't refund

3•devtailz•4h ago•2 comments

Tell HN: Declaration of Independence is 100% AI, according to AI Checker

7•whatamidoingyo•3h ago•2 comments

Ask HN: Photos corrupted on Google Pixel phones over time?

5•poolnoodle•7h ago•1 comments

Don't obsess with security and privacy unless they are your core business

6•amano-kenji•13h ago•10 comments

Malicious Bun Script Found in NPM Package Bumps

3•kothariji•10h ago•1 comments

Tell HN: Wanted to Give Dang Appreciation

37•razodactyl•20h ago•3 comments

Ask HN: What tools do you pay for today that feel overpriced or frustrating?

5•psicombinator•16h ago•6 comments

Why isn't There a open-source (project) game?

3•triilman•19h ago•6 comments

ZetaShare Building private file transfer with WebRTC

3•masterdegrees•22h ago•0 comments

Malware in PostHog NPM packages

9•roskoalexey•10h ago•7 comments

Ask HN: Is America in Recession?

20•register•1d ago•27 comments

Ask HN: Advice for feeling like a failure in PhD?

8•phdthrowaway1•1d ago•2 comments

Ask HN: How do you balance creativity, love for the craft, and money?

14•introvertmac•2d ago•9 comments

Boring Laser Eyes Simulator: Add laser beams to your eyes with your webcam

3•frankhsu•2d ago•0 comments

Ask HN: Working in a language that isn't your native one. How hard was it?

8•william-cooke•2d ago•16 comments

Ask HN: Where can you find old NetBSD packages?

12•GaryBluto•2d ago•5 comments

Facebook has made it impossible to delete Pages – dark patterns everywhere

47•ramharts•5d ago•16 comments

Ask HN: First Steps with a Patent Troll?

10•throwawaynvmbr•2d ago•11 comments

Ask HN: Is it time to measure Inflation and CPI without the government?

16•cyrusradfar•2d ago•6 comments

Ask HN: What is the current state of the art in BIG (>5TB) cloud backups?

22•jacobwilliamroy•5d ago•20 comments

Ask HN: Current state of Android USB tethering?

8•namesarehard•2d ago•0 comments

Ask HN: Where could I find early adopters?

4•nonmaskable•23h ago•1 comments

Ask HN: How would you architect a RAG system for 10M+ documents today?

21•Ftrea•3d ago•8 comments

Warp Terminal Doing a Pricing Shock for Canadian Users – 10× Increase on Credits

4•osmosismartini•2d ago•3 comments

Fun weekend task – Calculate your crypto relief or regret

6•shouldabought•3d ago•1 comments