frontpage.
newsnewestaskshowjobs

Made with ♥ by @iamnishanth

Open Source @Github

fp.

Discuss – Do AI agents deserve all the hype they are getting?

4•MicroWagie•4h ago•1 comments

Ask HN: Anyone Using a Mac Studio for Local AI/LLM?

48•UmYeahNo•1d ago•30 comments

LLMs are powerful, but enterprises are deterministic by nature

3•prateekdalal•8h ago•6 comments

Ask HN: Non AI-obsessed tech forums

29•nanocat•19h ago•26 comments

Ask HN: Ideas for small ways to make the world a better place

18•jlmcgraw•21h ago•21 comments

Ask HN: 10 months since the Llama-4 release: what happened to Meta AI?

44•Invictus0•1d ago•11 comments

Ask HN: Who wants to be hired? (February 2026)

139•whoishiring•5d ago•520 comments

Ask HN: Who is hiring? (February 2026)

313•whoishiring•5d ago•514 comments

Ask HN: Non-profit, volunteers run org needs CRM. Is Odoo Community a good sol.?

2•netfortius•16h ago•1 comments

AI Regex Scientist: A self-improving regex solver

7•PranoyP•23h ago•1 comments

Tell HN: Another round of Zendesk email spam

104•Philpax•2d ago•54 comments

Ask HN: Is Connecting via SSH Risky?

19•atrevbot•2d ago•37 comments

Ask HN: Has your whole engineering team gone big into AI coding? How's it going?

18•jchung•2d ago•13 comments

Ask HN: Why LLM providers sell access instead of consulting services?

5•pera•1d ago•13 comments

Ask HN: How does ChatGPT decide which websites to recommend?

5•nworley•1d ago•11 comments

Ask HN: What is the most complicated Algorithm you came up with yourself?

3•meffmadd•1d ago•7 comments

Ask HN: Is it just me or are most businesses insane?

8•justenough•1d ago•7 comments

Ask HN: Mem0 stores memories, but doesn't learn user patterns

9•fliellerjulian•2d ago•6 comments

Ask HN: Is there anyone here who still uses slide rules?

123•blenderob•4d ago•122 comments

Kernighan on Programming

170•chrisjj•5d ago•61 comments

Ask HN: Anyone Seeing YT ads related to chats on ChatGPT?

2•guhsnamih•1d ago•4 comments

Ask HN: Does global decoupling from the USA signal comeback of the desktop app?

5•wewewedxfgdf•1d ago•3 comments

Ask HN: Any International Job Boards for International Workers?

2•15charslong•18h ago•2 comments

We built a serverless GPU inference platform with predictable latency

5•QubridAI•2d ago•1 comments

Ask HN: Does a good "read it later" app exist?

8•buchanae•3d ago•18 comments

Ask HN: Have you been fired because of AI?

17•s-stude•4d ago•15 comments

Ask HN: Anyone have a "sovereign" solution for phone calls?

12•kldg•4d ago•1 comments

Ask HN: Cheap laptop for Linux without GUI (for writing)

15•locusofself•3d ago•16 comments

Ask HN: How Did You Validate?

4•haute_cuisine•2d ago•6 comments

Ask HN: OpenClaw users, what is your token spend?

14•8cvor6j844qw_d6•4d ago•6 comments
Open in hackernews

A logging loop in GKE cost me $1,300 in 3 days – 9.2x my actual infrastructure

9•nthypes•2mo ago
Last month, a single container in my GKE cluster (Sao Paulo region) entered an error loop, outputting to stdout at ~2k logs/second. I discovered the hard way that GKE's default behavior is to ingest 100% of this into Cloud Logging with no rate limiting. My bill jumped nearly 1000% before alerts caught it.

Infrastructure (Compute): ~$140 (R$821 BRL) Cloud Logging: ~$1,300 (R$7,554 BRL)

Ratio: Logging cost 9.2x the actual servers.

https://imgur.com/jGrxnkh

I fixed the loop and paused the `_Default` sink immediately.

I opened a billing ticket requesting a "one-time courtesy adjustment" for a runaway resource—standard practice for first-time anomalies on AWS/Azure.

I have been rejected twice.

The latest response: "The team has declined the adjustment request due to our internal policies."

If you run GKE, the `_Default` sink in Log Router captures all container stdout/stderr.

There is NO DEFAULT CAP on ingestion volume which is an absurd!

A simple while(true); do echo "error"; done can bankrupt a small project.

Go to Logging -> Log Router. Edit _Default sink.

Add an exclusion filter: resource.type="k8s_container" severity=INFO (or exclude specific namespaces).

Has anyone successfully escalated a billing dispute past Tier 1 support recently?

It seems their policy is now to enforce full payment even on obvious runaway/accidental usage which is absurd since its LOGS! TEXT!

Comments

pants2•2mo ago
Congrats on finding it in within three days. I recently discovered something similar on our infra that had been going on for months (just an INFO log in a tight loop). This is the type of thing that major cloud providers should absolutely have sensible defaults and alerts on, but don't because it's how they make all their money.

It's one thing when it's company cash but I would never ever use a big cloud provider for a personal project (with my credit card). There are way too many ways to accidentally run up an infinite bill. DigitalOcean has plenty of services and more predictable costs, or your local data center will be happy to make a deal on some bare metal servers if you need more horsepower.

I hope you're able to get it reversed - try finding your GCP rep. Either way let this be a valuable lesson on using cloud.

nthypes•2mo ago
Yes, it's my 4th email to Billing Support and getting "No" as answers. Moving to Azure..
pants2•2mo ago
That's my point though. Azure or AWS are just as bad and that won't solve anything. What are you running that requires a major cloud provider?
bcrl•2mo ago
Implementing rate limiting in a scalable manner is harder than just making logging faster, and it makes the cloud provider money whereas rate limiting causes them to make less money. The incentive makes it pretty clear what side of the tradeoff cloud providers are going to come down on.