Haha, yep we were lucky to catch this early! It could easily have gotten lost with everything else in the monthly AWS bill.

Came here to say the same, take my vote

    - DevOops

That was truly my hope with this post! Glad to hear that

I think it's been calculated that data transfer is the biggest margin product in all AWS catalog by a huge difference. A 2021 calculation done by Cloudflare [0] estimated almost 8000% price markup in EU and US regions.

And I can see how, in very big accounts, small mistakes on your data source when you're doing data crunching, or wrong routing, can put thousands and thousands of dollars on your bill in less than an hour.

--

  0: https://blog.cloudflare.com/aws-egregious-egress/

I don't think its about profits, its about incentivising using as many AWS products as possible. Consider it an 'anti-lock-in fee'

I presume it depends on your ability to pay for your mistakes. A $20/month client is probably not going to pony up $1000, a $3000/month client will not care as much.

They do sometimes if you ask. Probably depends on each case though.

Hahaha. I'll update the post once I hear back from them. One could hope that they might consider an account credit.

Depends on various factors and of course the amount of money in question. I've had AWS approve a refund for a rather large sum a few years ago, but that took quite a bit of back and forth with them.

Crucial for the approval was that we had cost alerts already enabled before it happened and were able to show that this didn't help at all, because they triggered way too late. We also had to explain in detail what measures we implemented to ensure that such a situation doesn't happen again.

> Does Amazon refund you for mistakes

Hard no. Had to pay I think 100$ for premium support to find that out.

Reading all the posts about people who got bitten by some policies on AWS, I think they should create two modes:

- raw

- click-ops

Because, when you build your infra from scratch on AWS, you absolutely don't want the service gateways to exist by default. You want to have full control on everything, and that's how it works now. You don't want AWS to insert routes in your route tables on your behalf. Or worse, having hidden routes that are used by default.

But I fully understand that some people don't want to be bothered but those technicalities and want something that work and is optimized following the Well-Architected Framework pillars.

IIRC they already provide some CloudFormation Stacks that can do some of this for you, but it's still too technical and obscure.

Currently they probably rely on their partner network to help onboard new customers, but for small customers it doesn't make sense.

Yes uploading into AWS is free/cheap. You pay per GB of data downloaded, which is not cheap.

You can see why, from a sales perspective: AWS' customers generally charge their customers for data they download - so they are extracting a % off that. And moreover, it makes migrating away from AWS quite expensive in a lot of circumstances.

Yes. It’s not very subtle.

the statement is about aws in general, and yes, you pay for bandwith

Yes…?

Egress bandwidth costs money. Consumer cloud services bake it into a monthly price, and if you’re downloading too much, they throttle you. You can’t download unlimited terabytes from Google Drive. You’ll get a message that reads something like: “Quota exceeded, try again later.” — which also sucks if you happen to need your data from Drive.

AWS is not a consumer service so they make you think about the cost directly.

That is the business model and one of the figurative moats: easy to onboard, hard/expensive (relative to on-boarding ) to divest.

Though important to note in this specific case was a misconfiguration that is easy to make/not understand in the data was not intended to leave AWS services (and thus should be free) but due to using the NAT gateway, data did leave the AWS nest and was charged at a higher data rate per GB than if just pulling everything straight out of S3/EC2 by about an order of magnitude (generally speaking YMMV depending on region, requests, total size, if it's an expedited archival retrieval etc etc)

So this is an atypical case, doesn't usually cost $1000 to pull 20TB out of AWS. Still this is an easy mistake to make.

Made in California.

We are programmed to receive. You can check out any time you like, but you can never leave

Nine cents per gigabyte feels like cellphone-plan level ripoff rather than a normal amount for an internet service.

And people wonder why Cloudflare is so popular, when a random DDoS can decide to start inflicting costs like that on you.

Thats a year salary but hey think about how much more complicated your work would be if you had to learn to self-host your infra!

You could have been running a 24 core $60/month unlimited bandwidth server on IONOS.

Personally I don't even understand why NAT gateways are so prevalent. What you want most of the time is just an Internet gateway.

There's nothing to gain in punching down

They made a mistake and are sharing it for the whole word to see in order to help others avoid making it.

It's brave.

Unlike punching down.

Examples?

It's not that it's technically impossible. The very simple problem is that there is no way of providing hard spend caps without giving you the opportunity to bring down your whole production environment when the cap is met. No cloud provides wants to give their customers that much rope to hang themselves with. You just know too many customers will do it wrong or will forget to update the cap or will not coordinate internally, and things will stop working and take forever to fix.

It's easier to waive cost overages than deal with any of that.

We are also 100% customer-funded. AWS makes sense for us for the enterprise version of Geocodio where we are SOC2 audited and HIPAA-compliant.

We are primarily using Hetzner for the self-serve version of Geocodio and have been a very happy customer for decades.