I don't think this can be definitively answered without working for one of the hyperscalers. But here are some speculations: 1. Device speeds are intentionally capped to increase device lifetime (but this would only make sense for writes) 2. Networked storage services like EBS are more profitable, and AWS would like to phase out instance-attached storage. 3. Technical limitations/virtualization overhead (See comment above). I don’t have enough insight of how AWS SSDs work under the hood, but high network throughput (600 Gbit/s) is possible even in virtualized instances. Then again, we have certainly seen some weird noisy neighbor effects on cloud SSDs. However, it's worth mentioning that the same throughput limitations also apply to bare metal instances, where users don't benefit from virtualization (https://docs.aws.amazon.com/ec2/latest/instancetypes/so.html...). 4. There’s too little customer demand for fast SSDs, and optimization is not worth the effort.

Nvme pricing is pretty volatile in the past 2 years I’ve seen it move between 2-3x from its low post Covid.

I don’t think the prices have adjusted because of that. Additional during Covid the prices were very high and this is baked into the pricing.

Yeah, I would keep in mind that not everyone is optimmizing for $/perf, some use cases (where data is colder) are optimizing for $/GiB stored.

I've seen a lot of workloads that had multiple servers or large RAID'ed NAS devices get shrank down to a single server after a single NVMe could provide more than enough random IOPS.

I’m not disagreeing with this necessarily, but I do think a lot of people underestimate the costs of actually doing on-prem to a professional standard. You’ll almost certainly have to hire a dedicated team to manage your hardware, and you’re off in the woods as far as most of the rest of the world’s operating stack - an awful lot assumes you’re on EKS with infinite S3 and ECR available. It’s doable, but it’s not drag & drop - the cloud providers are expensive, but they are providing a lot.