I work on block storage at petabyte scale across AWS, Azure, and GCP. Lately I've been looking at what people actually run in production when they need shared/replicated block storage on NVMe hardware especially outside the hyperscaler managed offerings (EBS, Azure Disk, etc.). The landscape seems surprisingly thin:

Ceph RBD is the default but was designed for HDDs. Most benchmarks I've seen show 15–25% flash utilization and painful tail latencies on NVMe. Crimson/SeaStore is promising but still tech preview. LINSTOR/DRBD is solid and battle tested but not NVMe-oF native. Mayastor(OpenEBS) is SPDK-based and interesting but K8s-only, one pool per node, and I've seen users report 1/4 to 1/5 of raw NVMe performance. Vitastor has impressive performance but restrictive licensing (VNPL) and bus factor of 1. Lightbits / Simplyblock are proprietary.

Curious what others are running, especially:

If you're at a GPU neocloud or running large AI training clusters, what's your storage stack for checkpoints and shared data? If you run Ceph on NVMe, have you found it worth the overhead vs. simpler options? Anyone running NVMe-oF (TCP or RDMA) in production for shared block storage? What target are you using?