the hot object replication was inspired by groupcache [1], but that has some pretty nasty issues when the cluster resizes
s3-accelerator fixes that by: 1) using rendezvous hashing to reduce the remapping when the cluster loses a node, and 2) when new nodes are added, asking the old parent for a give key for the object rather than going to s3
It also uses learnings from cloudflare that it's better to cache on a second hit than a first hit[2]
It also uses ktls, splice, and sendfile so byte transfer of object chunks never hit userspace. That means it can hit millions of iops, or saturate 200Gb/s network cards, while sipping on CPU
it also is parquet-aware, and has an optional SQS integration to keep the cache fresh if an object is overwritten.
S3 stays the source of truth, and you can write through the cache (it wont cache the write by default)
these features make s3-accelerator uniquely performant, fresh, and s3 costs extremely low.
It's one of those "I'll build it one day" project ideas that Opus 5.5 finally made viable.
[1] https://github.com/golang/groupcache [2] https://blog.cloudflare.com/why-we-started-putting-unpopular...