UPLOAD: It stores the first file in a S3 path as a full-size (reference), but saves next uploaded archives as deltas (tiny binary diffs) with respect to the reference.
DOWNLOAD: it reconstructs the original file on the fly, bit-perfect and verified with SHA256.
Why Xdelta3? It's a compression-aware and block-level binary diff algorithm. Perfect for representing differences between archives, where small changes shift bytes but most content stays the same. It can efficiently delta compress ZIP/JAR/TAR archives up to 99.9% between versions, provided the difference in compressed content is overall small.
Killer use cases Software versioning, periodic db. backups, JAR, ZIP, TGZ.
The impact for us was "2 orders of magnitude" storage price reduction. I hope you can benefit from it too!
License: GPLv3
Feedback and contributions are super welcome!
dilawar•2mo ago
Perhaps it can be easily integrated into self-hosted gitlab to manage artifacts?
sscarduzio•2mo ago