I started indexed_bzip2 years ago as a seekable, parallelized bzip2 backend for ratarmount. It has been dormant for a while and seemed almost finished. Out of interest, I tried looking for bugs with DeepSeek V4.1 Flash, which found dozens of them, most with a minimal reproducing test. After reviewing and merging those and making a bugfix release, I prompted it with: "Speed up decompression by 2x". Surprisingly, it worked! You can try it out yourself with (timings on Ryzen 3900X):
pip install indexed_bzip2==2.0 # Depending on your default: --uploaded-prior-to P0D, or uvx, pipx
# single-threaded gzip decompression
time igzip -d -c silesia-20x.tar.gz | cksum # 7.5 s
time gzip -d -c silesia-20x.tar.gz | cksum # 22.4 s
# single-core bzip2
time bzip2 -d -c silesia-20x.tar.bz2 | cksum # 105 s
time lbzip2 -d -c -n 1 silesia-20x.tar.bz2 | cksum # 77.6 s
time ibzip2 -d -c -P 1 silesia-20x.tar.bz2 | cksum # 23.7 s
# parallelized
time lbzip2 -d -c silesia-20x.tar.bz2 | cksum # 6.7 s
time ibzip2 -d -c silesia-20x.tar.bz2 | cksum # 3.8 s
# single-threaded compression
time igzip -c silesia-20x.tar | cksum # 11.3 s
time gzip -c silesia-20x.tar | cksum # 147 s
time bzip2 -c -z silesia-20x.tar | cksum # 312 s
time ibzip2 -c -z -P 1 silesia-20x.tar | cksum # 164 s
# parallel compression
time lbzip2 -z -c -n 24 silesia-20x.tar | cksum # 12.7 s
time ibzip2 -z -c -P 24 silesia-20x.tar | cksum # 13.7 s
It not only added lots of dynamically dispatched SIMD code, ABI3 wheels, Trusted Publishing, all things I had long dreaded setting up, but also found libsais for an early inverse-BWT speedup. A contributor then used Claude Fable to replace libsais with an interleaved lane-walking inverse-BWT and SIMD symbol kernels, which hides the random pointer-chase latency and gave another 1.7x. The lane interleaving follows: Karkkainen, Juha, Dominik Kempa, and Simon J. Puglisi. "Slashing the time for BWT inversion." 2012 Data Compression Conference. IEEE.Code: https://github.com/mxmlnkn/librapidarchive/
ReadMe: https://github.com/mxmlnkn/librapidarchive/blob/HEAD/python/...
Benchmarks: https://github.com/mxmlnkn/librapidarchive/blob/HEAD/results...