An Ode to Bzip

https://purplesyringa.moe/blog/an-ode-to-bzip/

40•signa11•2h ago

Comments

elophanto_agent•1h ago

bzip2 is the compression algorithm equivalent of that one coworker who does incredible work but nobody ever talks about. meanwhile gzip gets all the credit because it's "good enough"

kergonath•1h ago

Bzip2 is slow. That’s the main issue. Gzip is good enough and much faster. Also, the fact that you cannot get a valid bzip2 file by cat-ing 2 compressed files is not a deal breaker, but it is annoying.

saidnooneever•1h ago

the catting issue might be more an implementation of bzip program problem than algorithm (it could expect an array of compressed files). that would only be impossible if the program cannot reason about the length of data from file header, which again is technically not something about compression algo but rather file format its carried through.

that being said, speed is important for compression so for systems like webservers etc its an easy sell ofc. very strong point (and smarter implementation in programs) for gzip

nine_k•1h ago

Bzip2 is great for files that are compressed once, get decompressed many times, and the size is important. A good example is a software release.

pocksuppet•20m ago

So is xz, or zstd, and the files are smaller. bzip2 disappeared from software releases when xz was widely available. gzip often remains, as the most compatible option, the FAT32 of compression algorithms.

joecool1029•44m ago

> the catting issue might be more an implementation of bzip program problem than algorithm (it could expect an array of compressed files). that would only be impossible if the program cannot reason about the length of data from file header, which again is technically not something about compression algo but rather file format its carried through.

Long comment to just say: ‘I have no idea about what I’m writing about’

These compression algorithms do not have anything to do with filesystem structure. Anyway the reason you can’t cat together parts of bzip2 but you can with zstd (and gzip) is because zstd does everything in frames and everything in those frames can be decompressed separately (so you can seek and decompress parts). Bzip2 doesn’t do that.

So like, another place bzip2 sucks ass is working with large archives because you need to seek the entire archive before you can decompress it and it makes situations without parity data way more likely to cause dataloss of the whole archive. Really, don’t use it unless you have a super specific use case and know the tradeoffs, for the average person it was great when we would spend the time compressing to save the time sending over dialup.

nine_k•1h ago

Gzip is woefully old. Its only redeeming value is that it's already built into some old tools. Otherwise, use zstd, which is better and faster, both at compression and decompression. There's no reason to use gzip in anything new, except for backwards compatibility with something old.

kergonath•1h ago

> Otherwise, use zstd, which is better and faster

Yes, I do. Zstd is my preferred solution nowadays. But gzip is not going anywhere as a fallback because there is a surprisingly high number of computers without a working libzstd.

stefan_•44m ago

bzip and gzip are both horrible, terribly slow. Wherever I see "gz" or "bz" I immediately rip that nonsense out for zstd. There is such a thing as a right choice, and zstd is it every time.

laurencerowe•16m ago

lz4 can still be the right choice when decompression speed matters. It's almost twice as fast at decompression with similar compression ratios to zstd's fast setting.

https://github.com/facebook/zstd?tab=readme-ov-file#benchmar...

sedatk•22m ago

> the fact that you cannot get a valid bzip2 file by cat-ing 2 compressed files

TIL. Now that's why gzip has a file header! But, tar.gz compresses even better, that's probably why it hasn't caught on.

pocksuppet•19m ago

tar packs multiple files into one. If you concatenate two gzipped files and unzip them, you just get a concatenated file.

sedatk•17m ago

Ah okay, I thought gzip would support decompressing multiple files that way.

joecool1029•1h ago

Just use zstd unless you absolutely need to save a tiny bit more space. bzip2 and xz are extremely slow to compress.

silisili•1h ago

I'd argue it's more workload dependent, and everything is a tradeoff.

In my own testing of compressing internal generic json blobs, I found brotli a clear winner when comparing space and time.

If I want higher compatibility and fast speeds, I'd probably just reach for gzip.

zstd is good for many use cases, too, perhaps even most...but I think just telling everyone to always use it isn't necessarily the best advice.

joecool1029•57m ago

> If I want higher compatibility and fast speeds, I'd probably just reach for gzip.

It’s slower and compresses less than zstd. gzip should only be reached for as a compatibility option, that’s the only place it wins, it’s everywhere.

EDIT: If you must use it, use the modern implementation, https://www.zlib.net/pigz/

hexxagone•51m ago

In the LZ high compression regime where LZ can compete in terms of ratio, BWT compressors are faster to compress and slower to decompress than LZ codecs. BWT compressors are also more amenable to parallelization (check bsc and kanzi for modern implementations besides bzip3).

saghm•1h ago

Early on the article mentions that xz have zstd have gotten more popular than bzip, and my admitted naive understanding is that they're considered to have better tradeoffs in teems of collision compression time and overall space saved by compression. The performance section heavily discusses encoding performance of gzip and bzip, but unless I'm missing something, the only references to xz or zstd in that section are briefly handwaving about the decoding times probably being similar.

My impression is that this article has a lot of technical insight into how bzip compares to gzip, but it fails actually account for the real cause of the diminished popularity of bzip in favor of the non-gzip alternatives that it admits are the more popular choices in recent years.

hexxagone•54m ago

Notice that bzip3 has close to nothing to do with bzip2. It is a different BWT implementation with a different entropy codec, from a different author (as noted in the GitHub description "better and stronger spiritual successor to BZip2").

fl0ki•49m ago

This seems as good a thread as any to mention that the gzhttp package in klauspost/compress for Go now supports zstd on both server handles and client transports. Strangely this was added in a patch version instead of a minor version despite both expanding the API surface and changing default behavior.

https://github.com/klauspost/compress/releases/tag/v1.18.4

klauspost•14m ago

About the versioning, glad you spotted it anyway. There isn't as much use of the gzhttp package compared to the other ones, so the bar is a bit higher for that one.

Also making good progress on getting a slimmer version of zstd into the stdlib and improving the stdlib deflate.

pella•32m ago

imho: the future is a specialized compressor optimized for your specific format. ( https://openzl.org/ , ... )

srean•18m ago

That is an interesting link.

Does gmail use a special codec for storing emails ?

What happens when US economic data becomes unreliable

Sunsetting Jazzband

Montana passes Right to Compute act (2025)

An Ode to Bzip

Baochip-1x: What it is, why I'm doing it now and how it came about

The $2 per hour worker behind the OnlyFans boom

NMAP in the Movies

Python: The Optimization Ladder

Show HN: Learn Arabic with spaced repetition and comprehensible input

Cookie jars capture American kitsch (2023)

Megadev: A Development Kit for the Sega Mega Drive and Mega CD Hardware

1M context is now generally available for Opus 4.6 and Sonnet 4.6

Everything you never wanted to know about visually-hidden

9 Mothers Defense (YC P26) Is Hiring in Austin

Wired headphone sales are exploding

In Praise of Stupid Questions

Show HN: GitAgent – An open standard that turns any Git repo into an AI agent

Philosoph Jürgen Habermas Gestorben

Online astroturfing: A problem beyond disinformation

Generalizing Knuth's Pseudocode Architecture to Knowledge

Digg.com Closing Due to Spam

XML Is a Cheap DSL

Nominal Types in WebAssembly

Digg is gone again

RAM kits are now sold with one fake RAM stick alongside a real one

Can I run AI locally?

The Isolation Trap: Erlang

I found 39 Algolia admin keys exposed across open source documentation sites

Secure Secrets Management for Cursor Cloud Agents

Atari 2600 BASIC Programming (2015)

An Ode to Bzip

Comments

What happens when US economic data becomes unreliable

Sunsetting Jazzband

Montana passes Right to Compute act (2025)

An Ode to Bzip

Baochip-1x: What it is, why I'm doing it now and how it came about

The $2 per hour worker behind the OnlyFans boom

NMAP in the Movies

Python: The Optimization Ladder

Show HN: Learn Arabic with spaced repetition and comprehensible input

Cookie jars capture American kitsch (2023)

Megadev: A Development Kit for the Sega Mega Drive and Mega CD Hardware

1M context is now generally available for Opus 4.6 and Sonnet 4.6

Everything you never wanted to know about visually-hidden

9 Mothers Defense (YC P26) Is Hiring in Austin

Wired headphone sales are exploding

In Praise of Stupid Questions

Show HN: GitAgent – An open standard that turns any Git repo into an AI agent

Philosoph Jürgen Habermas Gestorben

Online astroturfing: A problem beyond disinformation

Generalizing Knuth's Pseudocode Architecture to Knowledge

Digg.com Closing Due to Spam

XML Is a Cheap DSL

Nominal Types in WebAssembly

Digg is gone again

RAM kits are now sold with one fake RAM stick alongside a real one

Can I run AI locally?

The Isolation Trap: Erlang

I found 39 Algolia admin keys exposed across open source documentation sites

Secure Secrets Management for Cursor Cloud Agents

Atari 2600 BASIC Programming (2015)