You could split the file into independently compressed blocks as well. But that'd reduce compression rate and require adding some kind of index for seeking.
Or they have an upper size limit for the file size they compress, since large files are rarely compressible text.
In any case it is something that needs the be handled before going live with a compressed cache. But the article sounds like they simply didn't implement compressed caching for those cases, which makes no sense.
MayeulC•44m ago
Weird, I would have compressed cold content instead, if the goal was to save on CPU time during decode.
genxy•28m ago
Zstd 3 to 5 is nearly free in terms of not bottlenecking disk or network. Zstd 12 to 19 gives amazing compression results and still result in speedups when reading from disk. It really is a wonderful all purpose compressor.
One of the nice things about Zstd is if you try to compress and already compressed stream, it short circuits. So even if you are given say HVEC MP4 and run zstd -19 on it, it will "compress" immediately and not DOS your pipeline.